Repository navigation
Expand file tree
/
Copy path2018-06-text-analytics-song-lyrics.Rmd
More file actions
374 lines (275 loc) · 12.1 KB
/
Copy path2018-06-text-analytics-song-lyrics.Rmd
File metadata and controls
374 lines (275 loc) · 12.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
---
title: Text analytics & topic modelling on music genres song lyrics
author: ''
date: '2018-06-24'
description: "Text analytics & topic modelling in more than 380,000 song lyrics, of various music genres in the past 50 years"
slug: text-analytics-topic-modelling-songs-lyrics
categories:
- R
tags:
- Text analytics
- Tidytext
- LDA
- Topic modelling
---
This analysis uses a dataset of more than 380,000 songs since 1970 published
in [kaggle](https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics/data).
The main objective is to **develop clusters of music genres by the song lyrics**
and the steps are the following:
* Data preparation (cleansing, transform etc.)
* Exploratory analysis
* Topic modelling
Various R libraries were used, but it is mainly based on [#tidytext](https://twitter.com/hashtag/tidytext?src=hash)
and [#tidyverse](https://twitter.com/hashtag/tidyverse?src=hash) environment.
```{r message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
# Libraries
library(readr)
library(tidyverse)
library(stringr)
library(tidytext)
library(lubridate)
library(wordcloud)
library(topicmodels)
library(tm)
library(stopwords)
library(quanteda)
library(ggthemes)
```
Data preparation was a very important part of the analysis. The first
step was to exclude all songs, where the text field of lyrics was less than 10
characters long.
Then, after looking at the lyrics text field, it seems that there were a lot
of non-English songs. The decision was to include only English
language songs in the final dataset. So, by using the **cld3** library, the
origin of the songs was detected & added to the dataset. The language detection wasn't perfect, as it
misclassified a few songs. But because of the large original dataset, i
decided to use it and remove all non-English songs from the dataset.
Finally all songs that music genre was either missing or unknown were removed.
Also all songs with invalid year input (less than 1970) were removed.
More details about these steps can be found, in the actual code, at the end of
the article.
```{r eval=FALSE, message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
# Insert data
songs <- read_csv("/Users/manos/Onedrive/Projects/R/Blog/data/songs.csv")
```
```{r eval=FALSE, message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
# DATA CLEANSING
## Filter data, handle missing values
# Keep songs with 10 or more characters in the lyrics
songs <-
songs %>%
filter(str_length(lyrics) > 10)
# Detect the language of the song
library(cld3)
songs$lang <- detect_language(songs$lyrics)
# Filter the songs that
songs <-
songs %>%
filter(lang == "en" & is.na(genre) == FALSE & year >= 1970)
songs$characters <- str_count(songs$lyrics)
```
```{r eval=FALSE, message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
## _text cleaning
# Create a vector with stopwords
stopwords <- c(stopwords())
# Clean text
songs$lyrics <- tolower(songs$lyrics)
songs$lyrics <- removePunctuation(songs$lyrics)
songs$lyrics <- removeNumbers(songs$lyrics)
songs$lyrics <- stripWhitespace(songs$lyrics)
songs$lyrics <- removeWords(songs$lyrics, stopwords)
songs$lyrics <- stemDocument(songs$lyrics)
# Save processed data for future use
saveRDS(songs, file = "/Users/manos/OneDrive/Projects/R/All_Projects/Songs_Lyrics/data/cleaned_data.RDS")
```
```{r message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
## _Insert processed dataset
songs <- readRDS(file = "/Users/manos/OneDrive/Projects/R/All_Projects/Songs_Lyrics/data/cleaned_data.RDS")
```
# MAIN ANALYSIS
Below there is a statistical table and a frequency plot to indicate the
differences between the music genres.
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
# library(DT)
#
# knit_print.data.frame = function(x, ...) {
# knit_print(DT::datatable(x), ...)
# }
songs_data <-
songs %>%
group_by(genre) %>%
count() %>%
ungroup() %>%
mutate(`Proportion(%)` = round((n/sum(n))*100, 2)) %>%
arrange(-n) %>%
rename(Genre = genre,
`Total songs` = n)
knitr::kable(songs_data, caption = "Songs per genre")
# knit_print.data.frame()
```
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
songs %>%
group_by(genre) %>%
count() %>%
ungroup() %>%
mutate(Freq = n/sum(n)) %>%
arrange(-Freq) %>%
ggplot() +
geom_col(aes(reorder(genre, -Freq), Freq), fill = "steelblue", alpha = 0.7) +
labs(y = "Number of songs", x = "Music Genre",
title = "Proportion of songs per music genre",
subtitle = "From 1970 to 2016")+
theme_fivethirtyeight() +
scale_y_continuous(labels = scales::percent_format())
```
Most songs belong to the **Rock** genre, almost 50% of all songs
in this dataset. The top 5 music genres are Rock, Pop, Hip-Hop, Metal & Country.
It would be interesting to see if this is constant through time, or not.
Below there is a plot indicates the variation through years.
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
songs %>%
mutate(date = as_date(paste(as.character(songs$year), "-01", "-01"))) %>%
mutate(decade = floor_date(date, years(5))) %>%
group_by(decade, genre) %>%
summarise(N = n()) %>%
mutate(freq = round(N/sum(N), 2)) %>%
filter(genre %in% c("Country", "Hip-Hop", "Metal", "Pop", "Rock")) %>%
ggplot(aes(decade, freq, colour = genre)) +
# geom_line() +
geom_smooth(se = FALSE) +
labs(y = "Smoothed proportion (%)", x = "Year",
title = "Smoothed proportion of total songs per Music Genre",
subtitle = "Top 5 genres are selected")+
theme_fivethirtyeight() +
scale_y_continuous(labels = scales::percent_format())
```
There are a few findings here. At first Rock genre decreases (from 60 % to 25 %)
in overall proportion of songs. On the other hand, hip-hop gradually rises
(currently near to 20 %). Pop is around to 20 % and fairly steady through years.
Now let's try to figure out which music genre uses more lyrics.
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
## number of characters per song
songs %>%
ggplot() +
geom_boxplot(aes(genre, characters), fill = "steelblue", alpha = 0.7) +
labs(y = "Length of song lyrics (in characters)", x = "Music Genre",
title = "Length (in characters) of song lyrics per music genre",
subtitle = "From 1970 to 2016")+
theme_fivethirtyeight() +
ylim(0, 10000)
songs %>%
group_by(genre) %>%
summarise(characters = round(mean(characters, na.rm = TRUE), 0)) %>%
ggplot(aes(reorder(genre, -characters), characters)) +
geom_col(fill = "steelblue", alpha = 0.7) +
labs(y = "Length of song lyrics (in characters)", x = "Music Genre",
title = "Average lyrics characters per music genre song",
subtitle = "From 1970 to 2016")+
theme_fivethirtyeight()
```
Hip-Hop seems to be a significantly different music genre, as it uses
**more than double** lyrics per song than the rest of the genres.
# WORDCLOUD
It is interesting to see which are the most used words in each music genre.
Below there are word clouds for the top 5 music genres.
Word clouds (also known as text clouds or tag clouds) work in a simple way:
the more a specific word appears in a source of textual data, the bigger
and bolder it appears in the word cloud.
Below there are word clouds for the top 5 music genres.
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
words <-
songs %>%
unnest_tokens(word, lyrics) %>%
group_by(genre, word) %>%
count() %>%
arrange(-n) %>%
group_by(genre) %>%
top_n(n = 100, wt = n)
# Select top 5 genres
genres <- c("Rock", "Hip-Hop", "Pop", "Country", "Metal")
for(i in 1:length(genres)){
temp <- filter(words, genre == genres[i])
# Create a word cloud
par(bg="grey30")
wordcloud(words = temp$word, freq = temp$n, col=terrain.colors(length(temp$word), alpha=0.9), random.order=FALSE, rot.per=0.3 )
title(main = genres[i] , font.main = 1, col.main = "cornsilk3", cex.main = 1.2)
}
```
# TOPIC MODELLING
Consider, for example, a situation in which you are confronted with a large
collection of documents but have no idea what they are about. One of the first
things you might want to do is to classify these documents into topics or themes.
Among other things this would help you figure out if there’s anything interest
while also directing you to the relevant subsets of the corpus. For small
collections, one could do this by simply going through each document but this
is clearly unfeasible for corpuses containing thousands of documents.
**Topic modeling** deals with the problem of automatically
classifying sets of documents into themes. The algorithm chosen is
**Latent Dirichlet Allocation or LDA**, which essentially is a technique that
facilitates the automatic discovery of themes in a collection of documents.
The basic assumption behind LDA is that each of the documents in a collection
consist of a mixture of collection-wide topics. However, in reality we observe
only documents and words, not topics – the latter are part of the hidden (or latent)
structure of documents. The aim is to infer the latent topic structure given the
words and document. LDA does this by recreating the documents in the corpus by
adjusting the relative importance of topics in documents and words in topics
iteratively.
In our case an LDA model with two topics was developed. After computing the topic
probabilities for all songs, we can see if this unsupervised learning, distinguish
or reveal associations between music genres (regarding their lyrics).
The box-plot below, reveals the probabilities of each music genre song to belong
in each of the three topics.
```{r eval=FALSE, message=FALSE, warning=FALSE, include=FALSE, paged.print=FALSE}
# _Build Model ###############################################################
# split into words
by_word <-
songs %>%
unnest_tokens(word, lyrics)
# find document-word counts
word_counts <-
by_word %>%
count(song, word, sort = TRUE) %>%
ungroup()
# Create document term matrix
songs_dtm <- word_counts %>%
cast_dtm(song, word, n)
songs_lda <- LDA(songs_dtm, k = 3, control = list(seed = 1234))
# Save for future use
save(songs_lda, file = "/Users/manos/OneDrive/Projects/R/All_Projects/Songs_Lyrics/objects/songs_lda_3.RDA")
```
```{r echo=FALSE, message=FALSE, warning=FALSE, paged.print=FALSE}
# Load the model
load(file = "/Users/manos/OneDrive/Projects/R/All_Projects/Songs_Lyrics/objects/songs_lda_2.RDA")
# _Calculate Tables ##########################################################
library(tidytext)
ap_topics <- tidy(songs_lda, matrix = "gamma")
topics_probs <- right_join(ap_topics, songs[, c("song", "genre")], by = c("document" = "song"))
topics_probs %>%
mutate(genre = reorder(genre, gamma * topic)) %>%
ggplot(aes(factor(topic), gamma, colour = factor(topic))) +
geom_boxplot(alpha = 0.7) +
labs(y = "Probability", x = "Topic",
title = "Topic probabilities per music genre",
subtitle = "") +
theme_fivethirtyeight() +
labs(col="Topics",
y = "Probabilities") +
facet_wrap(~ genre)
```
Hip-Hop genre is almost uniquely identified as a single topic (topic 2).
The rest of the music genres seem to be identified as another topic.
So we can say that **Hip-Hop is definetely a music genre that uses significantly different language**
in the lyrics than the rest of the genres.
More LDA models were developed (three & four topics) but the outcome of the initial
model (two topics) was more relevant and significant.
**CONCLUSION**
Finally we can conclude that:
- There is one music genre, Hip-hop, that is **significantly** different than
the rest of the genres, as it uses much more & different lyrics.
- Rock genre is gradually decreases in popularity through years, by including less
songs and new genres are emerging.
**HARDWARE ENVIRONMENT**
All tasks of the analysis were accomplished on a laptop with 8 GB RAM & Intel
Core I5 2.1 GHz. Some of the tasks can be demanding (for large datasets with
text data). e.g. the LDA modelling task took around 7-8 minutes to complete.
[Full R code](https://github.com/mantoniou/Blog/blob/master/content/post/2018-06-text-analytics-song-lyrics.Rmd)