Exploring the UK Charts with Neo4j

The code in this repository goes with the <> blog post.

Downloading and scraping charts

If you want to download the different charts, you can do this by running the following script:

python scripts/download.py

The HTML page for each chart will be written into files in the raw/charts directory.

We now need to call the Spotify API to pull in the full list of collaborating artists for each track. I’ve written a blog post explaining how to create a Spotify application and get the corresponding credentials. Once we’ve skimmed that blog post, let’s create a file called .env and populate it with our Spotify application credentials:

.env

export CLIENT_ID="<our-client-id>"
export CLIENT_SECRET="<our-client-secret>"

We can then run the following command to add these values as environment variables:

source .env

Now let’s execute the following script to pull down the collaborating artists:

python scripts/download_songs_spotify.py

This script writes its results into the raw/songs directory.

And finally, let’s run the following script, which will create a JSON document per chart entry in import/items.json:

python scripts/scrape.py