Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created online docs, fixed div class issue used to scrape lyrics #153

Merged
merged 23 commits into from
Aug 31, 2020
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = src
BUILDDIR = build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
35 changes: 35 additions & 0 deletions docs/make.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
@ECHO OFF

pushd %~dp0

REM Command file for Sphinx documentation

if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=src
set BUILDDIR=build

if "%1" == "" goto help

%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)

%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end

:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%

:end
popd
2 changes: 2 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
sphinx==3.2.0
sphinx-rtd-theme==0.5.0
Binary file added docs/src/404.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
27 changes: 27 additions & 0 deletions docs/src/404.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
.. image:: 404.png


Whoops!
========
The page you're looking for doesn't exist. Let's go
:ref:`home <index>`.

.. toctree::
:maxdepth: 1
:hidden:
:caption: Library

reference
release_notes

.. toctree::
:maxdepth: 1
:hidden:
:caption: Guide

setup
usage
contributing


.. _Genius.com: https://www.genius.com
35 changes: 35 additions & 0 deletions docs/src/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import os
import sys

sys.path.insert(0, os.path.abspath('../..'))

# -- Project information -----------------------------------------------------

project = 'lyricsgenius'
copyright = '2020, John W. Miller, Allerter'
author = 'John W. Miller, Allerter'

# -- General configuration ---------------------------------------------------

extensions = [
'sphinx_rtd_theme',
'sphinx.ext.napoleon',
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.extlinks',
]

exclude_patterns = ['build']
master_doc = 'index'
autosummary_generate = True
# -- Options for HTML output -------------------------------------------------

highlight_language = 'python3'
html_theme = 'sphinx_rtd_theme'

# -- Other -------------------------------------------------------------------

extlinks = {
'issue': ('https://github.com/johnwmillr/LyricsGenius/issues/%s', '#'),
'commit': ('https://github.com/johnwmillr/LyricsGenius/commit/%s', '')
}
16 changes: 16 additions & 0 deletions docs/src/contributing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Contributing
------------
Please contribute! Genius has a lot of `undocumented API endpoints`_, and
the ones in that link are only some of them. You could try to implement the
ones in the link or look through Genius yourself to uncover new ones, and
implement them in LyricsGenius to make this client richer in features, and
the access it offers to Genius.com.

If you want to fix a bug, suggest improvements, or
add new features to the project, just `open an issue`_ on GitHub.



.. _open an issue: https://github.com/johnwmillr/LyricsGenius/issues
.. _undocumented API endpoints: https://github.com/shaedrich/
geniusly/wiki/Undocumented-API-endpoints
16 changes: 16 additions & 0 deletions docs/src/examples/example projects.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Example projects
==================

- `Trucks and Beer: A textual analysis of popular country music`_
- `Neural machine translation: Explaining the Meaning Behind Lyrics`_
- `What makes some blink-182 songs more popular than others?`_
- `Sentiment analysis on hip-hop lyrics`_
- `Does Country Music Drink More Than Other Genres?`_
- `49 Years of Lyrics: Why So Angry?`_

.. _`Trucks and Beer: A textual analysis of popular country music`: http://www.johnwmillr.com/trucks-and-beer/
.. _`Neural machine translation: Explaining the Meaning Behind Lyrics`: https://github.com/tsandefer/dsi_capstone_3
.. _What makes some blink-182 songs more popular than others?: http://jdaytn.com/posts/download-blink-182-data/
.. _Sentiment analysis on hip-hop lyrics: https://github.com/Hugo-Nattagh/2017-Hip-Hop
.. _Does Country Music Drink More Than Other Genres?: https://towardsdatascience.com/does-country-music-drink-more-than-other-genres-a21db901940b
.. _`49 Years of Lyrics: Why So Angry?`: https://towardsdatascience.com/49-years-of-lyrics-why-so-angry-1adf0a3fa2b4
73 changes: 73 additions & 0 deletions docs/src/examples/snippets.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
.. _snippets:

Snippets
==================
Here are some snippets showcasing how the library can be used.
allerter marked this conversation as resolved.
Show resolved Hide resolved


- `All the songs of an artist`_
- `Artist's least popular song`_
- `YouTube URL of artist's songs`_
- `Searching for a song by lyrics`_


All the songs of an artist
------------------------------

.. code:: python

from lyricsgenius import Genius

genius = Genius(token)
genius.search_artist('Andy Shauf')
artist.save_lyrics()


Artist's least popular song
----------------------------
.. code:: python

genius = Genius(token)

artist = genius.search_artist('Andy Shauf', max_songs=1)
page = 1
songs = []
while page:
request = genius.get_artist_songs(artist._id,
sort='popularity',
per_page=50,
page=page)
songs.extend(request['songs'])
page = request['next_page']
least_popular_song = genius.search_song(songs[-1]['title'], artist.name)
print(least_popular_song.lyrics)


YouTube URL of artist's songs
------------------------------
.. code:: python

import json
# we have saved the songs with artist.save_lyrics() before this
with open('saved_file.json') as f:
data = json.load(f)
for song in data['songs']:
links = song['media']
if links:
for media in links:
if media['provider'] == 'youtube':
print(print(['song'] + ': ' + media['url'])
break


Searching for a song by lyrics
-------------------------------
.. code:: python

from lyricsgenius import Genius

genius = Genius(token)

request = genius.search_genius_web('Jeremy can we talk a minute?')
for hit in request['sections'][2]['hits']:
print(hit['result']['title'])
Binary file added docs/src/header.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
55 changes: 55 additions & 0 deletions docs/src/how_it_works.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
.. _how-it-works:


How It Works
#############
LyricsGenius makes two kinds of calls to the Genius API. The first one is
the API that requires an access token, and the other one to the public API.

The API
**********
This API is available through
`api.genius.com <http://api.genius.com>`_, and provides access to features like
searching songs, getting song or annotation data by
providing their ID, and some other features. But the problem with this API
is that Genius has only exposed a little of the whole API.
This is not the case for the second kind of calls: the ones to the public API.

The Public API
***************
The public API is the API that doesn't require an access token and can be
accessed by anyone (end-users use this on their browsers). The public API
can be called at `genius.com/api <http://genius.com/api>`_. This API
exposes a whole other set of features (and sometimes access to the same) than
the *token API*, prominently searching Genius web. Although these two calls
can provide you with a lot of information, they still don't let you anywhere
near what's probably the most important thing you may want from Genius:
the lyrics.


The Lyrics
**********
Genius has legal agreements with music publishers and considers the lyrics
on their website to be a legal property of Genius, and won't allow you
to re-use their lyrics without explicit licensing. They even
`sued Google on grounds of stolen lyrics`_, asking for $50 million in damages,
but `to no avail`_. So it shouldn't come as a surprise if they don't
provide lyrics in calls to
the API. So how does LyricsGenius get the lyrics?

LyricsGenius uses a web-scraping library called `Beautiful Soup`_
to scrape lyrics from the song's page on Genius. Scraping the lyrics in
this way violates Genius' terms of service. If you intend to use the lyrics for
personal purposes, that shouldn't be cause for trouble, but other than that,
you should inquire what happens when you violate the terms this way.
As a reminder, LyricsGenius is not responsible for your usage of the library.


.. _the Genius API: http://genius.com/api-clients\
.. _create a new API client: https://genius.com/api-clients/new
.. _sued Google on grounds of stolen lyrics: https://www.theverge.com/
2020/8/11/21363692/google-genius-lyrics-lawsuit-scraping-copyright
-yelp-antitrust-competition
.. _to no avail: https://www.theverge.com/2020/8/11/21363692/
google-genius-lyrics-lawsuit-scraping-copyright-yelp-antitrust-competition
.. _Beautiful Soup: https://pypi.org/project/beautifulsoup4/
86 changes: 86 additions & 0 deletions docs/src/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
.. _index:
.. image:: header.png


LyricsGenius: a Python client for the Genius.com API
====================================================
.. image:: https://travis-ci.org/johnwmillr/LyricsGenius.svg?branch=master
:target: https://travis-ci.org/johnwmillr/LyricsGenius
.. image:: https://badge.fury.io/py/lyricsgenius.svg
:target: https://pypi.org/project/lyricsgenius/
.. image:: https://img.shields.io/badge/python-3.x-brightgreen.svg
:target: https://pypi.org/project/lyricsgenius/

`Genius.com`_ is a fun website. If you aren’t familiar with it, Genius hosts a
bunch of song lyrics and lets users highlight and annotate passages with
interpretations, explanations, and references. Originally called RapGenius.com
and devoted to lyrics from rap and hip-hop songs, the website now includes
lyrics and annotations from all genres of music. You can figure out what
`“Words are flowing out like endless rain into a paper cup”`_ from Across the
Universe really means, or what Noname was referring to when she said `“Moses
wrote my name in gold and Kanye did the eulogy”`_.

It’s actually not too difficult to start pulling data from the Genius website.
Genius is hip enough to provide a free application programming interface (API)
that makes it easy for nerds to programmatically access song and artist data
from their website. What the Genius API doesn’t provide, however,
is a way to download the lyrics themselves. With a little help from
`Beautiful Soup`_ though, it’s possible to grab the song lyrics without too
much extra work. And LyricsGenius has done all of that for you already.

..
source::https://www.johnwmillr.com/scraping-genius-lyrics/
allerter marked this conversation as resolved.
Show resolved Hide resolved


``lyricsgenius`` provides a simple interface to the song, artist, and
lyrics data stored on `Genius.com`_.

Using this library you can convienently access the content on Genius.com
And much more using the public API.

You can use ``pip`` to install lyricsgenius:

.. code:: bash

pip install lyricsgenius

LyricsGenius provides lots of features to work with. For example, let's
download all the lyrics of an artist's songs, and save them to a JSON
file:

.. code:: python

from lyricsgenius import Genius

genius = Genius(token)
genius.search_artist('Andy Shauf')
artist.save_lyrics()

But before using the library you will need to get an access token. Head over
to :ref:`setup` to get started.

.. toctree::
:maxdepth: 1
:hidden:
:caption: Library

reference
release_notes

.. toctree::
:maxdepth: 1
:caption: Guide

setup
usage
how_it_works
text_formatting
contributing


.. _Genius.com: https://www.genius.com
.. _“Words are flowing out like endless rain into a paper cup”:
https://genius.com/3287551
.. _“Moses wrote my name in gold and Kanye did the eulogy”:
https://genius.com/10185147
.. _`Beautiful Soup`: https://www.crummy.com/software/BeautifulSoup/