Saved Songs Download and User Authentication #1240

AZMCode · 2021-03-29T00:48:22Z

What's on the tin.

It does add a few packages, but allows the downloading of private playlists & full library download.
Still not working, but I'm nearly there.

Switch to rich (#1098)

Catching up to Dev

…to dev

- user auth - library download

AZMCode · 2021-03-29T00:51:57Z

I believe this would solve #1234

spotdl/__main__.py

Problem is how can I feed the access and refresh tokens to spotipy?

ghost · 2021-03-29T15:45:36Z

Please keep your commit messages clear, for example the last commit message:

Problem is how can I feed the access and refresh tokens to spotipy?

Is that the issue you fixed or the one you need to fix? If it's the latter, just what did you put into the commit then?

Even with the title:

Finished implementing OAuth2.

It can mean a lot of things. It could be a redesign of the auth process or you implemented spotipy oauth (which is done by default) which leaves the question of why are the access tokens an issue anymore...

See point 2 at this subsection

P.S. I haven't read the code, I just wanna say that your commit messages should cover (a) why changes were made and (b) how do those changes deal with the issue. What changes are made are indicated by the git diff so you don't have to mention that part.

AZMCode · 2021-03-29T15:47:39Z

It's the latter. I'm sorry for the slip, I'll keep commit messages more focused in future.

Over all these commits I've added a flag to spotdl, which is --user-authorization.

This triggers a different authorization flow, which is the Authorization Code Flow, which I've implemented up to the point where the access/refresh tokens are first fetched.

To keep things easy for the user, I've used pywebview to open the login screen, which is necessary for the Authorization Code Flow.

I'm currently having issues integrating my implementation of the OAuth flow and spotipy.

AZMCode · 2021-03-29T16:08:20Z

I've also begun adding support for the "saved" keyword

If, when parsing arguments, the word "saved" is found in the place of a URL, spotipy will download the user's Saved Songs. That's if the user also has used the --user-authorization flag. Otherwise it will raise an Exception telling the user to use the flag.

- Turned get_tokens function into SpotifyUserAuth (custom Spotipy-compatible Auth Manager) - SpotifyUserAuth now automatically refreshes access tokens before they expire - Added function to utils.py: get_saved_tracks, which gets all saved tracks by the current user. Fails without --user-auth - Added "saved" keyword, so that if it appears as a url in the command line, it downloads all saved tracks instead. - Shut off logging by http.server instance

AZMCode · 2021-03-29T23:09:16Z

There goes the functionality as far as I'm aware. Now I need to write the tests for literally 300 lines of new code. That's for tomorrow though. I'm guessing it will be kinda hard to write tests with webview though.

ghost · 2021-03-30T06:14:01Z

One more note, Its a good practice to keep commits restricted to one-change-a-commit. If you have to rollback a commit and you've grouped multiple changes into a single commit you'll loose a lot of good work.

You can see the first point here and this page for more details if you wish.
"How to figure out if my commit is good?" - simple diagram

AZMCode · 2021-03-30T11:22:09Z

Thanks, I'll keep that in mind

AZMCode · 2021-03-30T12:55:54Z

I'll be the first to admit: I'm new to python, and even after reading through the existing tests, I can't seem to figure out how to test downloading private playlists and the saved songs library. All current tests check out though, so after reverting to the old API keys I'm opening this for review.

AZMCode · 2021-03-30T14:51:34Z

Well I'm stumped. Tests run fine on my machine.
Also, one of the tests failed when installing dependencies. Ffmpeg again.

xnetcat · 2021-03-30T17:43:50Z

Well I'm stumped. Tests run fine on my machine.
Also, one of the tests failed when installing dependencies. Ffmpeg again.

Sometimes python3.8 checks are failing, you just have to re-run all jobs

AZMCode · 2021-03-30T17:55:51Z

I don't believe the macos 3.8 test should be taking this long... 20 minutes in when other tests finished in 1

xnetcat · 2021-03-30T17:58:34Z

I don't believe the macos 3.8 test should be taking this long... 20 minutes in when other tests finished in 1

GitHub actions are not perfect

xnetcat · 2021-03-30T18:08:23Z

I think that pywebview is overkill for this task. We could open the default browser instead with webbrowser that comes included with stdlib

https://docs.python.org/3/library/webbrowser.html

AZMCode · 2021-03-30T18:23:11Z

The reason I'm choosing webview over the browser is that it makes the experience seamless. If the client closes the login window, for example, we can detect that and raise an Exception. In any case, webview could be replaced by a browser if you really think it's best.
Edit: I'm also lobbying over at pywebview to make this process serverless, to make things cleaner.

Silverarmor · 2021-03-30T23:37:38Z

Please don't send the .tox/ directory.
add it to the .gitignore maybe

AZMCode · 2021-03-30T23:38:45Z

Will do

Silverarmor · 2021-03-30T23:40:32Z

ah I see #1233 adds it to gitignore

spotdl/__main__.py

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

spotdl/search/spotifyClient.py

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

spotdl/search/spotifyClient.py

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

xnetcat · 2021-04-01T21:06:58Z

spotdl/search/utils.py

+def get_saved_tracks() -> List[SongObj]:
+    '''
+    returns a `list<songObj>` containing Url's of each track in the user's saved tracks
+    '''
+
+    spotifyClient = SpotifyClient()
+
+    initialRequest = spotifyClient.current_user_saved_tracks()
+    librarySize = int(initialRequest["total"])
+    with ThreadPoolExecutor() as pool:
+        futures = []
+        for i in range(librarySize):
+            if i % 20 == 0:
+                futures.append(
+                    pool.submit(
+                        spotifyClient.current_user_saved_tracks,
+                        offset=i
+                    )
+                )
+        doneFutures = wait(futures)[0]
+        doneLists = map(lambda x: x.result()["items"], doneFutures)
+        songObjFutures = []
+        for currList in doneLists:
+            for songSimplified in currList:
+                songObjFutures.append(
+                    pool.submit(
+                        SongObj.from_url,
+                        'https://open.spotify.com/track/' + songSimplified['track']['id']
+                    )
+                )
+        doneSongObjsFutures = wait(songObjFutures)[0]
+        doneSongObjs = list(map(lambda x: x.result(), doneSongObjsFutures))
+        return doneSongObjs


I don't know if we should run this in parallel (we might hit youtube rate limit faster if user has a lot of songs). What do you think @Silverarmor?

If we run it in parallel, spotDl can no longer be run on termux. Neither will it provide a significant boost over asyncio.

Between parallel and asyncio. Asyncio is better.

/cc @aklajnert

Should I go back to sequential fetching then?

Yup, just like the other functions in utils.py

xnetcat

LGTM 👍

AZMCode · 2021-04-01T22:13:45Z

:D

Silverarmor · 2021-04-11T06:00:02Z

Please pull from upstream/dev

spotdl/__main__.py

Co-authored-by: Silverarmor <23619946+Silverarmor@users.noreply.github.com>

spotdl/__main__.py

xnetcat · 2021-04-29T20:56:45Z

Please pull changes from upstream/dev

ghost

@Silverarmor, this PR is a Go.

Silverarmor · 2021-05-11T02:32:20Z

@aklajnert Please re-review to verify your requested changes have been made

AZMCode · 2021-05-18T15:23:28Z

Nice! Happy to see this finally come through.

Publish v3.6.0 * ignore .cache and other hidden files (#1274) * Bump minimal required python version to 3.6.1 (#1278) * Remove FFmpeg normalization causing "quiet" songs. (#1276) * Saved Songs Download and User Authentication (#1240) * regenerate cassettes (#1290) * Use ffmpeg_path to check for version (#1289) * Skip already downloaded songs before doing youtube search (#1287) * Fix security risk (#1285) * Song matching improvements (#1279) * Artist songs fixes (#1284) * More output formats (#1244) * Bump version number to 3.6.0 * Update .gitignore to remove duplicate cache * docs update (#1293) Co-authored-by: Silverarmor <23619946+Silverarmor@users.noreply.github.com> Co-authored-by: Jakub Kot <42355410+xnetcat@users.noreply.github.com> Co-authored-by: Peyton Creery <44987569+phcreery@users.noreply.github.com> Co-authored-by: AZMCode <adrianozambrana@protonmail.com> Co-authored-by: Aiden Gardner <19619206+aiden2480@users.noreply.github.com> Co-authored-by: Oliver Blanthorn <freedom4cows@gmail.com> Co-authored-by: Andrzej Klajnert <github@aklajnert.pl>

AZMCode added 4 commits March 25, 2021 13:55

Merge pull request #1 from spotDL/dev

db21ee8

Switch to rich (#1098)

Merge pull request #3 from spotDL/dev

64b01aa

Catching up to Dev

Merge branch 'dev' of https://github.com/spotdl/spotify-downloader in…

38944cc

…to dev

Initial Attempts at implementing:

593e9cb

- user auth - library download

Silverarmor reviewed Mar 29, 2021

View reviewed changes

spotdl/__main__.py Outdated Show resolved Hide resolved

Finished implementing OAuth2.

f0284d9

Problem is how can I feed the access and refresh tokens to spotipy?

AZMCode added 3 commits March 29, 2021 17:52

Fixed flake8 and mypy errors

7a6c7bd

Attempt at fixing tests

f91aa11

Reverted to old API keys

06dbc70

AZMCode marked this pull request as ready for review March 30, 2021 12:59

AZMCode requested a review from Silverarmor March 30, 2021 12:59

Flash fix because of a missing comma

eedb4b1

Yet another typo

57d058a

xnetcat reviewed Apr 1, 2021

View reviewed changes

spotdl/__main__.py Outdated Show resolved Hide resolved

AZMCode and others added 2 commits April 1, 2021 16:36

Fixed flake8 mistakes

513e3af

Wrong example in help fixed

2d10583

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

AZMCode requested a review from xnetcat April 1, 2021 20:39

xnetcat reviewed Apr 1, 2021

View reviewed changes

spotdl/search/spotifyClient.py Outdated Show resolved Hide resolved

Typo in docstring

18c9d01

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

AZMCode requested a review from xnetcat April 1, 2021 20:56

xnetcat reviewed Apr 1, 2021

View reviewed changes

spotdl/search/spotifyClient.py Outdated Show resolved Hide resolved

Another typo

9e046e3

Co-authored-by: Jakub <42355410+xnetcat@users.noreply.github.com>

xnetcat reviewed Apr 1, 2021

View reviewed changes

xnetcat approved these changes Apr 1, 2021

View reviewed changes

Silverarmor reviewed Apr 11, 2021

View reviewed changes

spotdl/__main__.py Outdated Show resolved Hide resolved

spotdl/__main__.py Outdated Show resolved Hide resolved

xnetcat and others added 3 commits April 11, 2021 13:15

update client id

76334c4

Co-authored-by: Silverarmor <23619946+Silverarmor@users.noreply.github.com>

update client secret

3a495a1

Co-authored-by: Silverarmor <23619946+Silverarmor@users.noreply.github.com>

Merge branch 'dev' into saved

b3e4fdc

aklajnert requested changes Apr 11, 2021

View reviewed changes

spotdl/__main__.py Outdated Show resolved Hide resolved

AZMCode added 2 commits April 11, 2021 13:02

Auto --user-auth behavior

f90a251

Fixed flake8 error

ac457b4

AZMCode requested a review from aklajnert April 11, 2021 17:08

xnetcat linked an issue Apr 30, 2021 that may be closed by this pull request

Downloading all of a user's saved songs #1234

Closed

ghost approved these changes May 11, 2021

View reviewed changes

aklajnert approved these changes May 11, 2021

View reviewed changes

Silverarmor merged commit 9c0966a into spotDL:dev May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Saved Songs Download and User Authentication #1240

Saved Songs Download and User Authentication #1240

AZMCode commented Mar 29, 2021

AZMCode commented Mar 29, 2021

ghost commented Mar 29, 2021 •

edited by ghost

AZMCode commented Mar 29, 2021 •

edited

AZMCode commented Mar 29, 2021

AZMCode commented Mar 29, 2021 •

edited

ghost commented Mar 30, 2021 •

edited by ghost

AZMCode commented Mar 30, 2021

AZMCode commented Mar 30, 2021 •

edited

AZMCode commented Mar 30, 2021

xnetcat commented Mar 30, 2021

AZMCode commented Mar 30, 2021

xnetcat commented Mar 30, 2021

xnetcat commented Mar 30, 2021 •

edited

AZMCode commented Mar 30, 2021 •

edited

Silverarmor commented Mar 30, 2021

AZMCode commented Mar 30, 2021

Silverarmor commented Mar 30, 2021

xnetcat Apr 1, 2021 •

edited

ghost Apr 2, 2021

AZMCode Apr 11, 2021

xnetcat Apr 29, 2021

xnetcat left a comment

AZMCode commented Apr 1, 2021

Silverarmor commented Apr 11, 2021

xnetcat commented Apr 29, 2021

ghost left a comment

Silverarmor commented May 11, 2021

AZMCode commented May 18, 2021

Saved Songs Download and User Authentication #1240

Saved Songs Download and User Authentication #1240

Conversation

AZMCode commented Mar 29, 2021

AZMCode commented Mar 29, 2021

ghost commented Mar 29, 2021 • edited by ghost

AZMCode commented Mar 29, 2021 • edited

AZMCode commented Mar 29, 2021

AZMCode commented Mar 29, 2021 • edited

ghost commented Mar 30, 2021 • edited by ghost

AZMCode commented Mar 30, 2021

AZMCode commented Mar 30, 2021 • edited

AZMCode commented Mar 30, 2021

xnetcat commented Mar 30, 2021

AZMCode commented Mar 30, 2021

xnetcat commented Mar 30, 2021

xnetcat commented Mar 30, 2021 • edited

AZMCode commented Mar 30, 2021 • edited

Silverarmor commented Mar 30, 2021

AZMCode commented Mar 30, 2021

Silverarmor commented Mar 30, 2021

xnetcat Apr 1, 2021 • edited

Choose a reason for hiding this comment

ghost Apr 2, 2021

Choose a reason for hiding this comment

AZMCode Apr 11, 2021

Choose a reason for hiding this comment

xnetcat Apr 29, 2021

Choose a reason for hiding this comment

xnetcat left a comment

Choose a reason for hiding this comment

AZMCode commented Apr 1, 2021

Silverarmor commented Apr 11, 2021

xnetcat commented Apr 29, 2021

ghost left a comment

Choose a reason for hiding this comment

Silverarmor commented May 11, 2021

AZMCode commented May 18, 2021

ghost commented Mar 29, 2021 •

edited by ghost

AZMCode commented Mar 29, 2021 •

edited

AZMCode commented Mar 29, 2021 •

edited

ghost commented Mar 30, 2021 •

edited by ghost

AZMCode commented Mar 30, 2021 •

edited

xnetcat commented Mar 30, 2021 •

edited

AZMCode commented Mar 30, 2021 •

edited

xnetcat Apr 1, 2021 •

edited