Download error: KeyError: 'playNavigationEndpoint' #1038

itmax78 · 2020-12-16T06:44:35Z

In the evening I started downloading two playlists, they downloaded normally. In the morning I start downloading the next playlist and get an error:

Fetching Playlist...                                                                                                                                                                                                                  
Traceback (most recent call last):
File "/root/envspotdl/bin/spotdl", line 8, in <module>
sys.exit(console_entry_point())
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/__main__.py", line 114, in console_entry_point
 songObjList = get_playlist_tracks(request)
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
 song = SongObj.from_url(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/songObj.py", line 66, in from_url
  youtubeLink = SongObj.searchProvider(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 478, in search_and_get_best_match
 results = search_and_order_ytm_results(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 366, in search_and_order_ytm_results
  results = __query_and_simplify(songSearchStr)
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 176, in __query_and_simplify
  linkBlock = contents['musicResponsiveListItemRenderer'] \
KeyError: 'playNavigationEndpoint'

I installed spotdl like this:

pip3 install spotdl
pip3 uninstall -y pytube3
pip3 install git+https://github.com/nficano/pytube

other installation options did not solve the problem. When installing from master branch, I get another error:
TypeError: cannot unpack non-iterable NoneType object

The text was updated successfully, but these errors were encountered:

issue-label-bot · 2020-12-16T06:44:37Z

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.98. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

itmax78 · 2020-12-16T07:41:17Z

A little more information, maybe it will be useful. Tried redirecting traffic through Tor to bypass YouTube blocking, it didn't work. I get similar errors on three different servers, on different subnets. I'm using Ubuntu 18.04 and CentOS 7

Silverarmor · 2020-12-16T10:51:22Z

2nd error (from master branch install) Appears to be duplicate of #1025 further down.

Try the following:
Try the following:

pip install pip-autoremove
pip-autoremove spotdl
pip install https://codeload.github.com/spotDL/spotify-downloader/zip/master
pip uninstall pytube
pip install git+https://github.com/nficano/pytube
spotdl $trackUrl

itmax78 · 2020-12-16T11:14:03Z

Now I get both errors at once, a complete list of what I did:

> python3.8 -m venv master
> source master/bin/activate
> pip install pip-autoremove
> pip-autoremove spotdl
> pip install https://codeload.github.com/spotDL/spotify-downloader/zip/master
> pip uninstall pytube
> pip install git+https://github.com/nficano/pytube
> spotdl https://open.spotify.com/playlist/3lsjGhv5VTN0mjnmDP3HI9?si=1Y6rH7qCRJq8k1H7RSpJOw
> 
> Fetching Playlist...                                                                                                                                                                                                                  
> Traceback (most recent call last):
>   File "/root/master/bin/spotdl", line 33, in <module>
>     sys.exit(load_entry_point('spotdl==3.1.4', 'console_scripts', 'spotdl')())
>   File "/root/master/lib/python3.8/site-packages/spotdl/__main__.py", line 111, in console_entry_point
>     songObjList = get_playlist_tracks(request)
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
>     song = SongObj.from_url(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/songObj.py", line 66, in from_url
>     youtubeLink = SongObj.searchProvider(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 478, in search_and_get_best_match
>     results = search_and_order_ytm_results(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 366, in search_and_order_ytm_results
>     results = __query_and_simplify(songSearchStr)
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 176, in __query_and_simplify
>     linkBlock = contents['musicResponsiveListItemRenderer'] \
> KeyError: 'playNavigationEndpoint'
> Exception ignored in: <function tqdm.__del__ at 0x7f502d906820>
> Traceback (most recent call last):
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1124, in __del__
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1337, in close
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1516, in display
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1127, in __repr__
>   File "/root/master/lib/python3.8/site-packages/spotdl/download/progressHandlers.py", line 23, in format_dict
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1477, in format_dict
> TypeError: cannot unpack non-iterable NoneType object

tytydraco · 2020-12-17T01:34:05Z

I'm getting this error as well. @Silverarmor's solution did not help, sadly.

KeyError: 'playNavigationEndpoint'

s1as3r · 2020-12-17T08:10:18Z

While Debugging, here's what I found:

When searching for a song, spotdl gathers all the results that the youtube music api provides.
If one of the songs in the results is not available in a country, 'playNavigationEndpoint' key would not be present in it's result, hence this error.
In this specific case, while searching for the song 'KUURO - Don't Stop', youtube music api also returns a result for a song 'KUURO - Rapture' which isn't available in some countries and it causes the error.
- If you try to download the song 'KUURO - Don't Stop' you'd get the same error.
- Downloading any other song won't give this error.

The Fix:

I will be opening a PR, till it gets merged, here's how you can fix it manually.
- Add a try, except block in site-packages/spotdl/search/provider.py line 176 like this:

try:
    linkBlock = contents['musicResponsiveListItemRenderer'] \
        ['overlay'] \
            ['musicItemThumbnailOverlayRenderer'] \
                ['content'] \
                    ['musicPlayButtonRenderer'] \
                        ['playNavigationEndpoint']
except KeyError:
    continue

You can also try and remove "KUURO - Don't Stop" from this playlist and try again.

Please close this issue, if this fixes it.

tytydraco · 2020-12-17T08:15:13Z

Please close this issue, if this fixes it.

It could be a good idea to put an error message there too; not sure how the rest of the program handles errors. Also, it could be a bad idea to wrap that whole thing. If there's an error in the future regarding any of those keys, we would never be able to know. Maybe we can do something like this:

_linkBlock = contents['musicResponsiveListItemRenderer'] \
        ['overlay'] \
            ['musicItemThumbnailOverlayRenderer'] \
                ['content'] \
                    ['musicPlayButtonRenderer']
try:
    linkBlock = _linkBlock ['playNavigationEndpoint']
except KeyError:
    continue

s1as3r · 2020-12-17T08:20:43Z

Also, it could be a bad idea to wrap that whole thing

Yup, You're right. Will make the suggested changes.

lvlzyro · 2020-12-17T10:26:39Z

Here is what I got when I add that block:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.spotify.com', port=443): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 114, in console_entry_point
    songObjList = get_playlist_tracks(request)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
    song = SongObj.from_url(
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/search/songObj.py", line 41, in from_url
    rawTrackMeta = spotifyClient.track(spotifyURL)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 338, in track
    return self._get("tracks/" + trid)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 291, in _get
    return self._internal_call("GET", url, payload, kwargs)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 240, in _internal_call
    response = self._session.request(
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.spotify.com', port=443): Read timed out. (read timeout=5)

tzarad · 2020-12-17T13:12:23Z

For me all youtubelink are None

itmax78 · 2020-12-17T13:24:53Z

Corrected the code as suggested above. When downloading any individual songs, I get:

spotdl https://open.spotify.com/track/2iyVwlrSE1jJOrohg5tv0z?si=jPNPRwi6T3W-LKkBq6e3PA
Fetching Song...
Skipping Don't Stop (https://open.spotify.com/track/2iyVwlrSE1jJOrohg5tv0z?si=jPNPRwi6T3W-LKkBq6e3PA) as no match could be found on youtube
0%| |ETA: ?, ~min/song

I launch a small playlist:

spotdl https://open.spotify.com/playlist/37i9dQZF1DWVi45nh2EuPP?si=phjVN11FSCOaQNrwXQ4pLg
Fetching Playlist...
Traceback (most recent call last):
File "/root/master/bin/spotdl", line 8, in
sys.exit(console_entry_point())
File "/root/master/lib/python3.8/site-packages/spotdl/main.py", line 116, in console_entry_point
downloader.download_multiple_songs(songObjList)
File "/root/master/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs
self.downloadTracker.load_song_list(songObjList)
File "", line 2, in load_song_list
File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
raise convert_to_error(kind, result)
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

I launch the playlist from my first message:

https://open.spotify.com/playlist/3lsjGhv5VTN0mjnmDP3HI9?si=1Y6rH7qCRJq8k1H7RSpJOw
Fetching Playlist...
and so on for more than two hours :(

Strangely enough, I made another virtual environment, fixed only the code and launched the playlist in it that I managed to download for the first time, now the process is also frozen.

s1as3r · 2020-12-17T13:32:53Z

For me all youtubelink are None

Corrected the code as suggested above.
When downloading any individual songs, I get:

That's a different issue altogether.
See #1039

s1as3r · 2020-12-17T13:35:07Z

Here is what I got when I add that block:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

.....

This seems to be an issue with requests. Retry downloading a few times.

lvlzyro · 2020-12-17T14:03:32Z

This seems to be an issue with requests. Retry downloading a few times.

Here is what I get which is same with itmax78's one:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 116, in console_entry_point
    downloader.download_multiple_songs(songObjList)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs
    self.downloadTracker.load_song_list(songObjList)
  File "<string>", line 2, in load_song_list
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

s1as3r · 2020-12-17T14:11:47Z

This seems to be an issue with requests. Retry downloading a few times.

Here is what I get which is same with itmax78's one:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 116, in console_entry_point
    downloader.download_multiple_songs(songObjList)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs
    self.downloadTracker.load_song_list(songObjList)
  File "<string>", line 2, in load_song_list
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

See #968 and #929

lvlzyro · 2020-12-17T14:17:39Z

See #968 and #929

YouTube Music is available in my country and I tried to install and uninstall pytube but it did not help.

kommersch · 2020-12-17T16:09:56Z

I have the same issue as lvlzyro. And yes, Youtube Music is available in my country as well.

ghost · 2020-12-19T05:54:14Z

A fix for this is in the pipeline. Please wait a few days and install from master.

Neltherion · 2020-12-31T08:49:43Z

This issue hasn't been fixed yet. I'm getting:

Fetching Song...
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\spotdl.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\__main__.py", line 94, in console_entry_point
    song = SongObj.from_url(request)
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\songObj.py", line 70, in from_url
    duration
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 480, in search_and_get_best_match
    songAlbumName, songDuration
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 366, in search_and_order_ytm_results
    results = __query_and_simplify(songSearchStr)
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 181, in __query_and_simplify
    ['playNavigationEndpoint']
KeyError: 'playNavigationEndpoint'
  0%|          |ETA: ?, ~min/song

YouTube Music is available along YouTube.

EDIT: This made it work again.

@s1as3r

Author: @s1as3r * Fix Issue #1038 Fixes [Issue #1038](#1038) - When searching for a song, spotdl gathers all the results that the youtube music api provides. - If one of the songs in the results is not available in a country, 'playNavigationEndpoint' key would not be present in it's result, hence this error. * Update provider.py Made Changes as suggested by @tytydraco * Update provider.py - Used `.get()` method instead of a try-except block.

issue-label-bot bot added the Bug Unexpected problem or unintended behavior that needs to be fixed label Dec 16, 2020

Silverarmor mentioned this issue Dec 16, 2020

Can't Download My Playlist: KeyError: 'playNavigationEndpoint' #1042

Closed

s1as3r mentioned this issue Dec 17, 2020

Fix Issue #1038 #1043

Merged

ghost closed this as completed Dec 19, 2020

Silverarmor mentioned this issue Dec 22, 2020

Issues on downloading artist from the latest github ver. #1052

Closed

This was referenced Jan 2, 2021

Can't download a playlist. Console hangs then gives a TypeError: cannot unpack non-iterable NoneType object #1067

Closed

Playlist download fails: KeyError: 'playNavigationEndpoint' #1071

Closed

pithuene mentioned this issue Jan 5, 2021

Playlist Download error - TypeError: remove #1044

Closed

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download error: KeyError: 'playNavigationEndpoint' #1038

Download error: KeyError: 'playNavigationEndpoint' #1038

itmax78 commented Dec 16, 2020 •

edited by Silverarmor

issue-label-bot bot commented Dec 16, 2020

itmax78 commented Dec 16, 2020

Silverarmor commented Dec 16, 2020

itmax78 commented Dec 16, 2020 •

edited by Silverarmor

tytydraco commented Dec 17, 2020

s1as3r commented Dec 17, 2020 •

edited

tytydraco commented Dec 17, 2020

s1as3r commented Dec 17, 2020

lvlzyro commented Dec 17, 2020

tzarad commented Dec 17, 2020

itmax78 commented Dec 17, 2020

s1as3r commented Dec 17, 2020 •

edited

s1as3r commented Dec 17, 2020 •

edited

lvlzyro commented Dec 17, 2020

s1as3r commented Dec 17, 2020

lvlzyro commented Dec 17, 2020 •

edited

kommersch commented Dec 17, 2020

ghost commented Dec 19, 2020

Neltherion commented Dec 31, 2020 •

edited

Download error: KeyError: 'playNavigationEndpoint' #1038

Download error: KeyError: 'playNavigationEndpoint' #1038

Comments

itmax78 commented Dec 16, 2020 • edited by Silverarmor

issue-label-bot bot commented Dec 16, 2020

itmax78 commented Dec 16, 2020

Silverarmor commented Dec 16, 2020

itmax78 commented Dec 16, 2020 • edited by Silverarmor

tytydraco commented Dec 17, 2020

s1as3r commented Dec 17, 2020 • edited

tytydraco commented Dec 17, 2020

s1as3r commented Dec 17, 2020

lvlzyro commented Dec 17, 2020

tzarad commented Dec 17, 2020

itmax78 commented Dec 17, 2020

s1as3r commented Dec 17, 2020 • edited

s1as3r commented Dec 17, 2020 • edited

lvlzyro commented Dec 17, 2020

s1as3r commented Dec 17, 2020

lvlzyro commented Dec 17, 2020 • edited

kommersch commented Dec 17, 2020

ghost commented Dec 19, 2020

Neltherion commented Dec 31, 2020 • edited

itmax78 commented Dec 16, 2020 •

edited by Silverarmor

itmax78 commented Dec 16, 2020 •

edited by Silverarmor

s1as3r commented Dec 17, 2020 •

edited

s1as3r commented Dec 17, 2020 •

edited

s1as3r commented Dec 17, 2020 •

edited

lvlzyro commented Dec 17, 2020 •

edited

Neltherion commented Dec 31, 2020 •

edited