Spotipy doesnt doesnt respond but no error

[nickhir]

Hello,

I have been using spotipy for the last year and never ran into any problems.
Today, I realized that something is not working anymore and looked into it.
I can initializing the sp.Spotify instance as usual without any problems, but if I then call a function (for example spotify.me() or spotify.devices(), it simply hangs and doesnt return anything.

This is the code I used for the last months:

with open("CONFIG.json", "r") as f:
    config = json.load(f)

client_id = config["client_id"]
client_secret = config["client_secret"]
device_name = config["device_name"]
redirect_uri = config["redirect_uri"]
username = config["username"]
scope = config["scope"]

# authenticate account
auth_manager = SpotifyOAuth(
    client_id=client_id,
    client_secret=client_secret,
    redirect_uri=redirect_uri,
    scope=scope,
    username=username
)
spotify = sp.Spotify(auth_manager=auth_manager)

I checked my Spotify dashboard, and noticed, that noticed that the number of daily requests dropped to zero just when December started:

Do you have any idea what might cause the issue? I already tried to regenerate a new client secret, but it didnt work.

I am using the version 2.21.0 on a raspberry pi (Raspbian version 10 [Buster])

[JohnWhy]

I believe something occurs that spams the Spotify API and they essentially blacklist your app. I wrote my own requests function to query the API using my existing credentials I used with spotipy and it was giving me a 429 error for "Too Many Requests" even when waiting a long time between search requests.

I made a new Spotify app on the website (new client/secret) and used my custom requests function and it worked perfectly.

Made another Spotify app on the website (new client/secret) and tried plugging those into the existing spotipy application and was still getting an infinite hanging problem. So I think what's happening is spotipy is getting a 429 error and not handling/reporting it. It must be spamming requests at some point causing the issue, notice how you have a bump in API activity for a brief moment, my guess is spotipy unintentionally spammed the API with thousands of requests in a second, and Spotify auto-blocked your application from sending more.

My app had a similar spike (although more pronounced)

[Rsalganik1123]

Hi! I had a similar issue. I found that once a request gets denied, with each additional call you make following this, the wait time grows exponentially. Since I had a lot of calls to make (I'm doing a dataset augmentation task), I added an error catch for time out that looks like this:

try: 
       results = sp.albums(uri)
except SpotifyException as e: 
        if e.http_status == 429:
            print("'retry-after' value:", e.headers['retry-after'])
            retry_value = e.headers['retry-after']
            if int(e.headers['retry-after']) > 60: 
                print("STOP FOR TODAY, retry value too high {}".format(retry_value))
                exit() 
                

Normally, it makes me wait like 8 hours and then I can come back.

Also, a few tips I have for making it less likely to time out:

• if you lower the size of the batches, it's less likely to time out. So for example calling sp.albums(batch) where batch size is 10 worked well for me without timeouts.
• if you add a few second sleep before each call, it helps.
• different calls have different timeout times, so tracks seems particularly sensitive while music_features is less sensitive.

Not sure whether you setting allows for these changes, but in the case that it does - hope it helps :)

[stephanebruckert]

spotipy should already be waiting/sleeping when the Spotify API response requests to wait (retry-after value). Maybe there is a scenario where this does not apply.

Would anyone have a Minimal reproducible example that would allow us to reproduce this rate limiting? This would need to work with a new Spotify app.

Feel free to include logs about the current retry-after value, maybe with inspiration from #766 (comment)?

[CalColistra]

Hey, I was just curious if Rsalgnakit1223's solution with the try catch blocks would work even if my spotipy api doesn't throw an error. Mine just gets stuck in sleep_for_rety() and hangs forever. Would the try catch block work in my case if an error is never thrown to trigger the catch?

[stephanebruckert]

@CalColistra, sorry Rsalgnakit1223's solution is incomplete, here is a better example which will raise an exception #766 (comment)

[CalColistra]

@stephanebruckert Hey, I am sorry if I missed anything obvious in any of the issues made for this sleep_for_retry() problem with the spotipy api. But are there any working solutions that we know of? I see you are keeping up with a lot of the discussions about this and really appreciate you taking the time to do so.

[stephanebruckert]

• to print how long to wait, this worked for me GET /me/player + 429 Rate Limit exceeded = blocking calls #766 (comment)
• to figure out why such long waiting times apply, it would be good to have an example that we can reproduce. Please share one, with or without the above solution

[reflexiveprophecy]

It gives me a 22 hour timeout the first time of bumping into 429 in this fresh run after 23k search requests, does the API assign rate limit based on IP too? I read on previous issues that the timeout would go up exponentially every time you hit the limit. How does the timeout get determined for search under Web API?

[main][INFO ]processing the 23210th title
[main][ERROR ]Rate limit hit, please try after:81654 The program now needs to rest for 81654 seconds

url = 'https://api.spotify.com/v1/search'
headers = {
    'Authorization': f'Bearer {self.access_token}'
}
payload = {'q': query, 'type': 'artist', 'limit': 1}

response = requests.get(
    url, 
    headers=headers, 
    params=payload
)

if response.status_code == 429:
    retry_after = int(response.headers['retry-after'])
    raise SpotifyRateLimitError(
        f"Rate limit hit, please try after:{retry_after}",
        retry_after = retry_after
   )

I thought I could catch the retry_after and make the program sleep for the value, but 22 hours timeout renders that not very useful...

except SpotifyRateLimitError as error:
    logger.error(
        f"{error} "
        f"The program now needs to rest for {error.retry_after} seconds"
    )
    time.sleep(error.retry_after)

[stephanebruckert]

@reflexiveprophecy can you please provide a Minimal reproducible example? Something that we could just run once to reproduce the problem with a new Spotify app?

The code above doesn't look complete as it doesn't include any loop.

I thought I could catch the retry_after and make the program sleep for the value, but 22 hours timeout renders that not very useful...

I agree but the app is supposed to start sleeping from the very first 429s, not after 23k requests and 22 hours wait. Could your code show that as well?

---

My assumption is that at one point, it's not "respecting" the given retry-after value. For example retry-after requires a 10 seconds wait, but another request is started before that, which leads to a violation and therefore exponential wait times as a "punishment". See https://stackoverflow.com/a/70311787/1515819

I imagine examples of such violation could be occurring when:

• having 2 programs using the same Spotify app/credentials, or the same app being ran concurrently, or the same app using multi-threads. In this case one app wouldn't wait for the other's retry-after value as it's not aware of it,
• a program is correctly "sleeping" due to a retry-after value of say 10s, but the program is stopped and re-started after 5s, which would lead to a violation since it should have waited 5 more seconds.

[reflexiveprophecy]

@stephanebruckert Okay, please see the following self-contained example. You shouldn't have to do much else to run this example besides pip install requests and embedding CLIENT_ID and CLIENT_SECRET as env variables. This example would produce 100k artist names in a list for the search request to run through in a loop. I tested the script today with a new set of credentials and was able to run through 24.9k search requests this time without a 429 error and it gives me a 23 hour timeout this time after bumping into the error this time...I don't see other 429 messages as I searched through the entire log. And I don't think it's particularly blocking my ip address either as I was able to run through again by switching to new "non-timeout" account credentials. Also I didn't use Spotipy client as the client silences 429 messages for some reasons as previous issues also noted. Looking forward to some solutions :)

[__main__][INFO  ]processing the 24931th artist
[__main__][ERROR ]Rate limit hit, please try after:83276 The program now needs to rest for 83276 seconds

# python 3.8.10
# spotify_selfcontained_script.py

import re
import logging
import collections
import time
import json
import itertools
import os
import sys
import requests
import base64
from dataclasses import dataclass, field
from typing import List, Dict, NoReturn, Tuple


class SpotifyRateLimitError(Exception):
    """
    Catch Spotify rate limit error
    with 429
    """
    def __init__(
        self, 
        message: str,
        retry_after: int
        ):
        super().__init__(message)
        self.message = message
        self.retry_after = retry_after
        
        
def get_spotify_token(
    client_id: str, 
    client_secret: str
    ) -> str:
    """
    Generate access token
    
    Params:
    ----
    client_id: str
        the client id of api account
    client_secret: str
        the client secret of api account
        
    Return:
    ----
    generate token in str
    """
    url = 'https://accounts.spotify.com/api/token'
    client_credentials = f'{client_id}:{client_secret}'
        
    encoded_credentials = base64.b64encode(
        client_credentials.encode()
    ).decode()
    
    headers = {'Authorization': f'Basic {encoded_credentials}'}
    payload = {'grant_type': 'client_credentials'}

    response = requests.post(url, headers=headers, data=payload)
    response_json = response.json()

    if response.status_code != 200:
        raise Exception(
            f'Error: {response_json["error"]}, '
            '{response_json["error_description"]}'
        )
        
    return response_json['access_token']


def search_artist_genre(
    query: str, 
    token: str
    ) -> Tuple[str, str]:
    """
    Search genre of an artist
    
    Params:
    ----
    query: str
        the query input str
    token: str
        the generated access token
        
    Return:
    ----
    (artist_name, artist_genres)
    """
    
    url = 'https://api.spotify.com/v1/search'
    headers = {
        'Authorization': f'Bearer {token}'
    }
    payload = {'q': query, 'type': 'artist', 'limit': 1}

    response = requests.get(
        url, 
        headers=headers, 
        params=payload
    )

    if response.status_code == 429:
        retry_after = int(response.headers['retry-after'])
        raise SpotifyRateLimitError(
            f"Rate limit hit, please try after:{retry_after}",
            retry_after = retry_after
        )

    response_json = response.json()
    artists = response_json['artists']['items']

    if not artists:
        print(f"No artist found for query '{query}'.")
        return (None, None)

    artist = artists[0]
    artist_name = artist['name']

    if artist['genres']:
        artist_genres = ','.join(artist['genres'])
    else:
        return (artist_name, None)
    
    print(f"Artist: {artist['name']}")
    print(f"Genres: {artist_genres}")

    return (artist_name, artist_genres)

    
if __name__ == "__main__":
    
    token = get_spotify_token(
        client_id = os.environ["CLIENT_ID"], 
        client_secret = os.environ["CLIENT_SECRET"]
    )
    
    artist_sample = [
        "taylor swift",
        "bruno mars",
        "alicia keys",
        "eminem",
        "john legend",
        "kanye west",
        "bob dylan",
        "lil wayne",
        "katy perry",
        "kurt cobain"
    ]
    
    artist_duplicate_sample = artist_sample * 10000
    spotify_artist_genre_list = []
    
    for index, artist in enumerate(artist_duplicate_sample):
        print(f"processing the {index}th artist")
        
        try: 
            artist_name, artist_genres = search_artist_genre(
                artist, token
            )
        except SpotifyRateLimitError as error:
            print(
                f"{error} "
                f"The program now needs to rest "
                f"for {error.retry_after} seconds"
            )
            time.sleep(error.retry_after)
    

[spotifyr1]

Hi! I had a similar issue. I found that once a request gets denied, with each additional call you make following this, the wait time grows exponentially. Since I had a lot of calls to make (I'm doing a dataset augmentation task), I added an error catch for time out that looks like this:

try: 
       results = sp.albums(uri)
except SpotifyException as e: 
        if e.http_status == 429:
            print("'retry-after' value:", e.headers['retry-after'])
            retry_value = e.headers['retry-after']
            if int(e.headers['retry-after']) > 60: 
                print("STOP FOR TODAY, retry value too high {}".format(retry_value))
                exit() 
                

Normally, it makes me wait like 8 hours and then I can come back.

Also, a few tips I have for making it less likely to time out:

* if you lower the size of the batches, it's less likely to time out. So for example calling sp.albums(batch) where batch size is 10 worked well for me without timeouts.

* if you add a few second sleep before each call, it helps.

* different calls have different timeout times, so tracks seems particularly sensitive while music_features is less sensitive.

Not sure whether you setting allows for these changes, but in the case that it does - hope it helps :)

Thanks for posting this response, it was very helpful! I'm more of a beginner, but I'm trying to collect some tracks metadata. I'm getting a 429 response, which I expected (I'm making many requests to the API), but the header comes back empty. So when I use if int(e.headers['retry-after']) > 60 to exit the loop, it doesn't work because there is no 'retry-after' key in the headers. Do you know why that could happen? I'm printing out the values returned by the exception, and they all look fine except the header which is empty {}.

[reflexiveprophecy]

@spotifyr1 Hey, regarding this question "So when I use if int(e.headers['retry-after']) > 60 to exit the loop, it doesn't work because there is no 'retry-after' key in the headers. Do you know why that could happen? I'm printing out the values returned by the exception, and they all look fine except the header which is empty {}", I don't think spotipy has this returned. This is one of the reasons why I didn't use spotipy client as the api, I directly called the REST API with the requests module, hence you don't see me importing spotipy and just the requests module. With the requests module, you should be able to catch the retry-after value.

[Drag-3]

@reflexiveprophecy I took your advice, but I have the same issue with the requests module directly (The get command blocks forever even with a timeout applied) So I am unsure as to whether this is a problem with Spotipy.

[reflexiveprophecy]

Hey @Drag-3, got it, happy to test it out if you could share your code, thank you!

[Drag-3]

@reflexiveprophecy Here's my code! Hopefully you can find something I am missing.

            def __init__(self, cid: str, secret: str):
        try:
            self.auth_path = CONFIG_DIR / ".sp_auth_cache"
            self.session = requests.Session()
            retries = Retry(total=5, backoff_factor=0.1, status_forcelist=[429, 500, 502, 503, 504])
            adapter = HTTPAdapter(max_retries=retries)
            self.session.mount('https://', adapter)
            self.session.mount('http://', adapter)

            AUTH_URL = "https://accounts.spotify.com/api/token"
            auth_resp = requests.post(AUTH_URL, {"grant_type": "client_credentials",
                                                 "client_id": cid,
                                                 "client_secret": secret})
            auth_resp_json = auth_resp.json()
            access_token = auth_resp_json['access_token']
            self.session.headers.update({'Authorization': f'Bearer {access_token}'})

            self.cache = diskcache.Cache(directory=str(CONFIG_DIR / "spotifycache"))
            self.cache.expire(60 * 60 * 12)  # Set the cache to expire in 12 hours
            self.semaphores = {
                'search': threading.Semaphore(3),
                'track': threading.Semaphore(3),
                'audio-analysis': threading.Semaphore(2),
                'artists': threading.Semaphore(1)
            }
        except Exception as e:
            logging.exception(e)
            raise e

    def _get_item_base(self, endpoint: str, value):
        with self.semaphores[endpoint]:
            response = self.session.get(f"https://api.spotify.com/v1/{endpoint}/{value}", timeout=20)  # This line blocks according to  the debugger. 

            if response.status_code == 429:  # Added these trying to debug before I noticed the blocking problem
                retry_after = int(response.headers.get('retry-after', '1'))
                logging.warning(f" {endpoint} Rate limited. Waiting for {retry_after} seconds before retrying.")
                time.sleep(retry_after + random.randint(3, 7))
            elif response.status_code != 200:
                response.raise_for_status()
            return response.json()

According to curl, I should be getting an Error 429, but the code blocks instead.

[dnoegel]

I run into the same problem today with this reduced example with spotipy 2.23.0:

import spotipy
from spotipy.oauth2 import SpotifyOAuth

auth_manager=SpotifyOAuth(
    client_id="CLIENT ID",
    client_secret="CLIENT SECRET",
    redirect_uri="http://localhost:1234/callback",
    scope="user-read-playback-state user-library-read streaming app-remote-control"
)
spotify = spotipy.Spotify(auth_manager=auth_manager)

print("This prints")
spotify.current_playback()
print("This line is never printed")

My mistake was: I was in headless mode, and the "open the browser" part obviously can not work there. In the example section of spotify I found that you need to set open_browser=False as a param in SpotifyOAuth like this:

import spotipy
from spotipy.oauth2 import SpotifyOAuth


auth_manager=SpotifyOAuth(
    client_id="CLIENT ID",
    client_secret="CLIENT SECRET",
    redirect_uri="http://localhost:1234/callback",
    scope="user-read-playback-state user-library-read streaming app-remote-control",
    open_browser=False
)
spotify = spotipy.Spotify(auth_manager=auth_manager)

print("This prints")
spotify.current_playback()
print("Now this line is printed as well")

With this setup, the oauth flow will be handled in terminal. Hope this is helpful for some 👍

[eleqtrizit]

I found the cause of the lack of response, and this is mentioned in urllib3/urllib3#2911. In module urllib3/connectionpool.py, line #943 is code

            retries.sleep(response)

This is honoring the Retry-After that Spotify's API is sending back. And if you're like me, who somehow got a long retry time (mine is currently 20k+ seconds), it is going to hang.

A potential fix is simply doing the following...

sp = spotipy.Spotify(
    retries=0,
    ...
)

..so Spotify doesn't try the retry. But if you do this, it just raises an Exception and doesn't report back what the Retry-After value was. This is where the improvement can be made, and perhaps build Spotipy's own retry feature.