Skip to content

Conversation

@hornlukas
Copy link

Description

Closes #104

This PR addresses the broken Semantic Scholar API link and improves the handling of unauthenticated requests to mitigate 429 Too Many Requests errors.

Changes

  • Fixed API Link: Updated the URL to the correct endpoint.
  • Retry Strategy: Implemented a retry mechanism using urllib3 to handle transient failures (3 retries, 1s backoff).
  • Tests: Adapted existing tests to reflect the API updates.

Context & Results

The current global rate limit for unauthenticated requests is 1000 requests per second. Since this is a shared global limit, requests are often rejected with 429s.

I tested this locally:

  • The retry strategy significantly improves success rates.
  • However, due to the global limit, occasional failures may still occur.
  • Increasing retries/backoff further did not yield better results in my tests.

Recommendation

For consistent reliability, it is highly recommended to use an API key for authenticated requests (1 req/s) rather than relying on the unauthenticated global pool.

Discussion

I am open to suggestions regarding the retry configuration if others find more optimal settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deprecated Semantic Scholar API endpoint

1 participant