Skip to content

Conversation

@hanouticelina
Copy link
Contributor

This PR updates paginate() to use the default values for http_backoff instead of explicit max_retries=20, retry_on_status_codes=429. The default values are max_retries=5 and retry_on_status_codes=(429, 500, 502, 503, 504).
Note that with #3577, The retry mechanism now includes smart 429 handling that parses rate limit headers, it basically extracts the number of seconds until the rate limit resets from ratelimit header and waits exactly the time needed before retrying. This will be shipped in the next release!


cc @Weyaxi for visibility since you added the original retry mechanism in pagination in #2970 🤗 btw, if you're hitting rate limits frequently, you can get much higher limits with a PRO subscription (12K resolver calls and 2.5K API calls per minute instead of 5k and 1k respectively), you can find more info in the docs here

@hanouticelina hanouticelina requested a review from Wauplin December 2, 2025 14:03
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Wauplin
Copy link
Contributor

Wauplin commented Dec 2, 2025

Be careful that a mock test is failing due to this PR:

FAILED ../tests/test_utils_pagination.py::TestPagination::test_mocked_paginate - AssertionError: assert [call('GET', ...4034516176'>)] == [call('GET', ...4034516176'>)]
  
  At index 0 diff: call('GET', 'url_p2', headers=<Mock id='140404034516176'>) != call('GET', 'url_p2', max_retries=20, retry_on_status_codes=429, headers=<Mock id='140404034516176'>)
  
  Full diff:
  + [call('GET', 'url_p2', headers=<Mock id='140404034516176'>),
  +  call('GET', 'url_p3', headers=<Mock id='140404034516176'>)]
  - [
  -     call('GET', 'url_p2', max_retries=20, retry_on_status_codes=429, headers=<Mock id='140404034516176'>),
  -     call('GET', 'url_p3', max_retries=20, retry_on_status_codes=429, headers=<Mock id='140404034516176'>),
  - ]

(easy to miss it now that we have the oauth issues in CI)

@hanouticelina hanouticelina merged commit b71441f into main Dec 3, 2025
22 checks passed
@hanouticelina hanouticelina deleted the default-retry-values-for-paginate branch December 3, 2025 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants