New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PoolManager retries argument is not passed down to requests #2475
Comments
Thank you! it's a pity that doesn't help for requests, it doesn't seem to be using the poolmanager urlopen method. |
with a timeout error, retry is working fine
|
Indeed, for timeouts it goes through a different codepath in It default on urllib3/src/urllib3/connectionpool.py Lines 642 to 643 in fac7db1
So this only affects other type of errors like redirects (in |
…dling Fixes issue adbar#450 After setting `MAX_REDIRECTS` to 5, I could fetch the original URL from the issue: `trafilatura -u https://www.hydrogeninsight.com/production/breaking-us-reveals-the-seven-regional-hydrogen-hubs-to-receive-7bn-of-government-funding/2-1-1534596` I also fixed this old issue: adbar#128 The underlying urllib3 bug has not been fixed: urllib3/urllib3#2475 I had to pass the retry strategy to the actual request method: it doesn't propagate from the pool maanger
…dling (#461) * introduce `MAX_REDIRECTS` config setting and fix urllib3 redirect handling Fixes issue #450 After setting `MAX_REDIRECTS` to 5, I could fetch the original URL from the issue: `trafilatura -u https://www.hydrogeninsight.com/production/breaking-us-reveals-the-seven-regional-hydrogen-hubs-to-receive-7bn-of-government-funding/2-1-1534596` I also fixed this old issue: #128 The underlying urllib3 bug has not been fixed: urllib3/urllib3#2475 I had to pass the retry strategy to the actual request method: it doesn't propagate from the pool maanger * use httpbin everywhere instead of httpbun * restore MAX_REDIRECTS defaut config value to 2 * reset global objects after test_fetch to avoid side-effects * pin lxml to < 5 * test no_ssl True & False to fix coverage * move _reset_global_objects() to the test file * rewrite assignment in multiple lines for readability --------- Co-authored-by: Adrien Barbaresi <adbar@users.noreply.github.com>
Subject
The documentation states that
retries
argument can be passed to aPoolManager
to be used as default in pool requests:https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#utilities
It seems that
retries
is actually ignored in the pool requestsEnvironment
Also reproduced in development mode in
urllib3
main
branchSteps to Reproduce
Expected Behavior
I would expect the first request to raise
MaxRetryError
Actual Behavior
The first request actually use a default retry strategy instead of the pool's strategy
This fixes the issue. Does that seem like the right way to fix this? I can write a PR and a unit test.
Or maybe
connection_pool_kw
arguments should be merged in a more generic place like_merge_pool_kwargs()
.The text was updated successfully, but these errors were encountered: