Add max_pool_connections to client config #1026
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This config value allows you to control the size of the connection pools used in urllib3. We were previously using the hardcoded value of 10 (from requests). As an aside, the implementation for this was much more complicated than necessary. I'm thinking on sending a few follow up PRs that start to clean up some of this technical debt. Closes boto#773 Closes boto#766
Current coverage is 97.57% (diff: 100%)@@ develop #1026 diff @@
==========================================
Files 44 44
Lines 6884 6892 +8
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 6717 6725 +8
Misses 167 167
Partials 0 0
|
| @@ -75,7 +77,26 @@ def convert_to_response_dict(http_response, operation_model): | |||
|
|
|||
|
|
|||
| class PreserveAuthSession(Session): | |||
| """Internal session class used to workaround requests behavior. | |||
|
|
|||
| This class is intended to be used only by the Endpoint class. | |||
There was a problem hiding this comment.
I think the class should be renamed. It is no longer just for preventing the rebuilding of auth. It also is used to customize the adapter and the connection pool. Maybe call it BotocoreRequestsSession?
There was a problem hiding this comment.
I can rename it. Maybe something like BotocoreHTTPSession to disambiguate it from botocore.session.Session?
|
Looks good. Straightforward change. Just had a question about the existing name. |
|
@kyleknap renamed. |
|
Thanks. Looks good. 🚢 |
|
Close the loop... could you link to docs on this and how to use it from boto3? Found it can be passed to boto3 Client or Resource with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This config value allows you to control the size of the
connection pools used in urllib3. We were previously using the
hardcoded value of 10 (from requests).
I ended up not touching the
pool_connectionsbased on thediscussion from #766.
There ends up being quite a noticeable difference whenever
num_threads > max_pool_connection, so I think this willhelp people running requests with a large number of threads.
As an aside, the implementation for this was much more complicated
than necessary. I'm thinking on sending a few follow up PRs that
start to clean up some of this technical debt.
Closes #773
Closes #766
cc @kyleknap @JordonPhillips