-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
### Rationale for this change By default, S3Client instantiation is extremely slow (around 1ms for every instance). Investigation led to the conclusion that most of this time was spent inside the AWS SDK, parsing a hardcoded piece of JSON data when instantiating a AWS rule engine. Python benchmarks show this repeated initiatlization cost: ```python >>> from pyarrow.fs import S3FileSystem >>> %time s = S3FileSystem() CPU times: user 21.1 ms, sys: 0 ns, total: 21.1 ms Wall time: 20.9 ms >>> %time s = S3FileSystem() CPU times: user 2.37 ms, sys: 0 ns, total: 2.37 ms Wall time: 2.18 ms >>> %time s = S3FileSystem() CPU times: user 2.42 ms, sys: 0 ns, total: 2.42 ms Wall time: 2.23 ms >>> %timeit s = S3FileSystem() 1.28 ms ± 4.03 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) >>> %timeit s = S3FileSystem() 1.28 ms ± 2.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) >>> %timeit s = S3FileSystem(anonymous=True) 1.26 ms ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) ``` ### What changes are included in this PR? Instead of letting the AWS SDK create a new S3EndpointProvider for each S3Client, arrange to only create a single S3EndpointProvider per set of endpoint configuration options. This lets the 1ms instantiation cost be paid only when a new set of endpoint configuration options is given. Python benchmarks show the initialization cost has become a one-time cost: ```python >>> from pyarrow.fs import S3FileSystem >>> %time s = S3FileSystem() CPU times: user 20 ms, sys: 0 ns, total: 20 ms Wall time: 19.8 ms >>> %time s = S3FileSystem() CPU times: user 404 µs, sys: 49 µs, total: 453 µs Wall time: 266 µs >>> %time s = S3FileSystem() CPU times: user 361 µs, sys: 42 µs, total: 403 µs Wall time: 249 µs >>> %timeit s = S3FileSystem() 50.4 µs ± 227 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) >>> %timeit s = S3FileSystem(anonymous=True) 33.5 µs ± 306 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) ``` ### Are these changes tested? By existing tests. ### Are there any user-facing changes? No. * GitHub Issue: #40279 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
- Loading branch information
Showing
1 changed file
with
161 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters