GH-40279: [C++] Reduce S3Client initialization time #40299

pitrou · 2024-02-29T18:12:29Z

Rationale for this change

By default, S3Client instantiation is extremely slow (around 1ms for every instance). Investigation led to the conclusion that most of this time was spent inside the AWS SDK, parsing a hardcoded piece of JSON data when instantiating a AWS rule engine.

Python benchmarks show this repeated initiatlization cost:

>>> from pyarrow.fs import S3FileSystem

>>> %time s = S3FileSystem()
CPU times: user 21.1 ms, sys: 0 ns, total: 21.1 ms
Wall time: 20.9 ms
>>> %time s = S3FileSystem()
CPU times: user 2.37 ms, sys: 0 ns, total: 2.37 ms
Wall time: 2.18 ms
>>> %time s = S3FileSystem()
CPU times: user 2.42 ms, sys: 0 ns, total: 2.42 ms
Wall time: 2.23 ms

>>> %timeit s = S3FileSystem()
1.28 ms ± 4.03 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
>>> %timeit s = S3FileSystem()
1.28 ms ± 2.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
>>> %timeit s = S3FileSystem(anonymous=True)
1.26 ms ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

What changes are included in this PR?

Instead of letting the AWS SDK create a new S3EndpointProvider for each S3Client, arrange to only create a single S3EndpointProvider per set of endpoint configuration options. This lets the 1ms instantiation cost be paid only when a new set of endpoint configuration options is given.

Python benchmarks show the initialization cost has become a one-time cost:

>>> from pyarrow.fs import S3FileSystem

>>> %time s = S3FileSystem()
CPU times: user 20 ms, sys: 0 ns, total: 20 ms
Wall time: 19.8 ms
>>> %time s = S3FileSystem()
CPU times: user 404 µs, sys: 49 µs, total: 453 µs
Wall time: 266 µs
>>> %time s = S3FileSystem()
CPU times: user 361 µs, sys: 42 µs, total: 403 µs
Wall time: 249 µs

>>> %timeit s = S3FileSystem()
50.4 µs ± 227 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
>>> %timeit s = S3FileSystem(anonymous=True)
33.5 µs ± 306 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Are these changes tested?

By existing tests.

Are there any user-facing changes?

No.

GitHub Issue: [Python/C++] S3FileSystem slow to deserialize due to AWS rule engine JSON parsing #40279

pitrou · 2024-02-29T18:12:47Z

@github-actions crossbow submit -g cpp -g python -g wheel

pitrou · 2024-02-29T19:24:58Z

@github-actions crossbow submit -g cpp -g wheel

pitrou · 2024-02-29T19:59:33Z

Oh, it looks like the RTools 40 build is using a very old AWS SDK version (1.7.365):

2024-02-29T19:29:45.9968248Z ==> Making package: mingw-w64-arrow 15.0.0.9000-8000 (Thu, Feb 29, 2024  7:29:45 PM)
2024-02-29T19:29:46.0104530Z ==> Checking runtime dependencies...
2024-02-29T19:29:46.0933579Z ==> Installing missing dependencies...
2024-02-29T19:29:46.1681983Z resolving dependencies...
2024-02-29T19:29:46.1816658Z looking for conflicting packages...
2024-02-29T19:29:46.1842088Z 
2024-02-29T19:29:46.1846702Z Packages (12) mingw-w64-ucrt-x86_64-boost-1.67.0-9002  mingw-w64-ucrt-x86_64-libssh2-1.11.0-9801  mingw-w64-ucrt-x86_64-nghttp2-1.51.0-1  mingw-w64-ucrt-x86_64-openssl-3.1.1-9800  mingw-w64-ucrt-x86_64-aws-sdk-cpp-1.7.365-1  mingw-w64-ucrt-x86_64-brotli-1.0.9-4  mingw-w64-ucrt-x86_64-curl-8.1.2-9000  mingw-w64-ucrt-x86_64-libutf8proc-2.4.0-2  mingw-w64-ucrt-x86_64-lz4-1.8.2-1  mingw-w64-ucrt-x86_64-re2-20200801-1  mingw-w64-ucrt-x86_64-snappy-1.1.7-2  mingw-w64-ucrt-x86_64-thrift-0.13.0-1

Do we know why that is @paleolimbot @jonkeane @assignUser ?

pitrou · 2024-02-29T20:05:25Z

@github-actions crossbow submit -g cpp -g wheel

pitrou · 2024-02-29T21:25:40Z

@github-actions crossbow submit -g cpp -g python -g wheel

kou · 2024-03-01T02:43:01Z

Oh, it looks like the RTools 40 build is using a very old AWS SDK version (1.7.365):

It seems that RTools 40 still uses old MSYS2.

pitrou · 2024-03-14T15:40:05Z

@github-actions crossbow submit -g cpp -g wheel

pitrou · 2024-03-21T17:00:03Z

@github-actions crossbow submit -g cpp -g wheel

github-actions · 2024-03-21T17:02:57Z

Revision: 6b770c0

Submitted crossbow builds: ursacomputing/crossbow @ actions-8946df4af2

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

github-actions · 2024-03-21T18:08:07Z

⚠️ GitHub issue #40279 has been automatically assigned in GitHub to PR creator.

jonkeane · 2024-03-21T19:16:08Z

The R failures are resolved by #40710

kou

+1

kou · 2024-03-21T21:20:19Z

cpp/src/arrow/filesystem/s3fs.cc

+        options_.endpoint_override.empty() || options_.force_virtual_addressing;
+
+#ifdef ARROW_S3_HAS_S3CLIENT_CONFIGURATION
+    client_config_.useVirtualAddressing = use_virtual_addressing;


Do we need client_config_.payloadSigningPolicy = Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never?

No. We just had to pass the argument on older verisons:
https://github.com/aws/aws-sdk-cpp/blob/ffd81252bec92f3e0587e144b07c05f8aed28eb1/aws-cpp-sdk-s3/include/aws/s3/S3Client.h#L460-L465

cpp/src/arrow/filesystem/s3fs.cc

conbench-apache-arrow · 2024-03-26T00:56:32Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 3095344.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 30 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actions bot added Component: C++ awaiting review Awaiting review labels Feb 29, 2024

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the s3clientconfiguration branch from 35c040f to 3ba9dca Compare February 29, 2024 19:24

pitrou mentioned this pull request Feb 29, 2024

[Python/C++] S3FileSystem slow to deserialize due to AWS rule engine JSON parsing #40279

Closed

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the s3clientconfiguration branch from 790f39b to 2be5194 Compare February 29, 2024 21:23

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the s3clientconfiguration branch from 2be5194 to 09dc998 Compare March 14, 2024 15:07

This comment was marked as outdated.

Sign in to view

pitrou mentioned this pull request Mar 14, 2024

S3Client instantiation extremely slow aws/aws-sdk-cpp#2880

Open

pitrou force-pushed the s3clientconfiguration branch from 09dc998 to d51037a Compare March 14, 2024 16:46

pitrou force-pushed the s3clientconfiguration branch from d51037a to 6b770c0 Compare March 21, 2024 16:59

pitrou changed the title ~~EXPERIMENT: [C++] Use S3ClientConfiguration~~ GH-40279: [C++] Reduce S3Client initialization time Mar 21, 2024

pitrou marked this pull request as ready for review March 21, 2024 18:30

pitrou requested review from kou and felipecrv March 21, 2024 18:32

pitrou force-pushed the s3clientconfiguration branch from 6b770c0 to 1ca1d81 Compare March 21, 2024 21:25

kou approved these changes Mar 21, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Mar 21, 2024

felipecrv reviewed Mar 22, 2024

View reviewed changes

cpp/src/arrow/filesystem/s3fs.cc Outdated Show resolved Hide resolved

github-actions bot removed the awaiting merge Awaiting merge label Mar 22, 2024

felipecrv approved these changes Mar 22, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Mar 22, 2024

pitrou added 2 commits March 25, 2024 15:11

EXPERIMENT: [C++] Use S3ClientConfiguration

3492cc3

Address review comments

d862a6a

pitrou force-pushed the s3clientconfiguration branch from 1ca1d81 to d862a6a Compare March 25, 2024 14:24

felipecrv approved these changes Mar 25, 2024

View reviewed changes

pitrou merged commit 3095344 into apache:main Mar 25, 2024
35 of 36 checks passed

pitrou removed the awaiting merge Awaiting merge label Mar 25, 2024

pitrou deleted the s3clientconfiguration branch March 25, 2024 15:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40279: [C++] Reduce S3Client initialization time #40299

GH-40279: [C++] Reduce S3Client initialization time #40299

pitrou commented Feb 29, 2024 •

edited

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024 •

edited

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024

This comment was marked as outdated.

kou commented Mar 1, 2024

pitrou commented Mar 14, 2024

This comment was marked as outdated.

pitrou commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

jonkeane commented Mar 21, 2024

kou left a comment

kou Mar 21, 2024

pitrou Mar 25, 2024

conbench-apache-arrow bot commented Mar 26, 2024

GH-40279: [C++] Reduce S3Client initialization time #40299

GH-40279: [C++] Reduce S3Client initialization time #40299

Conversation

pitrou commented Feb 29, 2024 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024 • edited

pitrou commented Feb 29, 2024

This comment was marked as outdated.

pitrou commented Feb 29, 2024

This comment was marked as outdated.

kou commented Mar 1, 2024

pitrou commented Mar 14, 2024

This comment was marked as outdated.

pitrou commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

github-actions bot commented Mar 21, 2024

jonkeane commented Mar 21, 2024

kou left a comment

Choose a reason for hiding this comment

kou Mar 21, 2024

Choose a reason for hiding this comment

pitrou Mar 25, 2024

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Mar 26, 2024

pitrou commented Feb 29, 2024 •

edited

pitrou commented Feb 29, 2024 •

edited