[GLUTEN-12303][VL] Support async multipart upload for S3 writes

[ReemaAlzaid]

The PR is ported from facebookincubator/velox#14472. The original author is @weixiuli

What changes were proposed in this pull request?

This PR adds Gluten-side support for multi-threaded asynchronous multipart upload for S3 compatible object storage in the Velox backend.

When enabled, Gluten uses an async S3 write path that uploads multipart upload parts through a shared upload thread pool. The number of in-flight part uploads per file and the upload thread pool size are configurable.

The existing synchronous S3 write path remains the default behavior.

This PR also adds a benchmark for comparing synchronous and asynchronous S3 multipart upload performance.
#12303

How was this patch tested?

Built the Velox backend with S3 enabled and ran the S3 async upload benchmark against MinIO.

Example benchmark command:

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export GLUTEN_S3_BENCH_BUCKET=writedata
export GLUTEN_S3_BENCH_ENDPOINT=http://127.0.0.1:9000
export GLUTEN_S3_BENCH_REGION=us-east-1
export GLUTEN_S3_BENCH_PATH_STYLE_ACCESS=true
export GLUTEN_S3_BENCH_SSL_ENABLED=false
export GLUTEN_S3_BENCH_MAX_CONCURRENCY=8
export GLUTEN_S3_BENCH_UPLOAD_THREADS=8
export GLUTEN_S3_BENCH_MIN_PART_SIZE=32MB

./cpp/build/velox/benchmarks/s3_async_upload_benchmark \
  --bm_min_iters=1 \
  --bm_regex='(sync|async)_upload_(16|32|64|128|256|512|1024)M'
============================================================================
[...]/benchmarks/S3AsyncUploadBenchmark.cc     relative  time/iter   iters/s
============================================================================  
sync_upload_16M                                            80.58ms     12.41
async_upload_16M                                128.28%    62.81ms     15.92
sync_upload_32M                                           117.05ms      8.54
async_upload_32M                                93.949%   124.59ms      8.03
sync_upload_64M                                           204.63ms      4.89
async_upload_64M                                132.00%   155.03ms      6.45
sync_upload_128M                                          360.72ms      2.77
async_upload_128M                               135.32%   266.56ms      3.75
sync_upload_256M                                          709.47ms      1.41
async_upload_256M                               203.33%   348.93ms      2.87
sync_upload_512M                                             1.50s   667.72m
async_upload_512M                               237.77%   629.87ms      1.59
sync_upload_1024M                                            2.90s   345.41m
async_upload_1024M                              313.82%   922.55ms      1.08

[FelixYBW]

FYI, @weixiuli. Since the PR isn't imported into Velox. We extend Velox's S3 file system and move your enhancement into Gluten.

[zhouyuan]

@ReemaAlzaid the configuration issue should be re-generated by https://github.com/apache/gluten/blob/main/dev/gen-all-config-docs.sh

[Copilot]

Pull request overview

Adds an optional Velox-backend S3 write path that uploads multipart parts asynchronously using a shared upload thread pool, along with new Spark/Gluten configs and documentation to control concurrency.

Changes:

• Introduces spark.gluten.velox.s3UploadPartAsync, spark.gluten.velox.s3MaxConcurrentUploadNum, and spark.gluten.velox.s3UploadThreads (plus docs) to enable/tune async multipart uploads.
• Extends native config extraction to populate corresponding hive.s3.* keys and adds a focused C++ unit test.
• Adds a new s3_async_upload_benchmark to compare sync vs async upload performance.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/velox-configuration.md	Documents new S3 async upload knobs (and reorders/updates a couple of config rows).
docs/get-started/VeloxS3.md	Adds a section describing how to enable and tune async multipart uploads.
cpp/velox/utils/ConfigExtractor.cc	Maps the new Spark/Gluten configs into Velox hive.s3.* configuration.
cpp/velox/tests/GlutenS3FileSystemTest.cc	Adds a unit test validating extraction of the new async upload configs.
cpp/velox/filesystem/GlutenS3FileSystem.h	Extends the S3 FS wrapper with async-upload-related state.
cpp/velox/filesystem/GlutenS3FileSystem.cc	Implements async multipart upload write path and config handling.
cpp/velox/benchmarks/S3AsyncUploadBenchmark.cc	New benchmark for sync vs async upload throughput/latency comparisons.
cpp/velox/benchmarks/CMakeLists.txt	Registers the new benchmark behind ENABLE_S3.
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala	Adds the three new Spark configs to the Velox config registry.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.