S3FileSystem Cloudflare R2 workaround limits maximum filesize upload for default AWS S3

[justinvyu]

Describe the enhancement requested

This workaround for Cloudflare R2 storage sets the multipart upload size to a non-configurable 10MB, which results in a maximum filesize of 98GB for uploading. This is a limitation that should not spill over to the more common AWS S3 and Minio S3 options.

arrow/cpp/src/arrow/filesystem/s3fs.cc

Lines 1539 to 1546 in 640c101

	// Upload size per part. While AWS and Minio support different sizes for each
	// part (only requiring a minimum of 5MB), Cloudflare R2 requires that every
	// part be exactly equal (except for the last part). We set this to 10 MB, so
	// that in combination with the maximum number of parts of 10,000, this gives a
	// file limit of 100k MB (or about 98 GB).
	// (see https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html)
	// (for rational, see: https://github.com/apache/arrow/issues/34363)
	static constexpr int64_t kPartUploadSize = 10 * 1024 * 1024;

One possible solution: Make this configurable via some environment variable or API, and require Cloudflare R2 users to set it.

Component(s)

C++

[behcetm]

Dear arrow team,
This limitation is also affecting us and there is also another issue created for this problem #40420
Is there a solution planned for this issue?

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

[behcetm]

I think this is still a valid issue limiting the maximum object size