Feature Request / Improvement
Summary
GCSFileIO / GCSOutputStream always uploads via GCS WriteChannel (storage.writer(...)), which uses resumable uploads. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (storage.create(...)).
S3FileIO already switches between single PutObject and multipart based on size (s3.multipart.part-size-bytes, s3.multipart.threshold). GCS has no equivalent.
Motivation / Use case
Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.
Google Cloud Storage Java guidance also recommends:
- small content →
storage.create(blobInfo, bytes) (single request)
- large / streaming content →
storage.writer(...) (resumable)
Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.
Current behavior
In GCSOutputStream, uploads always open a write channel:
WriteChannel channel =
storage.writer(
BlobInfo.newBuilder(blobId).build(), writeOptions.toArray(new BlobWriteOption[0]));
gcpProperties.channelWriteChunkSize().ifPresent(channel::setChunkSize);
stream = Channels.newOutputStream(channel);
### Query engine
_No response_
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Feature Request / Improvement
Summary
GCSFileIO/GCSOutputStreamalways uploads via GCSWriteChannel(storage.writer(...)), which uses resumable uploads. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (storage.create(...)).S3FileIOalready switches between singlePutObjectand multipart based on size (s3.multipart.part-size-bytes,s3.multipart.threshold). GCS has no equivalent.Motivation / Use case
Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.
Google Cloud Storage Java guidance also recommends:
storage.create(blobInfo, bytes)(single request)storage.writer(...)(resumable)Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.
Current behavior
In
GCSOutputStream, uploads always open a write channel: