Skip to content

GCSFileIO: use single-shot upload for small objects below a configurable threshold #17524

Description

@manideep4597

Feature Request / Improvement

Summary

GCSFileIO / GCSOutputStream always uploads via GCS WriteChannel (storage.writer(...)), which uses resumable uploads. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (storage.create(...)).

S3FileIO already switches between single PutObject and multipart based on size (s3.multipart.part-size-bytes, s3.multipart.threshold). GCS has no equivalent.

Motivation / Use case

Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.

Google Cloud Storage Java guidance also recommends:

  • small content → storage.create(blobInfo, bytes) (single request)
  • large / streaming content → storage.writer(...) (resumable)

Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.

Current behavior

In GCSOutputStream, uploads always open a write channel:

WriteChannel channel =
    storage.writer(
        BlobInfo.newBuilder(blobId).build(), writeOptions.toArray(new BlobWriteOption[0]));
gcpProperties.channelWriteChunkSize().ifPresent(channel::setChunkSize);
stream = Channels.newOutputStream(channel);

### Query engine

_No response_

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions