opendal FileIO hard-wires TimeoutLayer defaults (10s per-IO) with no way to configure them; large Parquet row groups fail deterministically on S3

[bcgraham]

Apache Iceberg Rust version

Reproduced on iceberg 0.10.1 / iceberg-storage-opendal 0.10.1. The code path is unchanged on main as of 2026-08-07 (86d9d7d).

Describe the bug

iceberg-storage-opendal wraps every FileIO operator in TimeoutLayer::new(), and nothing in the crate's property or builder surface can override it:

iceberg-rust/crates/storage/opendal/src/lib.rs

Line 368 in 86d9d7d

let operator = operator.layer(TimeoutLayer::new()).layer(RetryLayer::new());

TimeoutLayer::new() uses opendal's defaults — 60 s overall, 10 s per IO operation (opendal 0.58 docs).

The 10 s io_timeout is the one that bites on the write path. opendal turns a single large write into a single S3 multipart part, and parquet-rs flushes a completed row group as one write — so in practice the row group is the part. Any row group whose encoded size exceeds what the writer's uplink can push in 10 s fails, and fails deterministically: the same request is retried by the RetryLayer above it and cannot ever fit inside the budget, so every attempt dies at the same place.

This is easy to hit on a table with a wide binary/string column, where row-group size is dominated by a handful of large values rather than by the row count the caller configured.

To Reproduce

Environment: iceberg 0.10.1 + iceberg-catalog-glue, parquet/arrow 58, rustc 1.97.1, S3 over a WAN link (developer laptop, not in-region).

1. Create an Iceberg table with a binary payload column carrying large values — in our corpus ~9,200 rows / ~1.3 GB total, p50 ≈ 2.9 KB, p99 ≈ 1.4 MB, max ≈ 33 MB.
2. Write it through ParquetWriterBuilder with row-count row-group sizing, WriterProperties::set_max_row_group_size(2048).
3. The write fails while flushing a row group:

Unexpected (persistent) at write, context: {
  upload_id: <redacted>, part_number: 0, service: s3,
  path: <table>/data/00000-....parquet,
  size: 125382156, written: 428988086
} => write part timeout
  1: External: External: Unexpected => Failure in doing io operation, ...
  ...
  4: Unexpected (persistent) at write, context: { timeout: 10 } => io operation timeout reached

size: 125382156 is the part being flushed; timeout: 10 is the hard-wired io_timeout. Three separate runs of the identical configuration failed at the identical byte offset.

Two controls isolate the cause:

• Same data, same row-group configuration, written with arrow-rs + object_store directly (no Iceberg FileIO): completed in 44 s, no error. So this is not parquet, not arrow, and not S3.
• Same data through Iceberg with smaller row groups (512 rows, ≈4× smaller parts; and byte-based row-group targets of 4/16/64 MiB set in code): all succeed.

The threshold itself is bandwidth-dependent — an in-region task will tolerate a much larger row group than this laptop did — but the ceiling exists at every bandwidth and is not visible or adjustable from the Iceberg API.

Expected behavior

Either of:

1. Make the timeouts configurable. A FileIO property in the existing client.* namespace (alongside client.region, client.assume-role.arn, …) — e.g. client.io-timeout-ms / client.timeout-ms — or a builder hook on the storage config, so a caller writing large parts can raise the per-IO budget.
2. Scale the per-IO timeout with the size of the write being attempted, so a large multipart part gets a proportional budget instead of a fixed 10 s.

The current default is a sensible one, and the comment at the call site is right that TimeoutLayer must sit inside RetryLayer so each attempt is independently bounded. The problem is only that it is a hard ceiling: it silently caps the maximum row-group size a table can be written with, as a function of the writer's uplink bandwidth rather than of anything the table or the data expresses — and it surfaces as a persistent, retry-proof write failure rather than as a configuration error.

Willingness to contribute

I cannot contribute a fix for this bug at this time.