Apache Iceberg version
1.11.0 (also reproduced on a recent 1.12.0-SNAPSHOT build)
Query engine
Kafka Connect
Please describe the bug 🐞
With iceberg.tables.schema-force-optional=true and auto-create enabled, the sink creates top-level columns as optional (as documented), but fields nested inside a struct column are still created as required when the incoming Connect schema marks them required.
Concrete case: a CDC pipeline using an SMT that attaches a metadata struct to each record (e.g. the bundled DebeziumTransform, whose _cdc.key struct carries the source table's primary-key fields). On auto-create, the struct's child fields inherit the Connect schema's required, so every table is born with required nested fields even though the operator asked for force-optional.
Why this matters: if the source table's primary key later changes (ALTER TABLE ... DROP/ADD PRIMARY KEY), the old key fields disappear from incoming records, but the Iceberg schema still requires them inside the struct. The writer then fails on every record for that table (an NPE in the writer path in our case), and because this happens inside put(), errors.tolerance=all / DLQ does not catch it — the task crash-loops and the whole connector's throughput stalls. We measured a multi-day bronze-ingest outage for one source before diagnosing; the fleet-wide fix was making ~16K nested key fields optional by hand with metadata-only schema commits.
Expected behavior: schema-force-optional=true applies recursively — every field at every nesting level created (or evolved in) by the sink should be optional, so a shrinking source schema can never make a table unwritable.
Happy to test a fix against a busy CDC deployment (thousands of auto-created tables), and to propose a patch if maintainers agree the recursive interpretation is the intended semantics.
Willingness to contribute
Apache Iceberg version
1.11.0 (also reproduced on a recent 1.12.0-SNAPSHOT build)
Query engine
Kafka Connect
Please describe the bug 🐞
With
iceberg.tables.schema-force-optional=trueand auto-create enabled, the sink creates top-level columns as optional (as documented), but fields nested inside a struct column are still created asrequiredwhen the incoming Connect schema marks them required.Concrete case: a CDC pipeline using an SMT that attaches a metadata struct to each record (e.g. the bundled
DebeziumTransform, whose_cdc.keystruct carries the source table's primary-key fields). On auto-create, the struct's child fields inherit the Connect schema'srequired, so every table is born with required nested fields even though the operator asked for force-optional.Why this matters: if the source table's primary key later changes (
ALTER TABLE ... DROP/ADD PRIMARY KEY), the old key fields disappear from incoming records, but the Iceberg schema still requires them inside the struct. The writer then fails on every record for that table (an NPE in the writer path in our case), and because this happens insideput(),errors.tolerance=all/ DLQ does not catch it — the task crash-loops and the whole connector's throughput stalls. We measured a multi-day bronze-ingest outage for one source before diagnosing; the fleet-wide fix was making ~16K nested key fields optional by hand with metadata-only schema commits.Expected behavior:
schema-force-optional=trueapplies recursively — every field at every nesting level created (or evolved in) by the sink should be optional, so a shrinking source schema can never make a table unwritable.Happy to test a fix against a busy CDC deployment (thousands of auto-created tables), and to propose a patch if maintainers agree the recursive interpretation is the intended semantics.
Willingness to contribute