Commit-validation failures

[martinskeem]

Query engine

Kafka Connect

Question

Hi,

We use the Iceberg connector with Kafka Connect to sink data to Azure Databricks using their Iceberg catalog implementation.

After some time of successful operation, we see errors such as the one below after which there is no progression in the sink connector.

Coordinator iceberg-sink-connector-epe-log-topics-v1-0 failed to commit for commit 51a29b27-d1b1-45f4-b6f3-61228b4c8481, will try again next cycle","debug_stacktrace":"org.apache.iceberg.exceptions.BadRequestException: Malformed request: Commit validation failed. Please contact Databricks support for assistance. [ErrorCode: 2010]
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:341)
	at org.apache.iceberg.rest.ErrorHandlers$CommitErrorHandler.accept(ErrorHandlers.java:137)
	at org.apache.iceberg.rest.ErrorHandlers$CommitErrorHandler.accept(ErrorHandlers.java:119)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:242)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:347)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:299)
	at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:112)
	at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:150)
	at org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:206)
	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:501)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:473)
	at org.apache.iceberg.connect.channel.Coordinator.commitToTable(Coordinator.java:286)
	at org.apache.iceberg.connect.channel.Coordinator.lambda$doCommit$1(Coordinator.java:173)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

We have done some analysis and it appears that Databricks on commit validation failures returns a HTTP 400 status code and not HTTP 409 error which appears to be how the Iceberg code recognises commit validation failures - after which it retries (optimistic concurrency).

I have a couple of questions:

1. Is HTTP 409 the Iceberg specification for commit validation failures? Is it formalised somewhere? If so, I would like to include this in a bug report to Databricks (if not, I wonder if it will be accepted as a bug).
2. Would you consider a PR that works around the issues such as this REST: treat HTTP 400 commit-validation failures as CommitFailedException #16644. We are currently testing this workaround in our development environment.

Br,
Martin

[wombatu-kun]

Hi @martinskeem, thanks for the detailed write-up and stack trace.

1. Is 409 the spec'd status for commit conflicts?

Yes. The REST catalog OpenAPI spec models the commit route (updateTable) so that a requirement/assertion failure is a 409, not a 400:

• 409 -> "Conflict - CommitFailedException, one or more requirements failed. The client may retry."
• 400 -> BadRequestErrorResponse, and the operation description states: "Server implementations are required to fail with a 400 status code if any unknown updates or requirements are received."

So the spec reserves 400 for a malformed commit (unknown updates/requirements), while a genuine concurrent-write conflict - "one or more requirements failed" - is a 409 that the client may retry. A catalog returning 400 for a concurrent-write conflict conflates those two cases, which looks like a server-side compliance gap. That citation (in open-api/rest-catalog-open-api.yaml, the updateTable responses) should be useful for a report to Databricks.

2. Why it wedges the connector (root cause)

The retry is keyed on exception type, not status code:

• ErrorHandlers.CommitErrorHandler maps 409 -> CommitFailedException and lets 400 fall through to DefaultErrorHandler, which raises BadRequestException.
• SnapshotProducer.commit retries with .onlyRetryOn(CommitFailedException.class).

So a 400 becomes a terminal BadRequestException that escapes the retry-with-refresh loop entirely, and the coordinator can't make progress - exactly what you're seeing.

3. On the workaround in #16644

The mechanism is sound - remapping to CommitFailedException is what restores the optimistic-concurrency retry. A few things worth weighing on the approach (I'm a contributor here, not a maintainer, so whether core should carry this is their call):

• Matching on message text (contains "commit" + "validation failed") is coupled to Databricks' current wording. If they reword the message it silently stops working, and it won't help other non-compliant catalogs that phrase it differently.
• It changes default behavior for every REST catalog user. A genuinely malformed 400 whose message happens to contain those tokens would be reclassified as a retryable conflict and retried up to commit.retry.num-retries times, which can mask real client errors.
• There are no tests yet; TestErrorHandlers has no CommitErrorHandler coverage, so it would be good to pin both the match and the fall-through.

Possible directions (maintainers' call):

• Cleanest: treat this as a Databricks compliance bug and file it - the spec citation above is the basis. That keeps core spec-compliant.
• If a client-side mitigation is still wanted, an opt-in catalog property (default off) that remaps 400 conflict responses to CommitFailedException would keep spec-compliant behavior as the default rather than applying an unconditional heuristic to everyone. Worth pairing with tests.
• Either way it is a behavior change in engine-agnostic core/, so a quick note on the dev@ list (or tagging REST-catalog maintainers) before investing further is probably wise.

Happy to help with tests or a config-gated variant if you decide to take it forward.

[martinskeem]

Excellent - thank you very much for the detailed response!

I will do some more testing our environment to confirm that the PR as-is resolves the issue + pursue a bug-fix with Databricks and Microsoft. That would (as you say) be the preferred and cleanest solution.

If not, I will think about adding tests and an opt-in property before publishing the PR.

[RussellSpitzer]

I would not consider a 400 as a 409, that doesn't make sense to me and I don't think we should support special handling based on error text generated by a specific vendor.