[Bug] Persistent metadata update failures can block batch session timeout cleanup

[ruanwenjun]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched the existing open and closed issues and found no similar issue.

Describe the bug

A persistent JDBC metadata update failure can remain in the asynchronous retry queue indefinitely and block batch session cleanup.

The relevant sequence is:

1. JDBCMetadataStore.updateMetadata throws when PreparedStatement.executeUpdate() returns 0.
2. With kyuubi.metadata.request.async.retry.enabled=true, MetadataManager.updateMetadata enqueues the failed update.
3. The retry worker removes the queue head only after a successful update. A deterministic failure therefore remains at the head and is retried indefinitely.
4. KyuubiBatchSession.close waits without a timeout until the metadata retry queue is empty.
5. The session timeout checker is a single-thread scheduled executor and calls closeSession synchronously. One affected batch session can therefore prevent later sessions from being checked and removed. Old sessions may continue to appear in /api/v1/sessions even while their metadata rows are independently removed by the metadata cleaner.

In the observed sequence, a batch session was closed while its application monitor was still running. The monitor later observed the driver as FAILED, updated the application information, and the asynchronous operation's finally block attempted another terminal metadata update. This explains repeated terminal updates, but it has not been established that repetition alone caused Connector/J to return 0.

The exact reason for the first zero update count is still under investigation. This report is about the deterministic failure amplification after any persistent zero-count update, not an assumption about its original cause.

Reproduction outline

1. Configure the MySQL JDBC metadata store with asynchronous retry enabled.
2. Submit a batch and make a terminal metadata update return 0, for example by deleting the row before the update or by using a test metadata store that produces a deterministic zero-count update.
3. Close the corresponding batch session.
4. Observe that the same update remains at the retry queue head.
5. Observe that KyuubiBatchSession.close does not return and that the single session timeout checker cannot process later sessions.

Expected behavior

• Metadata persistence failures must not block session lifecycle cleanup indefinitely.
• A zero-count update should be reconciled using its postcondition instead of being treated as unconditional success or unconditional transient failure:
  • if the row exists and all columns included in the update already match the intended values, the operation is idempotently complete;
  • if the row is missing or its values do not match, retain a classified failure without silently recreating deleted metadata;
  • preserve ordering when an earlier INSERT for the same identifier is pending.
• Persistent failures should use observable and bounded/rate-limited handling, and normal session close should not wait indefinitely for the asynchronous retry queue.

Affects Version(s)

Observed on a deployment based on 1.9.1. The relevant code paths are also present on master at 804c00239815d3b7eece5c4c8bb8f6263063beff.

Kyuubi Server Log Output

INFO  ... KyuubiBatchSession: Mark session SessionHandle [<batch-id>] closed
INFO  ... BatchJobSubmission: Batch report for <batch-id>, Some(ApplicationInfo(...,FAILED,...))
ERROR ... MetadataManager: Error updating metadata for session <batch-id>
org.apache.kyuubi.KyuubiException: Error updating metadata for <batch-id> by SQL: UPDATE metadata SET ... WHERE identifier = ?
INFO  ... MetadataManager: Retrying metadata requests for <batch-id>
ERROR ... MetadataManager: Error retrying metadata requests for <batch-id>
INFO  ... KyuubiBatchSession: There are still remaining metadata store requests for batch[<batch-id>]

Kyuubi Engine Log Output

Not required to reproduce the metadata retry and session-close behavior.

Kyuubi Server Configurations

kyuubi.metadata.store.jdbc.database.type: MYSQL
kyuubi.metadata.request.async.retry.enabled: true
kyuubi.metadata.request.retry.interval: 5s

The deployment is intended to use MySQL Connector/J 5.x and the JDBC URL does not explicitly set useAffectedRows=true; the effective runtime driver and connection properties are still being verified.

Additional context

Relevant current-master code:

• JDBCMetadataStore.updateMetadata
• metadata asynchronous retry loop
• KyuubiBatchSession waiting for retry completion

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix this.