Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 🐞
During a Fluss cluster upgrade, a running Flink job may fail in its enumerator with the following exception:
org.apache.flink.util.FlinkRuntimeException: Failed to get table snapshot
The enumerator uses KV snapshot leases to prevent snapshots from being removed while they are being scanned. After the CoordinatorServer is restarted or its address changes during an upgrade, the client may still cache the old coordinator address.
Read-only Admin APIs were made retryable in #3390. When a read request fails with a retriable network exception, the client refreshes its cluster metadata and retries the request against the updated server address.
However, the regular AdminGateway also contains write operations such as createTable, dropTable, and createDatabase. These operations are not necessarily idempotent. If an operation succeeds on the server but its response is lost, automatically retrying it may execute the mutation twice. For this reason, the write gateway was intentionally excluded from the retry mechanism in #3390.
KV snapshot lease operations are special. They are designed to be safe when the same request is executed repeatedly:
- ACQUIRE_KV_SNAPSHOT_LEASE
- RELEASE_KV_SNAPSHOT_LEASE
- DROP_KV_SNAPSHOT_LEASE
Therefore, these operations can refresh stale coordinator metadata and retry after a retriable network failure without exposing the coordinator restart to the Flink enumerator.
Solution
No response
Are you willing to submit a PR?
Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 🐞
During a Fluss cluster upgrade, a running Flink job may fail in its enumerator with the following exception:
The enumerator uses KV snapshot leases to prevent snapshots from being removed while they are being scanned. After the CoordinatorServer is restarted or its address changes during an upgrade, the client may still cache the old coordinator address.
Read-only Admin APIs were made retryable in #3390. When a read request fails with a retriable network exception, the client refreshes its cluster metadata and retries the request against the updated server address.
However, the regular AdminGateway also contains write operations such as createTable, dropTable, and createDatabase. These operations are not necessarily idempotent. If an operation succeeds on the server but its response is lost, automatically retrying it may execute the mutation twice. For this reason, the write gateway was intentionally excluded from the retry mechanism in #3390.
KV snapshot lease operations are special. They are designed to be safe when the same request is executed repeatedly:
Therefore, these operations can refresh stale coordinator metadata and retry after a retriable network failure without exposing the coordinator restart to the Flink enumerator.
Solution
No response
Are you willing to submit a PR?