runtime-next: discard hinted frontier when committed frontier is rebuilt - #3165
Merged
Conversation
When the materialization leader rebuilds a task's committed frontier from an authoritative recovered checkpoint — a legacy V1 checkpoint during a V1->V2 migration, or a remote-authoritative connector checkpoint — the recovered hinted frontier may reference producers the rebuilt committed frontier no longer carries. `project_hinted` zeroes each hinted producer's read offset and relies on a matching committed-frontier producer to restore it. An orphaned hint (no committed counterpart) therefore resumes at journal offset 0, forcing a full re-read of the journal from the beginning to replay a hint whose data the authoritative checkpoint already reflects. On large journals this can prevent startup from completing. Discard the recovered hinted frontier whenever the committed frontier is rebuilt: an authoritative checkpoint is a complete committed frontier, so there is nothing to replay. Also clear the hinted frontier from RocksDB via the startup cleanup Persist so it cannot resurface on a later recovery. Additionally, extract the startup recovery reconciliation into a pure `reconcile_recovered` function (no IO) and add unit tests covering the discard on both rebuild branches, hint preservation when there is no rebuild, the connector hinted-close delta path, and the drop-rollback path.
mdibaiee
added a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Jul 23, 2026
## What's Changed * docs: surface MCP and agent skills for coding agents by @jwhartley in estuary/flow#3093 * dekaf: serve the document containing a mid-document fetch offset by @jshearer in estuary/flow#3150 * runtime: let docker atomically assign published connector host ports by @dgreer-dev in estuary/flow#3162 * runtime-next: discard hinted frontier when committed frontier is rebuilt by @jgraettinger in estuary/flow#3165 * docs/source-postgres: Sync statement_timeout change by @willdonnelly in estuary/flow#3168 * Provide capture/materializations created_at date by @dgreer-dev in estuary/flow#3160 * docs: add Processing order section for materialization binding priority by @jwhartley in estuary/flow#3171 * docs: reorganize agent skills page by plugin, add derivations and schema by @jwhartley in estuary/flow#3170 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3180 * Service Accounts by @GregorShear in estuary/flow#3058 * Bound too-large error messages over RPC by @dgreer-dev in estuary/flow#3182 * control-plane: guarded, auto-expiring temporary support access to restricted tenants by @skord in estuary/flow#3115 * private links: model links as rows with controller-observed status by @jshearer in estuary/flow#3063 * Adding support for an stripe web hook endpoint. by @bbartman in estuary/flow#3174 * docs: Dekaf consumer behaviors + Spark Structured Streaming guidance by @jwhartley in estuary/flow#3094 * shuffle: log canonical journal name in slice read events by @jgraettinger in estuary/flow#3190 * runtime: persist the max-key fail-safe sentinel at adoption by @jgraettinger in estuary/flow#3191 * local: make flow-plane-link robust to control-plane agent start-up by @jgraettinger in estuary/flow#3158 * gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY by @jgraettinger in estuary/flow#3159 * docs: add warning about source-stripe-native's events API version by @nicolaslazo in estuary/flow#3166 * agent-api: Add `STRIPE_WEBHOOK_SECRET` to cloud run config by @jshearer in estuary/flow#3196 * shuffle, gazette: guard against wedged reads by @jgraettinger in estuary/flow#3193 * runtime-next: gate task-log stream by configured log level by @williamhbaker in estuary/flow#3198 * flowctl: preview-next appends a final drain session by @mdibaiee in estuary/flow#3195 * Allow localhost:3000 origin for the local control-plane agent by @GregorShear in estuary/flow#3206 * control-plane-api: don't lock injected ops collections during publications by @skord in estuary/flow#3164 * runtime-next: force txn close at any usage ceiling, bypassing min duration by @jgraettinger in estuary/flow#3203 * shuffle: replay gapped producers on restart by @jgraettinger in estuary/flow#3202 * docs: source-zuora by @Alex-Bair in estuary/flow#3204 * dekaf: only read back when a fetch offset lands inside a document by @mdibaiee in estuary/flow#3205 * mise: dev-VM zone configurability (create fallback, zone-agnostic SSH, vm:move-gcp) by @skord in estuary/flow#3208 * docs/postgres: Discovery Filters by @willdonnelly in estuary/flow#3215 * docs: document materialize-s3-iceberg nanosecond_timestamps option by @jacobmarble in estuary/flow#3199 * runtime v2 flag for new captures and derivations by @williamhbaker in estuary/flow#3219 * runtime-next: converge startup reconciliation via rescan Persists by @jgraettinger in estuary/flow#3218 * build(deps): bump fast-uri from 3.1.0 to 3.1.4 in /site by @dependabot[bot] in estuary/flow#3223 * build(deps): bump svgo from 3.3.3 to 3.3.4 in /site by @dependabot[bot] in estuary/flow#3224 * build(deps): bump dompurify from 3.3.3 to 3.4.12 in /site by @dependabot[bot] in estuary/flow#3222 * build(deps): bump shell-quote from 1.8.3 to 1.10.0 in /site by @dependabot[bot] in estuary/flow#3213 * build(deps): bump body-parser from 1.20.4 to 1.20.6 in /site by @dependabot[bot] in estuary/flow#3212 * build(deps): bump webpack-dev-server from 5.2.2 to 5.2.6 in /site by @dependabot[bot] in estuary/flow#3211 * build(deps): bump websocket-driver from 0.7.4 to 0.7.5 in /site by @dependabot[bot] in estuary/flow#3183 * Docs: MySQL discovery filters by @aeluce in estuary/flow#3217 * docs: source-smartsheet by @nicolaslazo in estuary/flow#3228 * control-plane-api: surface per-store health diagnostics in storage-mapping mutations by @GregorShear in estuary/flow#3181 * async-process: handle reaping-task JoinError in Child::wait() without panicking by @jgraettinger in estuary/flow#3225 * feat: end-to-end truncation by @williamhbaker in estuary/flow#3132 * docs: source-zuora uses the AQuA API now by @Alex-Bair in estuary/flow#3237 * Improve storage mapping GraphQL results by @GregorShear in estuary/flow#3200 * control-plane: add `closed` flag to data planes by @GregorShear in estuary/flow#3175 **Full Changelog**: estuary/flow@v0.6.11...v0.6.12 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
mdibaiee
added a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Jul 23, 2026
## What's Changed * docs: surface MCP and agent skills for coding agents by @jwhartley in estuary/flow#3093 * dekaf: serve the document containing a mid-document fetch offset by @jshearer in estuary/flow#3150 * runtime: let docker atomically assign published connector host ports by @dgreer-dev in estuary/flow#3162 * runtime-next: discard hinted frontier when committed frontier is rebuilt by @jgraettinger in estuary/flow#3165 * docs/source-postgres: Sync statement_timeout change by @willdonnelly in estuary/flow#3168 * Provide capture/materializations created_at date by @dgreer-dev in estuary/flow#3160 * docs: add Processing order section for materialization binding priority by @jwhartley in estuary/flow#3171 * docs: reorganize agent skills page by plugin, add derivations and schema by @jwhartley in estuary/flow#3170 * go.mod: bump gazette to latest by @williamhbaker in estuary/flow#3180 * Service Accounts by @GregorShear in estuary/flow#3058 * Bound too-large error messages over RPC by @dgreer-dev in estuary/flow#3182 * control-plane: guarded, auto-expiring temporary support access to restricted tenants by @skord in estuary/flow#3115 * private links: model links as rows with controller-observed status by @jshearer in estuary/flow#3063 * Adding support for an stripe web hook endpoint. by @bbartman in estuary/flow#3174 * docs: Dekaf consumer behaviors + Spark Structured Streaming guidance by @jwhartley in estuary/flow#3094 * shuffle: log canonical journal name in slice read events by @jgraettinger in estuary/flow#3190 * runtime: persist the max-key fail-safe sentinel at adoption by @jgraettinger in estuary/flow#3191 * local: make flow-plane-link robust to control-plane agent start-up by @jgraettinger in estuary/flow#3158 * gazette: surface fragment store diagnostic on FRAGMENT_STORE_UNHEALTHY by @jgraettinger in estuary/flow#3159 * docs: add warning about source-stripe-native's events API version by @nicolaslazo in estuary/flow#3166 * agent-api: Add `STRIPE_WEBHOOK_SECRET` to cloud run config by @jshearer in estuary/flow#3196 * shuffle, gazette: guard against wedged reads by @jgraettinger in estuary/flow#3193 * runtime-next: gate task-log stream by configured log level by @williamhbaker in estuary/flow#3198 * flowctl: preview-next appends a final drain session by @mdibaiee in estuary/flow#3195 * Allow localhost:3000 origin for the local control-plane agent by @GregorShear in estuary/flow#3206 * control-plane-api: don't lock injected ops collections during publications by @skord in estuary/flow#3164 * runtime-next: force txn close at any usage ceiling, bypassing min duration by @jgraettinger in estuary/flow#3203 * shuffle: replay gapped producers on restart by @jgraettinger in estuary/flow#3202 * docs: source-zuora by @Alex-Bair in estuary/flow#3204 * dekaf: only read back when a fetch offset lands inside a document by @mdibaiee in estuary/flow#3205 * mise: dev-VM zone configurability (create fallback, zone-agnostic SSH, vm:move-gcp) by @skord in estuary/flow#3208 * docs/postgres: Discovery Filters by @willdonnelly in estuary/flow#3215 * docs: document materialize-s3-iceberg nanosecond_timestamps option by @jacobmarble in estuary/flow#3199 * runtime v2 flag for new captures and derivations by @williamhbaker in estuary/flow#3219 * runtime-next: converge startup reconciliation via rescan Persists by @jgraettinger in estuary/flow#3218 * build(deps): bump fast-uri from 3.1.0 to 3.1.4 in /site by @dependabot[bot] in estuary/flow#3223 * build(deps): bump svgo from 3.3.3 to 3.3.4 in /site by @dependabot[bot] in estuary/flow#3224 * build(deps): bump dompurify from 3.3.3 to 3.4.12 in /site by @dependabot[bot] in estuary/flow#3222 * build(deps): bump shell-quote from 1.8.3 to 1.10.0 in /site by @dependabot[bot] in estuary/flow#3213 * build(deps): bump body-parser from 1.20.4 to 1.20.6 in /site by @dependabot[bot] in estuary/flow#3212 * build(deps): bump webpack-dev-server from 5.2.2 to 5.2.6 in /site by @dependabot[bot] in estuary/flow#3211 * build(deps): bump websocket-driver from 0.7.4 to 0.7.5 in /site by @dependabot[bot] in estuary/flow#3183 * Docs: MySQL discovery filters by @aeluce in estuary/flow#3217 * docs: source-smartsheet by @nicolaslazo in estuary/flow#3228 * control-plane-api: surface per-store health diagnostics in storage-mapping mutations by @GregorShear in estuary/flow#3181 * async-process: handle reaping-task JoinError in Child::wait() without panicking by @jgraettinger in estuary/flow#3225 * feat: end-to-end truncation by @williamhbaker in estuary/flow#3132 * docs: source-zuora uses the AQuA API now by @Alex-Bair in estuary/flow#3237 * Improve storage mapping GraphQL results by @GregorShear in estuary/flow#3200 * control-plane: add `closed` flag to data planes by @GregorShear in estuary/flow#3175 **Full Changelog**: estuary/flow@v0.6.11...v0.6.12 Co-authored-by: mdibaiee <mdibaiee@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the materialization leader rebuilds a task's committed frontier from an authoritative recovered checkpoint — a legacy V1 checkpoint during a V1→V2 migration, or a remote-authoritative connector checkpoint — the recovered hinted frontier may reference producers that the rebuilt committed frontier no longer carries.
project_hintedzeroes each hinted producer's read offset and relies on a matching committed-frontier producer to restore it. An orphaned hint (one with no committed counterpart) therefore resolves to journal offset 0, forcing a full re-read of the journal from the beginning to replay a hint whose data the authoritative checkpoint already reflects. On large journals this can prevent a task from completing startup.Fix
Persist(delete_hinted_frontier), so stale hints can't resurface on a later recovery.This applies to both rebuild paths (legacy V1 checkpoint and connector-authoritative checkpoint). The no-rebuild path, the connector hinted-close delta path, and the drop-rollback path are unchanged.
Refactor + tests
Extracts the startup recovery reconciliation out of the async
run()path into a purereconcile_recoveredfunction (no IO), making the reconciliation policy unit-testable. Adds unit tests covering:All
runtime-nextleader::materialize::startuptests pass.