Skip to content

Bootstrap node_state at DB startup to prevent hlc_high_water persistence failures#139

Merged
tayyebi merged 3 commits into
mainfrom
copilot/fix-hlc-high-water-db-error
May 9, 2026
Merged

Bootstrap node_state at DB startup to prevent hlc_high_water persistence failures#139
tayyebi merged 3 commits into
mainfrom
copilot/fix-hlc-high-water-db-error

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 9, 2026

On deployments where migration files are not present at runtime, replication startup reaches the HLC persist loop and repeatedly logs set_node_state key=hlc_high_water failed: db error. The failure is caused by node_state not existing when HLC state is written.

  • DB startup hardening

    • Added a minimal runtime bootstrap in run_migrations() to create required table(s) before file-based migrations are loaded.
    • Current bootstrap scope is intentionally minimal: node_state only.
  • Targeted bootstrap contract

    • Introduced minimal_runtime_bootstrap_sql() as the explicit source of bootstrap SQL.
    • Added a focused unit test asserting bootstrap includes CREATE TABLE IF NOT EXISTS node_state.
  • Operational impact

    • HLC persistence no longer depends on migration file availability for the node_state table path.
    • Existing migration flow remains unchanged for full schema creation.
fn minimal_runtime_bootstrap_sql() -> [&'static str; 1] {
    ["CREATE TABLE IF NOT EXISTS node_state (
        key TEXT PRIMARY KEY,
        value TEXT NOT NULL
    )"]
}

fn run_migrations(client: &mut Client) {
    // ...
    bootstrap_minimal_runtime_tables(client); // ensures node_state exists
    let migrations = load_available_migrations();
    // ...
}

Copilot AI linked an issue May 9, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits May 9, 2026 10:18
Copilot AI changed the title [WIP] Fix hlc high water db error on mailserver startup Bootstrap node_state at DB startup to prevent hlc_high_water persistence failures May 9, 2026
Copilot finished work on behalf of tayyebi May 9, 2026 10:23
Copilot AI requested a review from tayyebi May 9, 2026 10:23
@tayyebi tayyebi marked this pull request as ready for review May 9, 2026 11:13
@tayyebi tayyebi merged commit eb43e0f into main May 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

set_node_state key=hlc_high_water failed: db error

2 participants