Skip to content

Crash recovery does not restore secondary indexes dropped by bulk parse #1556

Description

@netbrah

Summary

If a process exits after beginBulkParseLoad() drops parse-lane secondary
indexes but before endBulkParseLoad() restores them, opening the project
does not recreate those indexes.

DatabaseConnection.open() calls healBulkNodeLoad(), but that recovery
path checks/rebuilds FTS triggers only. It does not restore the indexes named by
BULK_PARSE_INDEX_NAMES / BULK_EDGE_INDEX_NAMES, despite the bulk-load
comment promising that a crash in the window is healed on the next open.

Reproduction

  1. Start a fresh index large enough to remain in bulk parse for a while.
  2. Stop the process after beginBulkParseLoad() has dropped indexes.
  3. Reopen the project or run the documented recovery sync.
  4. Inspect sqlite_master.

After reopening, the database had only:

sqlite_autoindex_nodes_1
sqlite_autoindex_files_1
idx_edges_identity

Expected node lookup indexes and idx_unresolved_status were absent.

Impact measured on a large C/C++ graph

  • 6.57 million nodes and 24.24 million pending references.
  • SELECT DISTINCT name FROM nodes became a full node scan plus temporary
    B-tree.
  • Resolver cache warm-up ran for more than 10 minutes and reached 5.5 GiB RSS
    before being stopped.
  • Recreating only idx_nodes_name reduced that phase to about 27 seconds.
  • Missing unresolved-reference indexes added separate 13-second and 11-second
    scans even when a scoped lookup returned no rows.

The recovered sync can also finish without changing
project_metadata.index_state from indexing to complete, leaving a
healthy graph permanently reported as truncated.

Suggested direction

Persist an explicit bulk-load marker, or make open-time healing verify and
recreate every index dropped by both bulk-load paths. A successful recovery
sync should also validate counts/integrity and own the metadata transition back
to complete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions