Fix subpar caching usage during ingestion

[jviotti]

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

[github-actions]

Benchmark Index (community)

Details

Benchmark suite	Current: bd2ea84	Previous: ef1287c	Ratio
Add one schema (0 existing)	40 ms	41 ms	0.98
Add one schema (100 existing)	472 ms	455 ms	1.04
Add one schema (1000 existing)	4594 ms	4594 ms	1

This comment was automatically generated by workflow using github-action-benchmark.

[github-actions]

Benchmark Index (enterprise)

Details

Benchmark suite	Current: bd2ea84	Previous: ef1287c	Ratio
Add one schema (0 existing)	44 ms	51 ms	0.86
Add one schema (100 existing)	473 ms	476 ms	0.99
Add one schema (1000 existing)	4771 ms	4757 ms	1.00

This comment was automatically generated by workflow using github-action-benchmark.

[cubic-dev-ai]

2 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/index/index.cc">

<violation number="1" location="src/index/index.cc:811">
P2: Built artifacts are only hardlinked back to staging when missing, so stale existing files are left behind and can cause repeated cache misses on subsequent runs.</violation>
</file>

<file name="src/index/output.h">

<violation number="1" location="src/index/output.h:89">
P1: `files()` exposes `tracker` by reference without synchronization, allowing unsynchronized reads during concurrent writes and causing undefined behavior.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[augmentcode]

🤖 Augment PR Summary

Summary: This PR improves cache tracking during one index runs that use a staging directory, aiming to avoid unnecessary rebuilds on subsequent runs.

Changes:

• Introduces Output::FileStatus (Unseen/Skipped/Built) to distinguish cached vs rebuilt outputs.
• Updates the indexing dispatch path to explicitly track both generated targets and their .deps files as built or skipped.
• Adds a post-atomic_directory_swap step that hardlinks files built in the current run from the new committed output back into the staging directory to improve cache hits.
• Exposes tracked outputs via Output::files() to drive the hardlinking step.
• Adds a new CLI regression test (rebuild-two-to-three) to validate full cache hits after expanding the schema set.
• Registers the new test in the CLI CMake test list.

Technical Notes: The approach relies on hardlinking (not copying) to keep staging in sync with the committed output for incremental runs while preserving the atomic commit behavior.

_{🤖 Was this summary useful? React with 👍 or 👎}

[augmentcode]

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

The post-swap hardlink step only runs when !std::filesystem::exists(entry.first), so targets rebuilt during this run (status Built but already present in the old output) will remain stale in staging_path and may still miss cache hits on the next run. Is it intentional to only backfill new files rather than also refreshing existing ones that were rebuilt?

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

[augmentcode]

Output::files() returns tracker without taking tracker_mutex; since track() is called from multiple threads, calling files() before all work finishes would be undefined behavior (similar to the note in is_untracked_file). Consider guarding this accessor or returning a snapshot to make it safe to use in the presence of concurrent tracking.

Severity: low

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}