Incremental GC#15829
Open
tomberek wants to merge 3 commits into
Open
Conversation
tomberek
force-pushed
the
pr/fast-gc-rounds
branch
from
0bf2a4d to
cc018d6
Compare
The fast GC was comparing base32 hashes (from StorePath::hashPart()) against base16 hashes (from ValidPaths.hash column in SQL), causing the comparison to always fail. This meant NO roots were being protected, and rooted paths were incorrectly deleted! Fix: Extract hash part from the path column (which is already in base32 format) instead of using the hash column (which is in base16 format). This is a critical correctness bug that would cause data loss. Testing: Verify that paths with roots in /nix/var/nix/gcroots are not deleted: ls -l /nix/var/nix/gcroots/auto/ | head nix store gc --prune-older-than 7200000 --dry-run # Should NOT list paths that have roots pointing to them
tomberek
force-pushed
the
pr/fast-gc-rounds
branch
from
cc018d6 to
fb98a61
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Slow GC on a Hydra machine.
Context
Has some similarities to #14725.
The main idea is to run a SQL query to find the dead leaves and quickly GC those. This creates a GC that is not complete, but incremental and quick. Also allows using registration time because this is available in the sqlite to limit what is cleaned up.
Subsequent improvements are to make deletePath just move the contents into
/nix/store/.gc-trashatomically+quickly and allow for releasing the GC lock and unlinking in another thread, rather than being forced to slowly recursively unlink. Also looked at using io_uring for faster/nix/store/.linkscleanup, it works and is pretty fast, but unsure if we want the added complexity.Code written by Claude.... fair amount of iteration and cajoling to get the output right.
Limitation and next steps
This calls SQL directly and works only with a local store. Support for remote/daemon requires updating the daemon protocol to support the new options or to define a new operation.