residue v0.0.10

Removal and restore that stay safe when a worktree, or another residue run, changes under them, and less CPU and memory.

• Nothing that turns up after the move is deleted. remove refuses a repository, a mounted volume or a folder it cannot read that appears in a worktree after residue moves it aside, and puts the worktree back (#106). A folder named .git in another case, such as private/.GIT/, keeps the worktree (#102). The size walk never follows a symbolic link, also one swapped in while it reads (#94).
• Restore and its records tell the truth. A removal record whose worktree is back reads back, watch no longer removes that worktree again because its work landed, and residue restore of it finishes what an earlier restore left (#99). A restore racing a forget of the same record keeps the commits only its reflog held (#98), and a trash entry can no longer get stuck (#97).
• Less CPU and memory. residue notices that a Git call has finished as soon as it exits (#82), on macOS each Git call starts sooner (#81), and git status uses a third to a half of the system CPU it did (#75). explain of a large worktree spends about half as long in git status (#83) and uses about 5 MB less memory (#78). On macOS no child process inherits another child's pipes, in residue (#114) and in the Mac app (#116). The Mac app uses less CPU and memory (#75, #95), and its text is brief (#77).
• On macOS, an ignored folder whose name the disk stores decomposed gets its size (#103), and explain run from inside a worktree no longer names lsof as a process using it (#85). On Linux, the activity check sees a process that uses a worktree only through a mapped file (#84).
• Changes from 0.0.9. A removal record's state in --json can now be back, unknown, incomplete or unreadable, so a record that did not plainly remove its worktree never reads removed (#99). On Linux, a worktree that a process uses only through a mapped file is kept as in use (#84). On NetBSD, OpenBSD and DragonFly, and on macOS 12 and earlier, every folder above a worktree must be readable, not only searchable (#94). --help and result lines are shorter; JSON documents, exit statuses and the lines the Mac app reads are unchanged (#77).

Details

• A removal record whose worktree is back no longer reads as removed, and watch no longer removes that worktree again because its work landed. Back means Git lists a worktree with its checkout at the record's original path on its branch (detached when the record is), at the recorded HEAD or ahead of it: what a restore that kept its record leaves, or one stopped after git worktree add before it wrote its restored marker, or a git worktree add there by hand, or a run that moved the worktree back and could not drop the record. A locked worktree whose folder is gone is not back. Its state is now back; seeding adds no freed line for it; watch's landed rule keeps the worktree while the record lives, as it keeps a marked one; and residue restore of it, without --to, finishes the job: it writes the restored marker if there is none, puts back the saved config.worktree (next to the worktree's own when that one differs), and puts the saved HEAD reflog in front of the worktree's own, then drops the record once the worktree holds what its refs keep (checkout already_back). Before, a restore stopped after git worktree add left the commits only the old reflog held to git gc once the record was forgotten. In a reftable repository Git cannot take the old reflog back, so the record stays and the restore says why. When Git cannot compare the worktree's HEAD with the recorded one, the state is unknown, with back_error saying why, never removed: seeding adds nothing, watch keeps the worktree, and restore of it without --to refuses. A record with no info now reads incomplete and one residue cannot read unreadable, with their incomplete and error fields as before. Decision: new state values rather than new fields, so a script that reads only state never sees removed for a worktree that is back or for a record that deleted nothing; the Mac app's types take the new values, and it offers Restore only for removed, interrupted and back, and no freed size for a worktree that is back. A trash restore that lands while a purge deletes the same entry no longer makes the purge say "Git no longer lists it, so part of it may be gone": the purge also looks for the worktree at its original path, and when Git lists it there whole, the purge is skipped, says another run put it back, and drops its record. A worktree at a listed path counts as the record's only when its Git directory is the one the record was written for, the same directory and not one Git made again under its name, so another worktree added at that path never makes the purge drop the record and the refs that keep the trashed worktree's commits. Dropping a record another run already dropped is no longer an error. The removal child reads no records, so a worktree that is back is kept by watch's own filter, not by the child's --only-if-upstream-gone check. Tests: scripts/recovery-check.sh adds purge-restore-race (a purge held at git worktree remove while residue restore moves the worktree out of the trash), restore-stopped-before-mark (a restore killed at its first Git command in the worktree it recreated, then run again), purge-other-worktree and its reftable run (a purge's remove that stops part way while another worktree sits at the original path), back-locked-missing and back-unknown; .github/watch-smoke.sh adds step 9 (that worktree, its upstream gone, past --grace). purge-restore-race, restore-stopped-before-mark, back-unknown and step 9 fail on 0.0.9; the other three guard this change's own matching (#99).
• remove no longer deletes a repository or a mounted volume that appears in a worktree after residue moves it aside. The check after the move compared only what the plan recorded. git status lists neither a .git inside a folder Git tracks nor anything under an ignored folder, so a git init in src/, or a volume mounted under node_modules/, passed that check. git worktree remove then deleted the repository with its commits, or the files on the volume. A folder nobody can read passed too, and the removal stopped part way. Now the check after the move refuses what the check before it refuses: a .git entry other than the worktree's own, a mount point, or an entry residue cannot read. The reason says what it found and where, such as contains a .git entry at src/.git after residue moved it aside; it is back in place, and the worktree is moved back. Tests: scripts/recovery-check.sh adds nested-repo-after-move, mount-after-move and unreadable-after-move, which plant each one between the plan and that check. All three fail on 0.0.9. mount-after-move mounts a disk image on macOS, and a tmpfs as root on Linux, FreeBSD and illumos; it is skipped elsewhere. unreadable-after-move is skipped as root, who can read any folder (#106).
• On macOS, a child process residue starts, such as Git, no longer inherits the pipes of another child started at the same moment. The standard library makes a child's pipes and marks them close-on-exec in two steps, and residue inspects up to 8 worktrees at once, so a Git process started on one thread between the two steps held another's output open: that call ended only when the other process, and anything it left running, exited. Now one child starts at a time, and the lock is held for the start alone. Through residue's own process code, 335 of 48,000 children started on 8 threads had inherited a pipe before, and none now. A scan of 10 worktrees takes as long and uses as much CPU as before. This also fixes a test that failed now and then under load (#114, #111).
• On macOS, a child process the Mac app starts, such as residue, no longer inherits the pipes of another child the app started at the same moment, the race #114 found in residue's own children. The window's requests run on several threads, so a quick one, such as reading the Automatic cleanup status, could wait until an unrelated scan ended. Now the app also starts one child at a time, holding the lock for the start alone, and app/src-tauri/clippy.toml refuses any other way to start one. Through the app's own process code, 844 of 48,000 children started on 8 threads had inherited a pipe before, and none now. Starting them took no longer (#116).
• A worktree holding a folder named .git in another case, such as private/.GIT/, is now kept. On macOS Git's core.ignorecase is on, so Git skips that name in any case and git status lists nothing inside it. residue matched only .git, so it saw no ignored files and no nested repository, and could remove the worktree with that folder's files unlisted. The nested_repo check now blocks on any case of .git at any level, other than the worktree's own .git link. That includes a .GIT/ at the root, which a case-sensitive file system can hold beside the link: a repository made on macOS and copied to Linux keeps core.ignorecase on. The check names the entry it found: contains a .git entry at private/.GIT: another repository or checkout, or files Git does not list. Where Git is case-sensitive and lists such a folder's files, residue keeps the worktree when it need not, the safe direction for a name this rare (#102).
• A restore and a forget of the same removal record at the same time no longer lose the commits only its reflog held without saying so. When a remove run, such as a watch child, forgot a record past --keep-removed while residue restore recreated its worktree, restore found the saved reflog gone and put none back. Then it said the record stayed "with its refs, so nothing it keeps is lost", though the record and its refs were gone and git gc could drop those commits. Now restore reads the record's copies before it changes anything, so it puts the reflog back from them. A record forgotten before that read makes restore change nothing, exit 1, and say another run took it. Whether the record stays is said from its refs as Git has them then; when another run forgot it and the reflog could not go back (a reftable repository), restore names each commit no ref keeps any more. Tests: scripts/recovery-check.sh adds restore-forget-race (the restore held at git worktree add while a sweep forgets the record), forget-restore-race (the sweep held at its ref deletion while the restore finishes first; its forget fails, as a warning), forget-before-restore-reads and restore-forget-race-reftable. On 0.0.9, all but forget-restore-race fail (#98).
• On macOS, an ignored folder whose name is stored decomposed on disk, such as café/ written as cafe and a combining accent, now gets its size, and a path rule inside it can match. Git prints such a name precomposed, so residue never found the folder: it reported no size, and a rule like café/sub/*.txt kept the worktree with "the walk did not reach it". residue now also looks each non-ASCII path up the way Git spells it, with the same conversion Git uses (iconv from UTF-8-MAC), and matches a rule against either spelling. Nothing was removed unlisted before: a folder the walk does not reach is kept (#103).
• A trash entry can no longer get stuck. A remove run in the same repository, such as the Mac app's or a watch child's, cleared an entry that held only its record at once, also one another run had just made and was moving a worktree into; that worktree could land in the trash with no info, so residue restore could not take it and every later remove there exited 3. Now an entry that holds only its record is cleared an hour after it was made (removed::INCOMPLETE_AFTER, the hour removal records already wait). info is written as info.new, synced and renamed, as a removal record's is, so a power loss leaves a whole info or none, and an hour-old entry whose info is missing or damaged and that holds nothing else is cleared too, where it failed every run. A young entry with no info yet is no longer an error. One that holds anything else stays, and is still an error. Tests: scripts/recovery-check.sh adds trash-move-in-flight (a second run purges while the first is held at its move into the trash) and scripts/contract-check.sh adds trash-record-only; both fail on 0.0.9. A move that lands between the purge's own two steps, removing info and then the folder, is not something the Git wrapper can hold; the hour is what keeps a purge from reaching it (#97).
• The walk that measures a worktree never follows a symbolic link, also one that takes a folder's place while residue reads it. It opened each folder by its full path, so a folder, or one above it, that became a symbolic link during the walk was followed, and the sizes and the nested_repo, mount_point and contents checks described another tree. Now it opens each folder from the one above it with O_NOFOLLOW and checks that it is the folder the listing showed; a change makes contents unknown. With a folder swapped for a link to another tree over and over, explain read that tree in 12 of 400 runs before this change and in none of 300 after it, which reported contents unknown in 8. A walk holds at most 22 descriptors at any depth, 176 for scan's 8 workers. It is no slower: explain of a 98,466-entry worktree took 0.68 to 0.81 s, against 0.70 to 0.98 s, with 0.50 to 0.61 s of system CPU against 0.51 to 0.76 s and the same memory, and of a tree 41 folders deep ending in 3,000 folders 0.26 to 0.29 s, against 0.27 to 0.29 s. On NetBSD, OpenBSD and DragonFly, and on macOS 12 and earlier, every folder above the worktree must now be readable, not only searchable (#94).
• Shorter output: --help gives one line per command and option, then examples, and result lines and caveats say the same in fewer words. JSON documents, exit statuses and the lines the Mac app reads are unchanged (part of #77).
• residue's git status calls turn off Git's core.preloadIndex, which runs the index refresh on up to 20 threads. Each call uses a third to a half of the system CPU it did, and can take up to 0.23 s longer on a warm disk. A scan of 10 worktrees used 17 to 19 s of system CPU, down from 35 to 40 s, and took no longer (#75).
• residue notices that a Git call has finished when it exits, not up to 20 ms later. With the launcher bypassed (#81), the wait between one Git call and the next fell from a median of 7 to 17 ms to 4 to 9 ms; through the launcher, from 17 to 34 ms to 11 to 25 ms. It uses no more CPU (#82).
• explain is faster on a large worktree, because its git status call takes about half the time or less. residue reads Git's --ignored=matching list, which does not look inside ignored folders such as node_modules/, and keeps the old --ignored=traditional read for a worktree where the two lists can differ: an untracked folder holding only ignored files, or an ignored folder whose only content is a .git entry or could not be read. Verdicts and the ignored entries in --json are the same as before. On a worktree of 59,888 tracked files, on a loaded machine, git status took a median of 0.37 s, down from 1.0 s, and explain 1.19 s, down from 1.57 s. A worktree that needs the old read runs one more git status, about 0.05 s of user and 0.16 s of system CPU. A scan of 10 worktrees, which reads 8 at once, takes about as long as before: its git status calls used 15.5 s of system CPU, down from 16.0 s (#83).
• residue uses about 5 MB less memory to inspect a large worktree. It keeps only the entries it needs from git ls-files, not the whole list: explain of a 59,888-file worktree peaked at 10.5 to 11.0 MB of residue's own memory, down from 15.3 to 16.4 MB (#78).
• On macOS, each Git call residue makes starts sooner and costs less CPU. residue runs Git's own binary by its path, not through Apple's /usr/bin/git launcher, unless another git comes first on PATH or GIT_EXEC_PATH is set. The wait between one Git call and the next fell from a median of 17 to 34 ms to 7 to 17 ms, and explain of a 98,466-entry worktree used 0.17 to 0.23 s of user CPU, down from 0.21 to 0.29 s (#81).
• The Mac app's text is brief: short headings, a sentence or two under each, buttons that name what they do, and every caveat kept beside its action. Restore and Automatic cleanup move the rules on forgetting removal records into a disclosure. Restore shows about 25% fewer words, Automatic cleanup about 30% fewer (#77).
• The Mac app uses less CPU and memory. Moving the pointer over the Worktrees map redraws only its tooltip, a long list of worktrees is drawn only where you can see it, its stylesheet is half the size, and it keeps residue's reports as text instead of parsing them twice (#75).
• On macOS, explain run from inside a worktree no longer names lsof as a process using it: lsof now runs in /. residue counts the running processes with proc_listallpids in place of starting ps, so a snapshot starts one program, not two, and a process whose working directory or files lsof could not read no longer counts as inspected (#85).
• On Linux, the activity check now sees a process that uses a worktree only through a mapped file, such as a native module loaded from its node_modules by a process started elsewhere. It reads each process's /proc/PID/maps beside its working directory, executable and open files. Before, such a worktree could be a removal candidate while the process ran (#84).
• The Mac app uses less CPU while the pointer moves along the bar of files by size in a worktree's detail: only its tooltip moves, and the bar is drawn again only when the pointer reaches another block or leaves it. Over 20 s of pointer movement, WebKit's WebContent process used 0.97 to 1.22 s of user CPU, down from 1.27 to 1.79 s. The app also reads its bundled residue's version once per launch, not each time Automatic cleanup or Restore opens (#95).

Mac app: residue-macos.dmg is the app for macOS 13 or later, on Apple silicon and Intel, with this release's residue inside. It is not notarized yet, so macOS blocks it the first time: the README has the steps to open it.