Update the graph when the workspace changes - #979
Draft
paracycle wants to merge 1 commit into
Draft
Conversation
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 18:51
beda794 to
0255914
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 18:53
7665e79 to
2a0811a
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 18:59
0255914 to
2e05f1c
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 18:59
2a0811a to
4fdac7c
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
2 times, most recently
from
August 4, 2026 19:10
c7fde98 to
43bebb4
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
3 times, most recently
from
August 4, 2026 19:50
f839e9a to
2842fab
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 19:50
43bebb4 to
583658c
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 19:54
2842fab to
d4dcd65
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
2 times, most recently
from
August 4, 2026 19:58
96a8e47 to
bbc1a69
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
2 times, most recently
from
August 4, 2026 20:02
772b946 to
2fd98ce
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 20:02
bbc1a69 to
2be9a3e
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 20:06
2fd98ce to
dd31f1d
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 20:06
2be9a3e to
a72da34
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 20:20
dd31f1d to
c9ee992
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
2 times, most recently
from
August 4, 2026 20:25
562ec55 to
1ddcecd
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
2 times, most recently
from
August 4, 2026 20:39
aa9c8df to
51c15f3
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 20:39
1ddcecd to
254366b
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 21:09
51c15f3 to
a9f00eb
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
2 times, most recently
from
August 4, 2026 21:14
5ee43dd to
57aa496
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
2 times, most recently
from
August 4, 2026 21:39
9794016 to
a26f626
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
2 times, most recently
from
August 4, 2026 22:19
93bd725 to
c8d9108
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 22:19
a26f626 to
c3a6b51
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 22:31
c8d9108 to
df872ce
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 22:31
c3a6b51 to
0b67ee3
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 22:42
df872ce to
720e4e6
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 22:42
0b67ee3 to
6e4bbe2
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 22:50
720e4e6 to
d71de2a
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
2 times, most recently
from
August 4, 2026 23:02
adf1a2d to
5f42b74
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 23:02
d71de2a to
9d24c41
Compare
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 23:09
5f42b74 to
a2bbb2c
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 23:09
9d24c41 to
1dd0a1f
Compare
The server built its graph at boot and never looked again, so a file edited after the server started was still answered from the state it had at boot. This reconciles the graph with the workspace before answering a query. A walk records the mtime of every indexable file. Comparing it with the previous walk gives three sets: added or changed files go to the indexer, deleted files leave the graph, and the rest are untouched. The walk decides what gets re-indexed, so it has to see the workspace the way the indexer does. `File.directory?` follows symlinks and the Rust walker does not, which made a link pointing at an ancestor walk the same subtree again and again: one `ln -s .. sub/loop` recorded a single file 32 times, under ever longer paths, until the platform refused. The rule differs by depth, and both halves matter. `Graph#workspace_paths` uses `File.directory?` on the workspace's own children, so a symlinked directory there becomes an explicit root and the Rust walker traverses it. Below that, a symlink is never followed. `index_all` reports opaque messages for a whole batch, so a failure cannot be attributed to a file. The graph cannot answer it either, because a failed update leaves the previous document in place. So a failing batch is halved until each failure sits alone. A batch indexes far faster per file than single calls do: measured on 1081 files with one unreadable among them, 21 calls in 75ms against 1081 calls in 311ms. A file that failed keeps its previous mtime, so the next request tries it again. Recording the new one would call a file the server never read "fresh" for the rest of its life. The same rule covers the initial index, which is why `build_graph` now returns the errors it saw and drops the roots that do not exist: `workspace_paths` names gem directories an install may not have, and those phantom errors would hide the real ones. A directory the walk cannot read contributes no entries, and reading that silence as "every file below here was deleted" erased whole subtrees. Two cases reach it: a directory that cannot be read at all, and one that is readable but not searchable, where `each_child` lists the names and every `lstat` is refused. Both keep their entries. A directory that is really gone still takes its files with it. The rescues name the errors that describe one path, including `ELOOP`, which a self-referential symlink raises from `File.mtime` after `lstat` called it a plain file. A resource failure such as `EMFILE` is deliberately absent: swallowing one would empty the walk, and the refresh would then erase the graph. `Graph#delete_document` is keyed by the document's canonical URI, and the server has a filesystem path. `uri_for` reproduces the mapping in Ruby, and it does so imperfectly: it does not canonicalise, so it misses under a symlinked path, and `URI::File.build` raises for a path containing a space. This is the one place the server needs a URI at all, and closing that gap belongs with the graph API rather than here.
paracycle
force-pushed
the
uk-graph-partial-updates
branch
from
August 4, 2026 23:16
a2bbb2c to
b615dfe
Compare
paracycle
force-pushed
the
uk-add-client-server-for-cli
branch
from
August 4, 2026 23:16
1dd0a1f to
055b037
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: draft
This is not ready for review. The deletion path knowingly carries two defects, described under
"Known gap" below, and they are the reason this work was split out of #869. This pull request should
own the fix before it opens for review.
Goal
Stacked on #869. That pull request adds the client/server infrastructure and serves the graph as a
boot snapshot, so a file edited after the server starts is still answered from the state it had at
boot. This one reconciles the graph with the workspace before answering a query, which makes
rdx query --serveragree withrdx query.Review #869 first. The diff here is only the reconciliation layer.
What changes
A walk records the mtime of every indexable file. Comparing it with the previous walk gives three
sets: added or changed files go to the indexer, deleted files leave the graph, and the rest are
untouched.
Mirroring the indexer
The walk decides what gets re-indexed, so it has to see the workspace the way the indexer does.
File.directory?follows symlinks and the Rust walker does not, so a link pointing at an ancestormade the walk descend into the same subtree repeatedly: one
ln -s .. sub/looprecorded a singlefile 32 times, under paths up to 301 characters.
The rule differs by depth, and both halves matter:
Graph#workspace_pathsusesFile.directory?on the workspace's own children, so a symlinkeddirectory there becomes an explicit root, and the Rust walker traverses an explicit root.
Four tests pin this against the Rust tests that describe the same behaviour.
Attributing an index failure
index_allreports opaque messages for a whole batch, so a failure cannot be attributed to a file.The graph cannot answer it either, because a failed update leaves the previous document in place. A
failing batch is therefore halved until each failure sits alone. A batch indexes far faster per file
than single calls do:
A file that failed keeps its previous mtime, so the next request tries it again. Recording the new
one would call a file the server never read "fresh" for the rest of its life. The same rule covers
the initial index, which is why
build_graphreturns the errors it saw and drops the roots that donot exist:
workspace_pathsnames gem directories an install may not have, and those phantom errorswould hide the real ones.
Not erasing what it cannot see
A directory the walk cannot read contributes no entries, and reading that silence as "every file
below here was deleted" erased whole subtrees. Two cases reach it: a directory that cannot be read at
all, and one that is readable but not searchable, where
each_childlists the names and everylstatis refused. Both keep their entries. A directory that is really gone still takes its fileswith it.
The rescues name the errors that describe one path, including
ELOOP, which a self-referentialsymlink raises from
File.mtimeafterlstatcalled it a plain file. A resource failure such asEMFILEis deliberately absent: swallowing one would empty the walk, and the refresh would thenerase the graph.
Known gap: deletion and URIs
This is what keeps the pull request in draft.
Graph#delete_documentis keyed by the document's canonical URI, and the server has a filesystempath.
uri_forreproduces that mapping in Ruby, and it does so imperfectly:removefinds nothing, andthe deletion silently does nothing. Deleted files keep answering queries until the server restarts.
URI::File.buildraisesURI::InvalidComponentErrorfor a path containing a space, so the firstrefresh that sees a deletion fails the query.
Both are confirmed, not suspected.
This is the only place the server needs a URI at all. Rust builds the document URI from a path it has
already canonicalised and simplified (
path_helpers::resolved, thenUrl::from_file_path), so thecorrect fix belongs at the graph API: either
delete_document_at(path), or an exposed canonicalpath-to-URI function.
Test::Helpers::Context#uri_tocarries the same two lines and the commentTODO: This has to go away once we have a proper URI abstraction, which is the same gap.Tests
test/server/core_test.rbcovers the walk, the attribution and the preservation rules;test/server/integration_test.rbcovers a change and a deletion through the real executable.