Repository navigation
IndexNode 0.13.8
One ghost-aware reader. A ghosted blob always has its content (committed, as uncommitted blocks behind an empty commit, or in Nexis's snapshot while it transitions), so no read may fail because of a ghost state, and no file is excluded from ghosting to work around it.
- The retriever is
RestStore, the only store that talks to Azure;LocalCacheStore,StageZipStore,RedirectStore,SplitStoreandRecordingStoreforward to it, so every read (whole, ranged, conditional, HEAD/length, download, zip open) goes through it. - Identity is content, not ETag. A blob is the same when its length and
idx_sha256(when present) match. A ranged read pinned to an ETag that a ghost transition changed (412) is looked at, and read on under the new ETag (remembered, so later reads of the old ETag are not refused); a conditional read treats the same content under a new ETag as unchanged. Only a 404 on the base blob, or different content (Mismatch), is an error. - States. committed: read it. ghost: hydrate (commit the blocks, no data transfer), then read. transitioning: read the snapshot; when the snapshot has gone, wait for the base blob to settle, hydrate and read. An empty or out-of-range answer is settled and asked again (bounded), so a race with Nexis changing state between our read and our look is not an error.
- HEAD gives the real length of a ghost or transitioning blob (the recorded
ghostd_size); nothing is hydrated by a HEAD. - Folded in:
GhostSafeRead(0.13.4) is now only the pin,ZipOpen(0.13.5) hydrates by a plain whole read, the snapshot fallback of 0.13.5 is part of the one sender;CastCandidatesno longer callsRestStoredirectly. - Tests.
GhostRandomTransitionTests: a simulated Nexis flips a blob between committed, transitioning (its snapshot disappearing) and ghost at random moments while 8 readers run every kind of read in parallel (about 2 s per seed); every read must return the right bytes.OneReaderGuardTests: nothing outside the retriever uses a blob client, the snapshot bookkeeping or ghost hydration. - Ghostignore. The 1,129 per-file
centers.zipprefixes come out of.ghostignore.jsonon every machine that has them (the file is not in this repository); keepindex/jobs/,index/holds/andindex/gpu-ready/. - Groups build on a real store (live 0.13.7 dry run: 3.9 uploads/min at ~263% CPU of 8 cores, no faster than 0.13.6). The faster scan and the parallel uploads did not show because the time is in the member-block range reads, which are round trips. Found:
groups builddid start--threadsuploads at once (default = physical cores), but inside each upload the merged ranges of one centers.zip were read one after the other (the upload's own centers, and each frame chunk's member blocks), so most threads sat waiting on the network. Now the merged ranges of a zip go out together, and the bound is on requests in flight, not threads: at most 64 ranged reads of centers.zip are in flight in the whole process (CentersZip.MaxInFlight). Adjacent blocks of the same zip were already coalesced into one ranged read (gap up to 32 KB, up to 4 MB per read); that is unchanged. - The groups progress line has the breakdown:
groups: checked N of M (r/min, ETA ...); avg per upload: scan X ms, read Y ms (own centers Z ms), score W ms; R requests, B MB; reads in flight n now, peak p; U uploads at once, T threads each. A fake store with 20 ms per read (8 uploads, 216 reads) now takes about 0.8 s instead of about 4.3 s serial, peak 56 reads in flight. No real-store measurement from here: please re-run the dry run and compare thereadfigure. - Member shards are now 0.13.9.