Skip to content

IndexNode 0.13.8

Choose a tag to compare

@github-actions github-actions released this 21 Nov 07:42

One ghost-aware reader. A ghosted blob always has its content (committed, as uncommitted blocks behind an empty commit, or in Nexis's snapshot while it transitions), so no read may fail because of a ghost state, and no file is excluded from ghosting to work around it.

  • The retriever is RestStore, the only store that talks to Azure; LocalCacheStore, StageZipStore, RedirectStore, SplitStore and RecordingStore forward to it, so every read (whole, ranged, conditional, HEAD/length, download, zip open) goes through it.
  • Identity is content, not ETag. A blob is the same when its length and idx_sha256 (when present) match. A ranged read pinned to an ETag that a ghost transition changed (412) is looked at, and read on under the new ETag (remembered, so later reads of the old ETag are not refused); a conditional read treats the same content under a new ETag as unchanged. Only a 404 on the base blob, or different content (Mismatch), is an error.
  • States. committed: read it. ghost: hydrate (commit the blocks, no data transfer), then read. transitioning: read the snapshot; when the snapshot has gone, wait for the base blob to settle, hydrate and read. An empty or out-of-range answer is settled and asked again (bounded), so a race with Nexis changing state between our read and our look is not an error.
  • HEAD gives the real length of a ghost or transitioning blob (the recorded ghostd_size); nothing is hydrated by a HEAD.
  • Folded in: GhostSafeRead (0.13.4) is now only the pin, ZipOpen (0.13.5) hydrates by a plain whole read, the snapshot fallback of 0.13.5 is part of the one sender; CastCandidates no longer calls RestStore directly.
  • Tests. GhostRandomTransitionTests: a simulated Nexis flips a blob between committed, transitioning (its snapshot disappearing) and ghost at random moments while 8 readers run every kind of read in parallel (about 2 s per seed); every read must return the right bytes. OneReaderGuardTests: nothing outside the retriever uses a blob client, the snapshot bookkeeping or ghost hydration.
  • Ghostignore. The 1,129 per-file centers.zip prefixes come out of .ghostignore.json on every machine that has them (the file is not in this repository); keep index/jobs/, index/holds/ and index/gpu-ready/.
  • Groups build on a real store (live 0.13.7 dry run: 3.9 uploads/min at ~263% CPU of 8 cores, no faster than 0.13.6). The faster scan and the parallel uploads did not show because the time is in the member-block range reads, which are round trips. Found: groups build did start --threads uploads at once (default = physical cores), but inside each upload the merged ranges of one centers.zip were read one after the other (the upload's own centers, and each frame chunk's member blocks), so most threads sat waiting on the network. Now the merged ranges of a zip go out together, and the bound is on requests in flight, not threads: at most 64 ranged reads of centers.zip are in flight in the whole process (CentersZip.MaxInFlight). Adjacent blocks of the same zip were already coalesced into one ranged read (gap up to 32 KB, up to 4 MB per read); that is unchanged.
  • The groups progress line has the breakdown: groups: checked N of M (r/min, ETA ...); avg per upload: scan X ms, read Y ms (own centers Z ms), score W ms; R requests, B MB; reads in flight n now, peak p; U uploads at once, T threads each. A fake store with 20 ms per read (8 uploads, 216 reads) now takes about 0.8 s instead of about 4.3 s serial, peak 56 reads in flight. No real-store measurement from here: please re-run the dry run and compare the read figure.
  • Member shards are now 0.13.9.