Skip to content

status: preserve clean-status proofs across mixed Git writers - #21

Open
ttaylorr-oai wants to merge 126 commits into
codex-unstablefrom
tb/codex/status-preview-unstable
Open

status: preserve clean-status proofs across mixed Git writers#21
ttaylorr-oai wants to merge 126 commits into
codex-unstablefrom
tb/codex/status-preview-unstable

Conversation

@ttaylorr-oai

@ttaylorr-oai ttaylorr-oai commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

The APFS bulk-preload test barrier published its READY file before it
opened the resume FIFO. If the child was descheduled at that point, the
parent could observe READY, write and close the FIFO, and lose the byte
before any reader existed. The child then blocked forever while macOS CI
appeared hung.

Open the resume FIFO before publishing READY. The parent already holds
the FIFO open read/write before it starts status, so the child open does
not block; READY now proves that a reader owns the resume descriptor.

The previous topic tip is already published as
v2.55.0-openai.619.gb264c6f26648, so this correction is additive on top
of that released history rather than rewriting it. Its resulting tree is
byte-identical to the separately audited owner-amended replay.

Validation:

  • owner and final-tip builds
  • APFS focused cases 20-26 and 35-36
  • 100-iteration APFS stress of those cases
  • preload_index_bulk_darwin unit suite (6/6)
  • owner-amended replay audit: 112 nodes, 105 non-merges, 7 merges, and
    122/122 ordered parents
  • additive delivery head merges cleanly into the published unstable tip

ttaylorr-oai and others added 27 commits August 2, 2026 19:29
GitHub exposes manual dispatch only for workflows present on the default
branch.  The controller itself must remain on the orphan meta branch so
master stays identical to upstream.

Add the fixed caller as a dedicated active topic.  It delegates to the
meta-pinned reusable workflow without putting controller code on codex.
The reusable controller reads its publication key from the codex-publish
environment. The default-branch trampoline therefore needs only read
access to Actions and repository contents; it no longer forwards
repository secrets.

Keep this exact workflow in the automation topic so the Actions page can
dispatch the controller pinned to meta.
Codex rebuilds already call a trusted workflow on meta, but their
dispatch-only trampoline cannot run pull request or merge-queue checks.
Topics could therefore reach codex without verifying their review,
branch ownership, or published base.

Run the pinned admission workflow for pull requests targeting codex and
for merge-group checks. Keep rebuild preparation limited to explicit
workflow dispatch, and grant the admission job only read access.
The default-branch trampoline only listened for pull requests against
codex and ran one admission job for every event. A preview pull
request would therefore have no required check, and merge groups could
not distinguish the production and preview lanes.

Listen for both generated outputs, keep the existing production job
and check context unchanged, and add a target-specific preview job that
calls the trusted meta admission workflow with read-only permissions.
The release workflow cross-compiles Linux arm64 on an x64 runner
and skips the smoke test for arm64 POSIX bundles. That prevents
the workflow from executing the Linux artifact it just produced.

Run Linux arm64 on GitHub's arm64 runner and install native
development packages rather than configuring a foreign dpkg
architecture. All matrix entries can then run the existing
distribution smoke test.
Codex consumes Git release artifacts built with the Makefile's default
-O2 flags. The release job compiles each artifact without link-time
optimization.

Add a release-only config.mak.openai and copy it into Git's ignored
config.mak slot before building. Use thin LTO for Clang targets and
automatic LTO for GCC targets, then check GIT-CFLAGS records the
selected flag in every distribution job.

Keeping the setting in config.mak.openai avoids carrying release-only
policy in the upstream Makefile.
LTO can optimize across translation units, but the release job has no
execution profile for the status, diff, clone, fetch, and repack paths
Codex invokes frequently. Git's built-in profile target runs the
1,048-script test suite serially. That is too expensive for every
release target and weights test-harness paths more heavily than the
local workload.

Extend config.mak.openai with GCC and LLVM profile modes. Gate
GIT-CFLAGS on an instrumented build, run a short offline trainer,
merge LLVM raw profiles when needed, and rebuild with profile-use
flags. Each matrix entry runs on its target architecture, so it can
execute the instrumented binary.

Check that final GIT-CFLAGS includes a profile-use flag and increase
the timeout for the second compilation pass. The focused trainer took
about 30 seconds locally; the full macOS build/install validation
completed with thin LTO and LLVM profile-use enabled.
Integrate the current tb/codex/automation topic into the internally distributed codex branch.

Codex-Integration: tb/codex/automation@17738e2cc87ba67ed36cd1ffde983d43e01a5f41
Integrate the current tb/codex/geometric-maintenance-promisor topic into the internally distributed codex branch.

Codex-Integration: tb/codex/geometric-maintenance-promisor@dc2fffc37cead551f8036c9ecab5e52a4cbee37b
Integrate the current tb/codex/release topic into the internally distributed codex branch.

Codex-Integration: tb/codex/release@ba107e0ae8c7142238bb612e530d51d42f0280d3
Integrate the current dr/codex/dugite topic into the internally distributed codex branch.

Codex-Integration: dr/codex/dugite@988cecced01f69765d599a2d6c023406af98fa1b
Integrate the current tb/codex/lto-pgo topic into the internally distributed codex branch.

Codex-Integration: tb/codex/lto-pgo@88fcb4ac12c583bedf010e97ebf83cec240e3120
Clearing CE_FSMONITOR_VALID is not enough to make a provider event
authoritative. With core.trustctime disabled, core.checkStat set to
minimal, and a restored modification time, stat matching can still
accept changed file contents. The same stale match can affect diff,
apply, checkout, and unpack-trees.

Mark a reported entry with the in-memory CE_CONTENT_CHECK_REQUIRED
flag, clear CE_UPTODATE, and discard its cached stat data. Route diff,
apply, checkout, and unpack-trees comparisons through
ie_match_stat_with_content_check(), which calls ie_modified() only for
marked non-gitlinks. Other direct ie_match_stat() callers retain their
existing paths.

Marking an entry up to date clears the transient flag. Ordinary
entries, gitlinks, and unmarked zero-stat entries retain their existing
stat behavior.

Add hook regressions for restored timestamps, diff and status, indexed
apply, checkout, case-insensitive unpacking, unchanged reset, and
ordinary zero-stat behavior in t/t7519-status-fsmonitor.sh.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A filesystem-monitor provider can know that its event history is
incomplete without being able to identify every affected path. Treating
such a response as an ordinary path leaves tracked entries, cached
attributes, and untracked-cache state falsely valid.

Reserve // as a provider-only global invalidation record. It cannot
collide with a worktree-relative path. When the client receives it,
discard cached attribute stacks and untracked-cache state, invalidate
every tracked entry, and mark the fsmonitor extension changed.

Recognize the existing trivial response only when a complete record
consists of a single slash and NUL, newline, or carriage-return
terminators. This prevents the new double-slash record from being
discarded as a trivial response while preserving existing hook forms.

Add a hook regression in t/t7519-status-fsmonitor.sh that changes a
tracked file, restores its timestamp, emits the global marker, and
requires status to report the change. Global invalidation intentionally
scans the tracked index.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An implicitly started fsmonitor daemon inherits its caller's repository
environment and current directory. In a linked worktree, inherited Git
directory, worktree, common-directory, prefix, and index settings can
make the child discover a different repository than the worktree whose
status requested the daemon.

Resolve the requested worktree to its canonical path, start the child
from that directory, and remove repository-addressing variables from
its environment. Keep the existing daemon start command and return an
error if the worktree cannot be resolved.

Add a macOS regression that implicitly starts fsmonitor from a linked
worktree and checks the daemon child's working directory in Trace2.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A pathname monitor cannot establish that every name for a
multiply-linked regular file lies inside its watch cone. Persisting
CE_FSMONITOR_VALID after checking the tracked name can therefore hide a
later write through an unmonitored hardlink.

Use fsmonitor_stat_can_be_valid() to exclude regular files with more
than one link from persistent fsmonitor validity when the platform
reports real link counts. Apply that decision where index refresh,
threaded preload, and diff-files first consume an actual stat. Preserve
CE_UPTODATE for the current process and retain existing persistent
validity for single-link and nonregular entries. Windows and Cygwin
synthesize their link counts, so preserve their existing fsmonitor
behavior without claiming the hardlink guarantee there.

Add a hardlink regression in t/t7519-status-fsmonitor.sh on platforms
with trustworthy stat metadata. It keeps a tracked hardlink outside the
fsmonitor-valid bitmap and checks that a write through an alias outside
the worktree appears in status. The deliberate cost is another stat in
a subsequent process.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Implicit fsmonitor startup resolves a Git command through the execution
path and invokes its start subcommand. An overridden execution path can
therefore select a different Git than the dispatcher that initiated the
query, while adding another launcher between the client and daemon.

Retain the absolute executable path during dispatcher initialization
and expose it only for a real Git dispatcher. Start that executable
directly with fsmonitor--daemon run --detach, then wait until its IPC
socket is listening before accepting startup. Respect the configured
startup timeout, defaulting to 60 seconds, and retain Git-command
lookup when an authoritative dispatcher path is unavailable.

The canonical worktree and sanitized environment established by
S03/P01 remain in place. Update existing startup Trace2 checks
for the direct invocation and add a macOS regression with a fake Git
on the execution path to verify that the original executable is used.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Darwin FSEvents identifies the pathname associated with a hardlink
event, not every name referring to the same inode. Invalidating only
that pathname can leave another tracked hardlink trusted after its
contents change.

Classify the event's absolute path before handling its hardlink flags.
For worktree events, enqueue the provider-wide marker introduced by
S04/P02 so clients content-check the tracked set. Leave gitdir
events in the existing cookie and gitdir handling; otherwise reads of
hardlinked object files could repeatedly trigger global invalidation.

Add a MACOS,HARDLINKS daemon regression that rejects a marker for a
gitdir hardlink, then verifies the marker and correct status for a
changed worktree hardlink with its timestamp restored.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The fsmonitor.startTimeout setting controls how long a client waits
for daemon startup; the daemon's run subcommand does not consume it.
Nevertheless, daemon configuration parsing validates that setting for
every subcommand. A malformed value can consequently kill an implicitly
started daemon before it opens its IPC socket.

Pass a run-specific configuration flag into the callback and skip
startup-timeout parsing only for run. Continue parsing other daemon
settings normally, and preserve strict timeout validation for the
explicit start subcommand.

Add a macOS regression that verifies implicit status still starts the
daemon with a malformed timeout while explicit daemon start rejects
the same configuration.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Changing a .gitattributes file can change how tracked content is
converted without changing the tracked file's stat data. Invalidating
the attribute-file path alone therefore leaves cached conversion state
and affected fsmonitor-valid tracked entries falsely reusable.

Recognize an exact .gitattributes basename in the refresh callback.
Discard cached attribute stacks globally and strongly invalidate only
tracked entries beneath that file's parent directory. A root attribute
file invalidates all tracked entries; tracked entries in sibling
directories remain valid after a nested attribute-file event. Mark the
fsmonitor extension changed only when an entry is invalidated.

Add Clar unit coverage for unrelated paths, nested-directory scope,
root-directory scope, cleared validity, zeroed stat data, and the
content-check marker. Register u-fsmonitor-attributes in both Makefile
and t/meson.build so the suite is included in both build systems.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor socket is selected through the Git directory, so separate
worktree paths can reach the same daemon when they share that directory.
A client in the second worktree can then consume change history from a
daemon that watches the first, incorrectly treating changed files in its
own worktree as clean.

Hash the canonical worktree path together with its device and inode,
plus birth time and generation on Apple platforms. Cache the resulting
64-character SHA-256 identity in the daemon and attach it to every
client query. Check the identity before interpreting the requested
token; reject missing or mismatched bindings with a
cookie-synchronized trivial response that forces the ordinary refresh
path.

The protocol change must also tolerate a daemon left running by an
older Git. Such a daemon treats a bound query as an opaque token and
can return a plausible trivial response. After that exact response,
query an unbound capability command. If the daemon does not advertise
query-v1, serialize replacement through a per-socket restart lock,
stop it, and start the invoking Git executable before retrying the
bound query.

Keep quit, flush, and capability control commands unbound. Bound
daemon lifecycle retries, and fail the query instead of trusting
history when the root cannot be identified or an incompatible daemon
cannot be replaced.

Regression tests cover shared-gitdir worktree aliases, replacement of
a legacy daemon, and acceptance of a daemon that advertises a
capability superset. The replacement test also verifies that the next
status neither refreshes tracked entries nor starts another daemon.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider can report a directory move or modification without naming
a changed .gitattributes file beneath it. Existing directory handling
invalidates tracked entries in the reported cone but can leave cached
attribute stacks describing the old conversion rules.

Discard cached attribute stacks only after directory handling matches
at least one tracked index entry. Record semantic/attributes-cone with
the number of matched entries. An unmatched directory keeps its
existing case-correction and untracked-path fallback without
speculatively flushing attribute state.

Extend t/helper/test-read-cache.c to cache an old attribute, process a
directory event, and require the new attribute value. Add hook
regressions in t/t7519-status-fsmonitor.sh for both an indexed cone
and an unmatched directory, including their distinct Trace2 behavior.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
load_index_extensions() enters its extension loop only when a complete
eight-byte header fits before the trailing checksum. It nevertheless
trusts the declared payload size. An oversized payload can send an
extension parser beyond the mapped extension area, while an incomplete
trailing header is silently ignored.

Compute the checksum boundary once and require the initial offset, each
complete header, and each declared payload to fit within it. Advance
only by checked header and payload sizes, reject partial trailing
headers, and report framing failures as index file corruption.

Add a PERL_TEST_HELPERS regression test that overwrites an FSMN payload
length with 0xffffffff and checks that porcelain-v2 status fails with
the existing corruption diagnostic.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
read_one() allocates a subtree array even for leaf cache-tree nodes.
It also inserts each serialized child through cache_tree_sub(), which
searches children that the writer already emits in increasing order.

Allocate a child array only for non-leaf nodes and append increasing
child names directly. Retain subtree_nr + 2 pointer slots for each
non-leaf, but allocate them without zeroing because only populated
slots are inspected. Keep cache_tree_sub() as the compatibility
fallback for older, unsorted input.

Existing t/t0090-cache-tree.sh tests exercise ordinary cache-tree
decoding. This change adds no dedicated unsorted-input regression or
isolated benchmark.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
add_patterns() rejects pattern files larger than 100 MiB only after
allocating and reading their complete contents. An oversized filesystem
input can therefore exhaust the memory the limit is meant to protect,
or terminate Git when GIT_ALLOC_LIMIT rejects the allocation.

Check the size obtained from fstat() before allocating a filesystem
pattern buffer. Preserve the existing warning, close the descriptor,
and return the existing failure result. Keep the later size check for
index-backed fallback data, whose size is unavailable before it is
read.

Strengthen the existing EXPENSIVE regression by reading its 101 MiB
.gitignore under GIT_ALLOC_LIMIT=1m. The old ordering dies in
xmallocz(); the early rejection preserves the expected warning without
attempting the oversized allocation.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The index extension worker decodes TREE and UNTR serially even though
their parsers read the same immutable mapping and publish to different
index_state fields. An unconditional additional worker would consume
cache-entry workers and interfere with split-index assembly.

Use the bounded framing from S02/P01 to select exactly one
TREE and one UNTR extension. Require extension-offset metadata and at
least four index workers; start an additional TREE worker only when
both payloads reach 1 MiB. Leave at least two cache-entry workers
available and join the TREE worker before unmapping the index.

Keep LINK, duplicate or missing extensions, insufficient workers,
small payloads, and auxiliary-worker creation failures on the existing
serial path. Malformed framing still reports index file corruption.

Allow GIT_TEST_PARALLEL_INDEX_EXTENSIONS to bypass only the payload
threshold. Add a PTHREADS, UNTRACKED_CACHE, and SHA1 regression that
compares parallel and serial status, cache-tree, and untracked-cache
results and checks the extension/parallel/tree-untracked Trace2 marker.
The regression unsets GIT_TEST_SPLIT_INDEX because split indexes
intentionally remain on the serial path.

The eligible path adds one auxiliary worker and its stack. The
benchmark covers the complete series, not this patch in isolation.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/status-preview-unstable branch 2 times, most recently from f26a55f to c1113e5 Compare August 6, 2026 20:03
ttaylorr-oai and others added 25 commits August 8, 2026 00:01
An as-is commit refreshes its index before running the pre-commit
hook. If the hook rewrites a tracked path without changing its size
or mtime, the later in-process status must not certify the earlier
refresh as though it covered the hook.

For a nonsplit index using an IPC provider, perform the initial
refresh through status token closure. After an invoked hook, release
the saved attribute snapshot and reopen the last accepted provider
token before status runs again. Reject unavailable token state and
invalidate the manifest, tracked semantics, and untracked cache
before falling back to a complete refresh.

Pin the post-hook named index before persisting strong invalidation.
Write refreshed state only while its held descriptor, pathname,
stored trailer checksum, and in-memory index still match. Preserve a
hook-replaced index and the existing reread. Split indexes, platforms
without reliable file identity, and non-IPC providers retain their
original initial refresh.

Add prerequisite-guarded scripted cases for successful post-hook
closure without an untracked cache, failed closure with complete
worktree refresh, and a hook that updates the index itself.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S13 depends on S09, S11, and S12. Join those three histories once before applying S13.

S09 and S11 add adjacent cleanup and test-list entries. Release both clean-status and bulk-preload state in read-cache.c, and keep t7529 before t7531 in t/meson.build. S12 requires no further resolution.
A bulk untracked scan cannot reuse its result merely because an ignore
file has familiar stat data. A file or its parent may be replaced while
the scan runs, an absent source may appear, and repeated reads of the
same source may observe different patterns.

Record each source beneath its nearest available anchored parent, along
with its path, symlink policy, presence, size, and blob identity. Check
descriptor and parent identities while capturing an observation, then
resolve the current parent again and compare the actual source bytes at
validation. Coalesce equivalent observations and invalidate the proof
immediately when observations conflict.

Validation uses nonblocking opens, so replacing a source with a FIFO
cannot hang. Equal contents remain acceptable even if the source or its
parent has a different identity. This also preserves an empty
/dev/null and an equivalent empty FIFO; changed or missing contents,
unavailable anchored primitives, and failed parent callbacks invalidate
the proof.

Register the implementation and focused unit suite in both Make and
Meson. The tests cover source and parent replacement, stable absence,
repeated and conflicting observations, missing buffers, no-follow
policy, /dev/null, FIFO replacement, and parent-opener failure.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The ordinary exclude reader opens configured, repository, and
per-directory ignore files by pathname. That is sufficient for a
one-off walk, but a concurrent replacement or newly created ignore file
makes a retained bulk result unsafe.

Attach the optional source proof from S13/P01 to dir_struct
and capture the exact bytes or stable absence observed by
add_patterns(). Preserve symlink-following for standard excludes and
the existing no-follow policy for per-directory .gitignore files.
Visit configured and repository sources even when absent so their later
appearance invalidates the proof.

Mark failed, oversized, short, and index-backed reads unprovable rather
than treating their results as stable filesystem observations. Existing
callers without a proof retain their original opens, error handling,
pattern parsing, and oversized-source guard.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A tracked-file bulk preload can already walk directories that ordinary
status later scans for untracked files. Sharing those observations
requires a complete, independently validated result; a partial list
must never suppress the conventional untracked traversal.

Add an explicit backend capability and optional borrowed destination
for visible paths. Serialize the existing ignore matcher across scan
workers, collapse an untracked directory after its first visible
descendant, and sort the provisional results. Publish them only after
the directory scan and the anchored ignore-source proof both complete.

Reject duplicate paths and discard incomplete or conflicting untracked
results without discarding independently valid tracked observations.
Report completeness, visible-path count, and fallback reason through
Trace2, and release all temporary path and proof state.

No existing backend advertises the new capability and no status caller
requests it at this boundary. Ordinary tracked and untracked behavior
therefore remains unchanged.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The APFS tracked preload encounters untracked entries but ordinarily
discards them. Repeating the entire directory walk to rediscover those
entries costs work even when the existing scan can establish their
visibility.

Teach the APFS backend to advertise visible-path collection and supply
its root-anchored exclude-parent opener. Classify regular files and
symlinks with the normal exclusion machinery, and follow untracked
directories only until their first visible descendant establishes the
single directory entry that normal-mode status reports.

Keep the top-level Git directory and tracked gitlinks out of the
untracked result. Case aliases, embedded repositories, and foreign
mounts invalidate provisional untracked observations while retaining
separately valid tracked results. A replaced directory increments
changed_dirs, so scan-wide validation discards the entire bulk result.

Carry the collapsed-directory root through queued workers. Ordinary
status remains unchanged until a caller explicitly requests the new
backend capability.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Normal-mode status walks the worktree for untracked paths even after
the opted-in APFS preloader has visited the same directories. Reusing
that walk is incorrect if status requests different reporting semantics
or the bulk scan cannot prove that its exclusion sources remain valid.

Request visible paths only for an expanded index in normal untracked
mode without a pathspec, ignored output, or an untracked cache. The
bulk path requires core.preloadIndex, core.preloadIndexBulk, and a
disabled fsmonitor.

Transfer paths only after the bulk scan, worktree-namespace checks, and
anchored exclusion-source proof finish successfully. Complete the
ordinary tracked refresh, clear the borrowed index destination, and
skip the second directory walk only for a complete untracked result.

Retain ordinary traversal for unsupported backends, incomplete proofs,
case aliases, embedded repositories, changed exclusion sources, and
ineligible reporting modes. A failed untracked proof preserves
independently valid tracked results; a changed directory instead
invalidates the entire bulk scan.

Extend the APFS tests to compare output with ordinary status. Cover
collapsed directories, ignored-only and empty directories, special
files, activation guards, case aliases, nested repositories, tracked
submodules, separate Git directories, changed configured and
repository exclusions, a newly appearing configured exclusion,
bidirectional per-directory changes, hard-linked exclusions, and
tracked-file replacement.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A Linux directory scan cannot substitute its results for lstat() when
file identity, timestamps, or mount membership are missing. Depending
on libc's statx declarations would also tie the implementation to the
age of the installed Linux headers.

Define the required statx syscall ABI locally and request complete
basic statistics and a mount identifier. Reject invalid nanosecond
fields, foreign mounts, and device, inode, link-count, owner, size, or
timestamp values that cannot be represented in struct stat.

Register the metadata module in the Make, CMake, and Meson Linux
builds. The native Linux boundary build compiles it with DEVELOPER=1,
but this patch does not select a backend or change the fallback.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A directory name observed during enumeration may resolve outside the
original worktree after a rename, symlink replacement, magic-link
traversal, or mount change. Path-based reopening would then inspect an
unverified namespace.

Introduce descriptor-relative Linux directory-open helpers. Prefer
openat2() with beneath-root resolution and reject symlinks, magic
links, and mount crossings when that syscall is available.

Otherwise reject empty, absolute, dot-dot, and malformed paths. Open
root-relative paths one component at a time and verify each mount.
Keep direct child opens descriptor-relative; directory scanning verifies
their mounts before enumeration.

Register the opening module with Make, CMake, and Meson. The native
Linux boundary build compiles it with DEVELOPER=1, but backend
selection remains unchanged.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A getdents64 record supplies a type hint, not proof that a path is a
regular file or directory. Trusting that hint can hide a tracked
replacement, misapply ignore rules, or report the wrong visible
untracked shape.

Parse record lengths and names within a bounded 1 MiB worker buffer.
Require statx metadata and matching mount identity for tracked paths.
When collecting untracked paths, obtain authoritative metadata before
classifying a wholly untracked file or directory. Use directory hints
only to schedule paths with tracked descendants.

Preserve per-entry fallback for special and multiply linked tracked
files. Stop an untracked subtree once its normal-status witness is
visible, and invalidate uncertain untracked results. Register the
module in all three Linux builds; the native DEVELOPER=1 boundary build
compiles it without selecting the backend.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Holding a directory descriptor establishes what workers read, but does
not prove that the original directory stayed in the worktree. A child
can also move under a different parent while queued. Publishing
observations from either replacement could hide worktree changes.

Capture the complete directory statx observation and converted stat
identity before enumeration. Verify the descriptor mount and recheck
both identities afterward. For queued children, resolve the parent
through the held child descriptor and compare it with the recorded
parent identity.

Add the mount identifier to the shared directory identity and register
the Linux scan module with Make, CMake, and Meson. The native
DEVELOPER=1 boundary build compiles it, while recorded directory
changes prevent the completed scan from being accepted.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Individually anchored descriptors do not establish that the mount
namespace or named worktree root stayed unchanged throughout a scan.
A mount replacement can invalidate otherwise consistent directory
observations.

Accept only ext-family and XFS filesystems with complete root statx
and mount-identity data. Capture /proc/self/mountinfo before the scan,
compare it at completion, and freshly reopen the named worktree root
with O_NOFOLLOW to verify its original complete identity. Probe
openat2() without requiring it.

Register the topology module in all three Linux builds. The native
DEVELOPER=1 boundary build compiles it. Missing namespace proof,
unsupported filesystems, changed mount tables, or replaced roots reject
the result; the retained mount snapshot adds memory and can reject
unrelated namespace changes.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The separately registered Linux metadata, anchored-open, enumeration,
directory-validation, and topology modules cannot safely publish a
physical scan by themselves. They must share the existing bulk backend
lifecycle so every closing check runs before results are accepted.

Assemble those modules into the Linux backend and register the shared
and platform objects with Make, CMake, and Meson. Retain the existing
requirements that core.preloadIndex and core.preloadIndexBulk are
enabled and fsmonitor is disabled. Preserve ordinary preload when a
required syscall, filesystem, mount proof, or closing validation is
unavailable.

Cap Linux scans at 16 workers. Each worker can allocate a 1 MiB
directory buffer; mount snapshots and retained scan results add further
memory. Document ext-family and XFS support and keep directory-type
injection confined to the documented test environment.

Add and register t7532-preload-index-linux.sh with 12 Linux-only cases.
Native Linux validation passes 12/12 in the threaded build and 12/12
in a separate NO_PTHREADS build; CMake and Meson link Git. The suite
compares ordinary status for tracked changes, visible and ignored
paths, false type hints, fallback shapes, and a synchronized child
replacement.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The bulk preloader rejected an active fsmonitor provider, so status
verified ambiguous tracked entries through a separate semantic scan.
Publishing bulk observations or refreshed stat data before the closing
provider query would permit a concurrent change to invalidate a clean
result.

Pass held parent descriptors, basenames, and observed metadata from both
platform walkers to semantic_verify_file_at(). Borrow the captured
provider proof epoch, hash eligible raw-safe files during the bulk walk,
and retain clean states and stat updates provisionally.

After the closing provider query confirms the same epoch, validate all
pending positions before publishing clean states, refreshed stat data,
and fsmonitor-valid bits. Clear provisional state on provider failure,
epoch mismatch, or invalid updates, and retain the existing
complete-refresh fallback.

Choose the provider-backed bulk path from its actual safety conditions,
not from whether semantic history is awaiting adoption. This lets a
trivial daemon response or daemon restart rebuild and close an ordinary
bulk proof, including for a skipHash index, while retaining the complete
proof epoch and closing query.

Require both preload settings, an expanded index, a pending built-in
IPC token, and an eligible whole-worktree request. Keep APFS and Linux
within their platform and filesystem limits. Allocate a bounded hash
buffer and attribute check per content-verification worker, and retain
tracked states and stat updates only until closure.

Extend the APFS and Linux tests with same-size, restored-mtime content
changes. Cover accepted closure, provider failure, dirty status, daemon
token reset with a null-checksum index, and Trace2 evidence of hashing,
deferred publication, and token acceptance or rejection.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A live exclude-source proof uses filesystem identity to keep one
observation coherent. That identity cannot compare equivalent ignore
sources captured by separate status processes: replacing a file with
the same contents changes its identity without changing ignore
semantics.

Hash the existing, validated observations in first-observation order.
Frame the digest with its version, source object format, unique source
count, path, lookup policy, presence, and content identity. Exclude
transient stat identity so an equivalent replacement retains the same
semantic digest.

Extend the existing exclude-proof unit tests to capture independent
proofs across a same-content replacement and repeated observation.
The digest is independently testable without issuing a sidecar or
changing normal exclude-source validation.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A later status invocation cannot safely reuse an empty result unless
its persistent record identifies the exact index and semantic inputs
that the original scan proved. Accepting truncated, ambiguous, or
forward-versioned records would turn a cache miss into a false clean
result.

Define the version-one CSTS encoding and serialize index identity in
fixed-width network-byte-order fields. Bind the index format, entry
count, checksum, HEAD tree, configuration and repository hashes, one
exclude digest, and a bounded builtin-provider token. Protect the
complete record with the repository's object-format checksum.

Reject unknown flags, unsupported index formats, null required object
IDs, invalid token bounds or prefixes, bad checksums, truncation, and
trailing payload. Add fixed-width identity and sidecar unit coverage
for both SHA-1 and SHA-256. Register the new source and unit suite in
both Make and Meson.

This patch defines and tests the format; it neither writes a sidecar
nor changes status dispatch.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A valid sidecar encoding is not sufficient if its named index can be
replaced between proof capture and publication. Publishing that record
would let a later reader associate one clean result with another
index.

Expose the existing index-snapshot open and named-path revalidation
helpers at their first store consumer. Require a durable index identity
on local APFS, a matching index format, entry count, and checksum, and
agreement between the held descriptor and the named index. Encode the
sidecar under its own lockfile and repeat the index checks before
committing that lock.

Register the store unit suite with Make and Meson. Its local-APFS tests
cover successful installation for SHA-1 and SHA-256 and rejection when
the source index is replaced after pinning. Other filesystems fail
closed.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean provider response does not establish that every index entry
can be represented by an empty status result. Conflicted entries,
submodules, sparse entries, intent-to-add entries, and independently
trusted stat state can all require ordinary index processing.

Introduce a single conservative certifiability check. Require a
non-null index checksum and provider-valid ordinary entries. Reject
gitlinks, nonzero stages, intent-to-add, skip-worktree, CE_VALID, and
unrecognized entry flags while allowing the explicitly supported
in-memory flags.

Extend the existing index unit suite to exercise accepted ordinary
entries and each unsupported entry shape. The classifier does not
issue a proof or change status behavior by itself.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An empty status result cannot certify the next invocation unless its
tracked and untracked observations, ignore sources, provider token,
configuration, repository, HEAD, and named index belong to one
completed scan. Publishing a digest before provider-token closure, or
reusing cached replacement-ref state, could issue a false clean proof.

Retain the standard-exclude digest produced by the complete bulk
scan. Keep provider-originated digest state pending until token
closure accepts it, and preserve the accepted digest when consuming
single-use tracked results.

Inspect a fresh, uncached ref store and reject effective replacement
refs. Then fingerprint the held local-APFS index and worktree,
repository paths, locale, and external attribute state.

Issue a sidecar only for the literal, top-level, empty porcelain-v2
command after persistent semantic history, an eligible expanded index,
a complete untracked scan, and the HEAD cache tree all agree. Install
against the pinned index before rolling back its held index lock;
otherwise retain ordinary index-update behavior.

Also enable the preceding external-history checkpoint path only for a
literal normal status with no pathspec. After the full scan, publish a
complete checkpoint for the closed token. A successful save or restore
rolls back the acceleration-only index update, preserving another Git
implementation's physical index namespace. Optional-lock-free and
index-changing commands keep the ordinary path.

Add the focused sidecar integration suite and register its source and
production code with the relevant Make and Meson builds. Cover prior
semantic history, unchanged index contents, exact command shape,
rejection of unsupported exact-sidecar inputs, namespace-specific
external-history restoration across index re-encoding, and failed
checkpoint republication. No early status answer is introduced here.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An installed sidecar cannot be inspected safely by opening an
untrusted adjacent path without bounds. A symbolic link, named pipe,
oversized record, or growing file could redirect the read, block
status, or consume unbounded memory.

Open the named sidecar without following symbolic links and request a
nonblocking descriptor. Accept only a regular file of at most 8192
bytes, read exactly its recorded size, reject an additional byte, and
parse its checksummed contents into caller-owned storage. Clear failed
records and release storage explicitly. Platforms without nonblocking
support fail closed.

Extend the registered store unit suite to cover owned token storage
under both object formats, symbolic links, FIFOs, and an oversized
8193-byte record. The loader is testable at this boundary; it does
not yet bypass index deserialization.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Validating a sidecar must recapture standard excludes before status
can trust an empty result. Opening an exclude source that has become
a named pipe may otherwise block the supposedly cheap validation.

Add an explicit nonblocking flag to exclude-source proof creation and
carry it into the existing anchored source-open operation. Reject
unknown flags, request nonblocking captures for sidecar issuance, and
update the existing bulk-scan and unit-test callers to pass zero,
preserving their current blocking and symbolic-link policies.

Add a focused FIFO unit test showing that an opted-in proof captures
and validates an empty pipe without waiting. The later early-status
consumer can reuse nonblocking capture without changing ordinary
exclude handling.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An issued clean-status sidecar has no latency benefit while status
still deserializes the index before checking it. Moving the check
earlier is safe only if the recorded proof is revalidated around an
empty builtin-fsmonitor delta.

Attempt the sidecar only for the literal top-level porcelain-v2
command on an eligible main worktree. Load the bounded record, pin the
named local-APFS index, recapture excludes without blocking, and check
configuration, attributes, repository identity, HEAD, and provider
mode. Query the builtin provider directly from the stored token.

Keep the attribute and exclude proofs alive across that query.
Recheck configuration, HEAD, fresh replacement-ref and repository
state, attribute contents and namespace, exclude-source identity, and
both the held and named index before accepting an empty delta. Return
without deserializing index entries only when every check succeeds;
otherwise continue through ordinary status. Unsupported anchored-open
platforms take that ordinary path.

Register the fast-path source with Make and Meson. Extend the existing
sidecar integration suite for read-only hits, dirty worktree shapes,
loose, packed, and custom replacement refs, sidecar and exclude FIFOs,
changed configuration, attributes, HEAD, null-checksum indexes, and
post-query replacement or exclude races.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean-status sidecar is a narrowly scoped proof, not an alternate
index or a general cache. Documenting only its serialized bytes would
hide the full-scan issuance requirement and the revalidation needed
before an empty provider response can answer status.

Document the adjacent sidecar path, local-APFS and main-worktree
eligibility, fixed-width version-one CSTS fields, a checksum using
the repository object-format hash, the separate repository-identity
hash, a bounded builtin-provider token, and the 8192-byte read limit.
Explain why the source index, configuration, repository, HEAD,
attributes, and standard excludes must remain coherent.

Describe completed-scan issuance, persistent provider history, held
index locks, the post-query race fence, nonblocking source opens, and
read-only hits. State that every missing, unsupported, stale,
malformed, or raced proof falls back to ordinary status. Register the
technical document in both the documentation Makefile and Meson.

Also document resumable CSHS history checkpoints separately from CSTS
exact-result sidecars. Specify their namespace binding, complete FSMN,
UNTR/FSUC, and FSCF contents, canonical logical-index digest, bounded
local store, scratch-state validation, publication requirements, and
normal-status-only rollback behavior.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The complete stack has three maximal leaves: S17, S15, and S02. Join them once at the integration tip; S01 is now an ancestor of S05 so that paired FSUC state can safely reuse its recursive UNTR proof.

Retain both independent status bitfields in wt-status.h and every test block at its validated position in t/t7519-status-fsmonitor.sh. The result contains all 105 patches.

Extend the trivial-query recovery case to exercise normal status and require the immediate warm run to perform no tracked lstat work.
The APFS bulk-preload race tests pause status until the test driver
writes a byte to a resume FIFO. The child currently publishes its ready
file before it opens the FIFO. If it is descheduled between those
operations, the parent can observe readiness, write and close its
descriptor, and discard the byte before a reader exists. Status then
blocks forever in strbuf_read_file(), leaving a macOS CI job apparently
hung.

Open the resume FIFO first and read from that descriptor after
publishing readiness. The parent opens the FIFO read/write before
starting status, so the child open cannot block. Readiness now proves a
reader is attached, and the resume byte cannot be lost.

Signed-off-by: Taylor Blau <ttaylorr@openai.com>
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/status-preview-unstable branch 2 times, most recently from abc124f to e106175 Compare August 8, 2026 07:26
@ttaylorr-oai ttaylorr-oai changed the title t7529: prevent a lost resume wakeup status: preserve clean-status proofs across mixed Git writers Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants