fix(evals): build rust oracle binary once and exec directly to end cargo-run link race#751
Conversation
…rgo-run link race
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request resolves a race condition in the Rust oracle execution during parallel test runs by ensuring the binary is built exactly once and then executed directly, rather than using cargo run repeatedly. It introduces a cross-process file lock (filelock) to serialize the build step, queries the binary path dynamically via cargo metadata, and adds a dedicated test to verify this behavior. The review feedback recommends catching json.JSONDecodeError when parsing subprocess outputs (both for cargo metadata and the oracle payload) and raising a RuntimeError with stdout and stderr to improve error visibility.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| target_dir = Path(json.loads(proc.stdout)[ec.CARGO_META_TARGET_DIR_KEY]) | ||
| return target_dir / ec.CARGO_RELEASE_DIRNAME / ec.RS_ORACLE_BIN |
There was a problem hiding this comment.
When capturing subprocess output and parsing it as JSON, we should catch json.JSONDecodeError and raise a RuntimeError containing both stdout and stderr. This ensures that any underlying cargo or compilation errors are surfaced clearly instead of a context-free JSON decode failure.
| target_dir = Path(json.loads(proc.stdout)[ec.CARGO_META_TARGET_DIR_KEY]) | |
| return target_dir / ec.CARGO_RELEASE_DIRNAME / ec.RS_ORACLE_BIN | |
| try: | |
| target_dir = Path(json.loads(proc.stdout)[ec.CARGO_META_TARGET_DIR_KEY]) | |
| except json.JSONDecodeError as e: | |
| raise RuntimeError( | |
| f"Failed to parse cargo metadata JSON.\nStdout: {proc.stdout}\nStderr: {proc.stderr}" | |
| ) from e | |
| return target_dir / ec.CARGO_RELEASE_DIRNAME / ec.RS_ORACLE_BIN |
References
- When capturing subprocess output and parsing it as JSON, catch json.JSONDecodeError and raise a RuntimeError containing both stdout and stderr to ensure underlying errors are surfaced.
| payload: OraclePayload = json.loads(proc.stdout or "{}") | ||
| return payload |
There was a problem hiding this comment.
When capturing subprocess output and parsing it as JSON, we should catch json.JSONDecodeError and raise a RuntimeError containing both stdout and stderr. This ensures that any underlying execution errors are surfaced clearly instead of a context-free JSON decode failure.
| payload: OraclePayload = json.loads(proc.stdout or "{}") | |
| return payload | |
| try: | |
| payload: OraclePayload = json.loads(proc.stdout or "{}") | |
| except json.JSONDecodeError as e: | |
| raise RuntimeError( | |
| f"Failed to parse oracle payload JSON.\nStdout: {proc.stdout}\nStderr: {proc.stderr}" | |
| ) from e | |
| return payload |
References
- When capturing subprocess output and parsing it as JSON, catch json.JSONDecodeError and raise a RuntimeError containing both stdout and stderr to ensure underlying errors are surfaced.
Greptile SummaryThis PR changes the Rust oracle runner to avoid concurrent
Confidence Score: 5/5Safe to merge with minimal risk. No issues were found in the changed paths. The update preserves configurable Cargo target directories, handles the Windows executable suffix, and adds focused tests for the new runner behavior. No files require special attention.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Test as Rust oracle caller
participant Runner as rust_oracle.py
participant Cargo as cargo metadata/build
participant Lock as .build.lock
participant Binary as rs_oracle binary
Test->>Runner: _run_rust_oracle_payload(target)
Runner->>Cargo: cargo metadata --no-deps --manifest-path Cargo.toml
Cargo-->>Runner: target_directory
Runner->>Lock: acquire FileLock
alt binary missing or sources newer
Runner->>Cargo: cargo build --release -q --manifest-path Cargo.toml
Cargo-->>Binary: writes release binary
end
Runner->>Lock: release FileLock
Runner->>Binary: exec binary target
Binary-->>Runner: JSON payload
Runner-->>Test: OraclePayload
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Test as Rust oracle caller
participant Runner as rust_oracle.py
participant Cargo as cargo metadata/build
participant Lock as .build.lock
participant Binary as rs_oracle binary
Test->>Runner: _run_rust_oracle_payload(target)
Runner->>Cargo: cargo metadata --no-deps --manifest-path Cargo.toml
Cargo-->>Runner: target_directory
Runner->>Lock: acquire FileLock
alt binary missing or sources newer
Runner->>Cargo: cargo build --release -q --manifest-path Cargo.toml
Cargo-->>Binary: writes release binary
end
Runner->>Lock: release FileLock
Runner->>Binary: exec binary target
Binary-->>Runner: JSON payload
Runner-->>Test: OraclePayload
Reviews (2): Last reviewed commit: "fix(evals): resolve rust oracle binary w..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
@greptile review |
…-race # Conflicts: # uv.lock
1 participant
Problem
Six Rust-oracle test files (
test_rust_containment_oracle,test_rust_structure_oracle,test_rust_closure_containment_oracle,test_rust_inheritance_oracle,test_rust_span_oracle,test_rust_external_impl_containment_oracle) each shelled out tocargo runagainst the same sharedevals/oracles/rs_oraclemanifest, so all built/linked the sametarget/release/rs_oraclebinary. Underpytest -n autoon a cold CI runner, concurrent workers raced that link step: one worker exec'd the binary while another relinked it (ETXTBSY), and cargo exited 101.Observed as an intermittent red on the macos-py3.12 runner (passed on the four other runners and locally); a re-run went green, which is the signature of this race rather than a code fault.
Fix
_ensure_oracle_built()builds the syn oracle binary exactly once, serialized across pytest-xdist workers by afilelockcross-process lock, then all callers exec the prebuilt binary directly instead of viacargo run. A prebuilt binary is only ever read afterwards, so concurrent execs are safe._binary_path()resolves the real output path viacargo metadatarather than hardcodingtarget/release/, because a user/global cargo config orCARGO_TARGET_DIRcan redirect it (e.g. the common macOSbuild.target-dir = "target.noindex"Spotlight workaround). Memoized, read-only, no build/link.filelockpromoted from transitive to an explicittestdependency;.build.lockgitignored.Go's oracle uses
go run(per-invocation temp binary), so it has no shared-path race and is unaffected.Tests
test_rust_oracle_build_once.pypins the contract: build once, nevercargo run, exec the binary directly. Verified RED against the oldcargo runcode.target/under 8 workers, across repeated cold-build loops; mtime-rebuild path verified.