Fix the three defects blocking natspec spec publication - #148
Open
shellygr wants to merge 5 commits into
Open
Conversation
`temp_certora_file` yields a path already relative to the project root, so it
carries the `certora/` segment (its docstring: "callers use it verbatim (no
`certora/` prefixing)"). `ConfigurationBuilder._build_to` prefixed it a second
time and yielded `<root>/certora/certora/run_<uid>.conf`, a path nothing ever
wrote to, so every natspec typecheck died with:
read_from_conf_file: /tmp/tmpXXXXXXXX/certora/certora/run_YYYY.conf: not found
`publish` is gated on a passing typecheck, so greenfield natspec runs could
never emit a spec: the authoring agents produced complete, judge-approved CVL,
burned their remaining turns retrying, and gave up. The pipeline still wrote
the interface and the stub, which made the failure look like missing specs
rather than a broken backend.
The sibling call in the same `with` block, `with_verify(spec_file=...)`, already
uses the yielded path verbatim.
Adds a regression test asserting `build_to` yields a path that exists and sits
under exactly one `certora` segment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`with_verify` re-prefixed `certora/` onto a spec path that already carried it,
so the conf's `verify` attribute read
<Contract>:certora/certora/generated_<uid>.spec
The Certora CLI validates that path against the process CWD (which typecheck.py
sets to the project root), so it rejected every spec with
attribute/flag 'verify': file certora/certora/generated_<uid>.spec not found
This is the same mistake as the conf-location bug in the previous commit, one
layer up: fixing only that one moved the failure from "conf not found" to
"spec not found" without unblocking publish.
Widens the regression test to assert the invariant both bugs broke — every path
the conf hands to the CLI must resolve, from the project root, to a file on
disk. The test now fails on either bug alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Certora's scene assembly requires every entry in the conf's `files` to compile
to bytecode. An interface never does, so one interface entry fails the build
for every spec authored in that session:
Contract IFoo has no bytecode. It may be caused because the contract is
abstract, or is missing constructor code.
The pipeline registers only stubs, but the CVL-authoring agent registered the
interface too — reasonably, since its spec references it and
`register_verification_file` invited "any contract source the spec references".
There is no unregister tool, so a session poisoned itself irrecoverably: the
agent's own diagnosis after its sixth rejected publish was "blocking error is
scene assembly of the pre-registered interface-only file".
The registration also outlived the run that made it. FILES_NS is keyed by
document digest and not by cache namespace, so re-running the same document
under a fresh `--cache-ns` reused the poisoned entry. Hence two guards:
`register` refuses these paths, and `read_all` filters them so entries written
before this commit stay out of the conf.
The interfaces are in the scene regardless, via the stubs' imports. Verified by
running the real typecheck gate against a generated interface + stub with
`rule sanity { assert true; }`: with the interface excluded from `files`, the
gate passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_compile_stub` gates every stub update and trusted solc's exit status. An
`abstract contract` satisfies that happily: solc returns 0 and emits no
bytecode. Certora's scene assembly then rejects the verification unit —
Contract StreamShareSplitter has no bytecode. It may be caused because the
contract is abstract, or is missing constructor code.
— failing every subsequent typecheck in the session. `publish` is gated on the
typecheck, so nothing the CVL author does afterwards can recover: the bad stub
is already in the VFS and the spec is not what is wrong.
This is not hypothetical. A registry agent asked for storage fields and got
back `abstract contract StreamShareSplitter is IStreamShareSplitter`, which the
validator accepted. All four components then produced judge-approved specs that
could never be published, while the artifact dumped at the end was the pristine
concrete stub from generation — so the failure was invisible in the output.
Ask solc for the bytecode via --combined-json and require it to be non-empty.
The rejection message names the two ways a contract ends up without bytecode so
the agent can act on it within its retry loop.
Note the neighbouring case is already safe: a contract that inherits a function
it does not implement is a hard solc error ("should be marked as abstract"), so
the exit-status path catches that one. Only explicit `abstract` slipped through.
Both are pinned by tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Greenfield
tui-natspeccannot publish a spec, and never could.publishis gated on a passing typecheck, and three separate defects sat in that gate, each hidden behind the one before it. The symptom is the same every time: the authoring agents write complete CVL, the feedback judge approves it, every publish attempt is rejected, the agents burn their remaining turns and give up — and the pipeline still writes the interface and the stub, so the run looks like it merely failed to produce specs.1 & 2 — doubled
certora/segmentstemp_certora_fileyields a path already relative to the project root, so it carries thecertora/segment; its docstring says callers use it verbatim, "nocertora/prefixing". TwoConfigurationBuildermethods prefixed it again:typecheck.pyruns the CLI withcwdat the project root andvalidate_readable_fileresolves against the CWD, so both paths were simply missing:They mask each other — fixing the conf location just moves the failure to the second message.
3 — interfaces in the conf's
fileslistWith the paths fixed, the CLI got far enough to assemble a scene and refused:
Every entry in
filesmust compile to bytecode. The pipeline registers only stubs, but the authoring agent also registered the interface — reasonably, since its spec references it andregister_verification_fileinvited "any contract source the spec references". There is no unregister tool, so the session poisoned itself irrecoverably. The agent's own diagnosis, after its sixth rejected publish: "blocking error is scene assembly of the pre-registered interface-only file."The registration also outlives the run that made it:
FILES_NSis keyed by document digest, not by cache namespace, so re-running the same document under a fresh--cache-nsreuses the poisoned entry.Fix
FileRegistrylearns which paths are interfaces:registerrefuses them with an explanation the agent can act on, andread_allfilters them so entries persisted before this commit stay out of the conf. The tool docstring now says not to register interfaces and why. Interfaces reach the scene regardless, through the stubs' imports.Tests
tests/test_natspec_conf_path.pymirrorstypecheck.run_typecheck— materialize a spec, build a conf around it — and asserts the invariant both path bugs broke: every path the conf hands to the CLI must resolve, from the project root, to a file on disk.tests/test_file_registry_non_units.pycovers both guards, including the persisted-entry case.Verified against each code state in the AutoProver container:
origin/masterconf not written at yielded path: …/certora/certora/run_9c1cb9dc.confverify points at a file that was never written: certora/certora/generated_b80726c8.specContract IFoo has no bytecodeThe last row is an end-to-end run of the real typecheck gate — a generated interface + stub and
rule sanity { assert true; }throughtypecheck_spec— which now returns clean.pytest tests/test_natspec_conf_path.py tests/test_file_registry_non_units.py→ 3 passed.Still draft pending a full greenfield pipeline run.