[AUTOMATED] campaign(decbench): the O0 sweep instrument, and two corrections it forced - #279
Merged
Merged
Conversation
…ections it forced Round 4 targeted the slice the campaign had not attacked: the O0 cases where IDA's GED is 0 and kuna's is not. This lands the reusable parts only -- the instrument, the analysis, and the two documentation errors the round proved. The per-case triage records are deliberately not committed. ## The instrument `scripts/decbench/optsweep.py` -- a bidirectional corpus-scale A/B for any single option flip. `rescore` prices ONE mined case, which is the wrong unit for anything that changes a default: the mining pipeline only ever produces one direction of evidence, mining losses and never pricing the wins the same change destroys. At O0 the net deficit against IDA is 129 functions while every candidate cluster is 300-430 wide, so a change that flips 400 each way moves the scoreboard by zero and nothing upstream of this tool would show it. It reports both directions of movement by name, the summed GED delta per population, a per-project and per-architecture breakdown, and the standing-requirement-8 changed set, with three harness controls built in: functions the pass never fires on must score identically in both arms, banner-only diffs must score identically, and the ON arm must reproduce the published column. A run that fails any of them is not a measurement. Cross-validated against the ad-hoc harness that produced the returndup measurement: 157 of 157 functions agree in both arms on O0 zlib. ## Correction 1: the scored column is `aggressive`, not `reliable` `docs/decbench-loop.md` has been telling every triage agent that the benchmark injected `option listing on`, the equivalent of `--mode reliable`. It does not. `decbench/decompilers/raw/kuna_raw.py`'s `_build_command` builds `kuna decompile-all <bin> --json --max-fn-seconds 120`, appends `--mode` only when `DECBENCH_KUNA_MODE` is set, and appends `--option` only from `config.extra_options`, which is empty. So the benchmark ran `--mode auto` = aggressive on 768 of 803 binaries. An aggressive-only regression is therefore INSIDE the published number, and any triage that assumes `reliable` mis-attributes it. ## Correction 2: DIV-54's evidence never saw O0 `docs/decbench/returndup-o0-measurement.md` is the A/B that PR #246 did not run. #246 flipped `returndup` default-ON on 52,862 functions at O2 and O2-noinline; its table has two rows and neither is O0. Re-run over all three levels, 85,195 scored functions: O0 -192 GED-perfect, O2 +362, O2-noinline +470, total +640. Keep the default -- flipping back buys 192 at O0 and sells 832. But one structural sub-shape carries essentially all the O0 harm and almost none of the O2 benefit: the split that permanently blocks `rule_block_or` from folding a short-circuit chain (-665 at O0, +15 and +20 at the optimized levels). That is the gate PR #285 implements. ## The grounding `docs/decbench/round4-grounding.md` locates the gap before triaging anything. O0 is the only optimisation level where IDA holds more perfect functions than kuna; at O2 and O2-noinline kuna's lower percentage is a denominator effect, since it is scored on 2,343 more functions and holds more perfect at both. Total perfect: kuna 33,146, IDA 32,838. `docs/decbench/features-round4.md` is the ranked menu with its killed list. Zero symptoms fell to the adversarial pass, and four of six filed root causes were materially overturned -- continuing the campaign's most reliable finding exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8UQbPqALzdUQ3cLLjUeKH
mahaloz
force-pushed
the
campaign/decbench-round4
branch
from
August 11, 2026 21:29
4f863d3 to
0c7af88
Compare
mahaloz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[AUTOMATED] campaign(decbench): round-4 — the O0-vs-IDA gap is real, and it is one pass
Round 4 mined the slice the campaign had not attacked: 64 cases from the
--base idapool where IDA's GED is 0 and kuna's is not, at -O0. One triage agent per case, a
clustering pass, then three adversarial refuters per cluster on the top six.
The census overturns the campaign's standing prior
52 of 64 are real kuna structural defects that reproduce on HEAD. The "roughly half are
scoring artifacts" prior DOES NOT hold on this slice: 5 metric artifacts, 2 already
fixed. The gap is real, and it is far more concentrated than any previous round --
roughly 31 of 64 cases involve
returndupor its interaction with condition folding,and a further 10 trace to a single 60-line structuring rule.
Zero symptoms fell to the refuters. Four of six filed root causes were materially
overturned and six of six had a load-bearing claim refuted -- continuing the campaign's
most reliable finding exactly (round 1: 2 of 5 mechanisms right; round 2: 3 of 8
overturned).
Where the gap actually is
O0 is the ONLY optimisation level where IDA holds more perfect functions than kuna
(+274 on the published columns, +129 among functions both tools scored). At O2 and
O2-noinline kuna's lower percentage is a denominator effect -- it is scored on 2,343
more functions than IDA and holds more perfect ones at both. Total perfect: kuna 33,146
against IDA's 32,838.
Half the O0 deficit is recall, and it is one shape: 322 of the 324 O0 functions IDA
scores and kuna produces nothing for are INTERIOR to a kuna entry, all ARM Cortex-M.
That is PR #278.
The other half is structural, and the deficit is 4% of the disagreement volume (3,067
ida-only wins against 3,054 kuna-only). Standing requirement 8 is therefore not hygiene
here, it is the whole measurement.
returndup at O0: the slice DIV-54 never scored
PR #246 flipped
returndupdefault-ON on 52,862 functions at O2 and O2-noinline. Itstable has two rows and neither is O0. Re-run over all three levels, 85,195 scored
functions:
Keep the default -- flipping back buys 192 at O0 and sells 832. But one structural
sub-shape carries essentially all the O0 harm and almost none of the O2 benefit: the
split that permanently blocks
rule_block_orfrom folding a short-circuit chain(-665 perfect at O0, +15 and +20 at the optimized levels). Declining exactly that is a
measured +630 corpus-wide and -2,024 aggregate GED.
Against IDA on the 32,077 both-scored O0 functions, kuna is -53 with the pass on and
+142 with it off. One option decides the sign of the head-to-head.
Two method corrections, both verified
The scored kuna column is
--mode auto(= aggressive on 768 of 803 binaries), NOTreliable:kuna_raw.py's command builder appends--modeonly whenDECBENCH_KUNA_MODE is set and injects no
option listing on. An aggressive-onlyregression is inside the published number.
docs/decbench-loop.mdis corrected here.And DIV-54's evidence never saw O0, which
docs/options.mdrepeats without the caveat.The instrument
scripts/decbench/optsweep.py-- a reusable bidirectional corpus-scale A/B for anyoption flip: both directions of movement by name, per project and architecture, the
requirement-8 changed set, and three harness controls built in. Every decisive finding
this round came from a refuter hand-rolling a fragment of it. Cross-validated against
the ad-hoc harness at 157/157 functions on O0 zlib.
Triage records only in this PR; no engine change. The features it ranks ship separately.
🤖 Generated with Claude Code
https://claude.ai/code/session_01C8UQbPqALzdUQ3cLLjUeKH