[AUTOMATED] feat(p9): ctypes - emit valid per-architecture C type names instead of the Ghidra core-type vocabulary (DIV-75) - #302
Merged
Conversation
…es instead of the Ghidra core-type vocabulary (DIV-75) kuna interns its core types under the literal names uint1/int4/float8/float10/ code -- a verbatim port of upstream's no-<coretypes> fallback branch, which the real Ghidra application never takes because its Java side supplies its own names over the wire. The printer prints those names, so none of them is valid C. Worse, the output MIXES vocabularies: the DIV-6 `realtypes` relabel covers only residual TYPE_UNKNOWN, so `xunknown4` renders `unsigned int` while the genuine TYPE_UINT core type `uint4` beside it does not. One function declares `unsigned int v3;` and `int4 v1;` in the same block, which is what was reported. `p9_emit/kuna_ctypes.rs` extends the same single chokepoint (declarator_parts, which push_cast_type, type_name_for_decl and the --json type_to_c_string all route through) to every core type, matching the type's SIZE against the target's own declared widths in declaration order, first hit wins -- the port of Ghidra's DataOrganizationImpl.getIntegerCTypeApproximation. Declaration order is what makes it per-architecture: under LP64 both `long` and `long long` are 8 bytes and an 8-byte integer must read `long`, while under ILP32/LLP64 `long` is 4 and the same size lands on `long long`. Three cases are decided rather than left to fall out of the table: a 1-byte integer is signed char/unsigned char, never bare `char` (implementation-defined signedness, and kuna reserves `char` for text); `code` becomes `void`, so `code *` reads `void *`; and a float wider than `double` with no exact match spells `long double` -- an approximation of storage, since no target has a 10-byte sizeof and the x86 cspecs record 10 as the x87 VALUE width. Integer widths with no C type at all (3/5/6/7, 16-byte) keep undefined<N> and are NOT widened: (undefined3)x is a 24-bit truncation and (unsigned int)x is not. Renaming the interned core types instead was rejected on the merits: a core type's id is hash_name(name), Ghidra-style identifiers derive from the first character of the type's name (float8 is what makes fVar1), and the console's C-type parser resolves base types solely through TypeFactory::find_by_name, which 269 corpus script lines feed int4/float8. Invalid-C type tokens, off -> on: the reported i386 MSVC COFF bounce.obj 301 -> 1, x86-64 grep 5,153 -> 11, i386 i386_pie_nl 667 -> 0. Every survivor is an undefined3/undefined7. Gating: the shipped catalog default is off, because 42 datatest and 43 stage assertions pin the Ghidra spellings and the parity harness applies no mode -- so both corpora are untouched, with no per-test opt-out and no baseline re-pin. Preset membership is what makes valid C the actual default rendering: `auto` selects `aggressive` under 500 KiB. Known limit recorded in the DIV row: a binary at or above 500 KiB gets `reliable` and keeps the old vocabulary until the catalog default itself is flipped, which needs the corpus opt-outs and is its own PR. Speed: pure rendering, no measurable cost (interleaved, min of 3 on i386_pie_nl: on 1.641 s vs off 1.830 s -- the on-arm is faster, i.e. below the noise floor). Gates: make test 675/675 PARITY OK, make test-stages PARITY OK (460/460), make check-spec green, kuna catalog --check OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JyfU1KXNMWieY7HkFx2YbN
mahaloz
added a commit
that referenced
this pull request
Aug 17, 2026
The branch was cut from #302 while main advanced to #316; the engine PRs in between (spillargtrial #315, stackguard #306, snip-reads #307, loadguardrange #308, ...) legitimately shift the ghidra-path output, so CI's merge-ref run measured register leaks 108 vs the pinned 106. Rebase onto origin/main and re-pin every value to the rebased tree: registers 106/64/60 -> 108/58/60, unique 32/2/8 -> 34/4/8, mangled 21/13/7 -> 21/6/7, c_lines 241/174/89 -> 243/128/89, diff ratios 0.643/0.898/0.811 -> 0.646/0.867/0.811 (band unchanged); placeholders/resolvable/traffic unchanged (49/25/17, 24/18/14, 1477/1003). The sub_3320 shrink (c_lines -26%, FS_OFFSET gone from its register set, mangled 13 -> 6) is the stackguard/snip-reads work stripping the canary sequence -- explainable, not suspicious. Determinism verified: three consecutive runs of the pins test are measurement-identical. Gates on the rebased tree: harness release + --include-ignored green, make test 675/675 PARITY OK, make test-stages PARITY OK, check-spec OK, kuna-ghidra dev-profile spot check green. The full workspace suite ran green on this branch pre-rebase and every added commit is already-CI- green main history, so it is covered by the branch + main CI runs. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC
mahaloz
added a commit
that referenced
this pull request
Aug 17, 2026
…the GUI-path quality gap (#317) * feat(ghidra): ghidra-sim differential test harness — pin the GUI-path quality gap Extract the decompile_at_e2e MockJava loopback into a shared module (tests/ghidra_sim/mod.rs — AnswerSource-pluggable pump, wire builders, session tracer, dual-<function> doc parser, a markup->C flattener that replicates Java's getC() token cleaning (IllegalCharCppTransformer), badness scanners, line-diff metric) and build ghidra-sim v1 on top (tests/ghidra_sim/oracle.rs): a mock-Java answer source backed by kuna's own analysis of real vendored ELFs — bootstrap_from_object for bytes/labels, the real Sleigh re-encoded as wire <inst> docs for getPcode, and a tspec GENERATED from the loaded Sleigh's AddrSpaceManager so packed space indices agree by construction. getMappedSymbols/getExternalRef answer EMPTY at a marked PHASE-3 SEAM. tests/ghidra_sim_e2e.rs drives the full wire lifecycle (registerProgram -> setAction -> decompileAt x3 -> flushNative -> repeat -> deregister) over tests/bug-repro/faillog (sort/grep as an ignored breadth test), asserts the response-document schema (name/entry echo, markup opref/varref subset-of ast, 19-query legality + query-legal placement), and PINS today's Phase-2 reality: per-function raw-register leaks (106/64/60), Unique tokens (32/2/8), placeholders (49/25/17) of which the loader already knows names for 24/18/14 (Phase 3 drives to 0), getC()-mangled tokens (21/13/7, PR-C drives to 0), ghidra-vs-CLI line diff ratios (0.64/0.90/0.81 floors), getPcode traffic 1477/1003, and getMappedSymbols == 0 (Phase 3 flips to >=1). Pins move only with the provider/emitter change that earns them. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC * ci(ghidra): run the ghidra-sim harness in the gates job + make test-ghidra Add a `gates`-job step running `cargo test -p kuna-ghidra --release -- --include-ignored` (with the standard specs-skip canary grep) after the catalog checks: the workspace suite is skipped on internal PRs, and a GUI-path regression is exactly what the ghidra-sim pins exist to catch pre-merge. Cost: ~1-2 min of compile over the release deps `make binaries` already built, ~2 s of test runtime. `make test-ghidra` is the same run locally. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC * docs(ghidra): live-smoke rig + testing-strategy section for the harness integrations/ghidra/live-smoke/: a manual/dev pyghidra rig that swaps DecompileProcessFactory.exepath to kuna_ghidra inside a real Ghidra, decompiles the same functions with both cores, and writes a side-by-side report with the same badness-scanner counts the in-tree harness pins (README covers the offline-pyghidra setup and the getC()-vs-GUI-panel rendering distinction). docs/ghidra-integration.md §11 rewritten around the shipped harness: what it covers, how to run it, where the pins live. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC * test(ghidra): harden the ghidra-sim pins per adversarial review - Diff-ratio pins are now a BAND: ceilings [0.70,0.95,0.90] join the floors (a markup regression that makes the GUI text worse now fails instead of saturating), the vacuous `<= 1.0` assert is gone, and the flattened-C normalized line count is pinned per target (241/174/89) -- the assert that actually catches a <break>-token collapse. The pin comment now decomposes the ratio (Phase-3 symbol gap + option-preset skew until setOptions is wired) so nobody chases 0 with symbol work. - decompile_cli now routes through the SHARED per-function step (kuna_console::decompile_step::decompile_one, DIV-66) with the error-noreturn CALL_RETURN flow overrides built exactly as kuna_console::project does -- the previous direct drive call silently dropped them (latent: faillog has no error() sites, and every faillog pin re-measured IDENTICAL; sort/grep-class fixtures would have diverged from the real CLI). - Assert strength: warnings frames pinned trim-empty on registerProgram and decompileAt (no substring blocklists); markup oprefs AND varrefs pinned non-empty per class; ast varnode refs collected from the <varnodes> child only (all Java's buildVarnodeRefs keys -- an op operand ref undeclared there must not launder the subset assert); the name echo compares against the sim's code_label (which consults label_overrides) rather than the raw program lookup. - CI: the gates-job harness step tees its output to a log before the canary grep, so a FAILING pin still prints its diagnostics (the old command substitution aborted under bash -e before any echo). - live-smoke docs/script: kuna_ghidra is not built by `make binaries`; point at `cargo build --release -p kuna-ghidra`. Tests/CI/docs-only diff: the parity gates are untouched by construction. `cargo test -p kuna-ghidra --release -- --include-ignored` fully green; `make check-spec` green. No pinned value moved. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC * test(ghidra): re-measure the faillog pins on main @ 813ee13 (rebase) The branch was cut from #302 while main advanced to #316; the engine PRs in between (spillargtrial #315, stackguard #306, snip-reads #307, loadguardrange #308, ...) legitimately shift the ghidra-path output, so CI's merge-ref run measured register leaks 108 vs the pinned 106. Rebase onto origin/main and re-pin every value to the rebased tree: registers 106/64/60 -> 108/58/60, unique 32/2/8 -> 34/4/8, mangled 21/13/7 -> 21/6/7, c_lines 241/174/89 -> 243/128/89, diff ratios 0.643/0.898/0.811 -> 0.646/0.867/0.811 (band unchanged); placeholders/resolvable/traffic unchanged (49/25/17, 24/18/14, 1477/1003). The sub_3320 shrink (c_lines -26%, FS_OFFSET gone from its register set, mangled 13 -> 6) is the stackguard/snip-reads work stripping the canary sequence -- explainable, not suspicious. Determinism verified: three consecutive runs of the pins test are measurement-identical. Gates on the rebased tree: harness release + --include-ignored green, make test 675/675 PARITY OK, make test-stages PARITY OK, check-spec OK, kuna-ghidra dev-profile spot check green. The full workspace suite ran green on this branch pre-rebase and every added commit is already-CI- green main history, so it is covered by the branch + main CI runs. [AUTOMATED] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
The emitted "C" is not C, and it mixes two vocabularies inside a single function:
kuna interns its core types under the literal names
uint1/int4/float8/float10/code— a verbatim port of upstream's no-<coretypes>fallback branch (sleigh_arch.cc:204), which the real Ghidra application never takes because its Java side supplies its own names over the wire. The printer prints those names.The DIV-6
realtypesrelabel covers only residualTYPE_UNKNOWN, soxunknown4rendersunsigned intwhile the genuineTYPE_UINTcore typeuint4beside it does not. That one early-out is the entire cause of the mixture.The fix
One speller, at the one chokepoint.
declarator_partsalready funnels declarations, casts (push_cast_type), prototypes, struct members, array element types and the--jsontype_to_c_string;p9_emit/kuna_ctypes.rs (core_type_spelling)extends its existing relabel step fromTYPE_UNKNOWNto every core type. A user-defined or DWARF-recovered type is untouched — its name is already C.The rule is a port of Ghidra's
DataOrganizationImpl.getIntegerCTypeApproximation: match the type's size against the target's own declared widths, in declaration order, first hit wins. Never a hard-coded 2/4/8 table. Declaration order is exactly what makes it per-architecture — under LP64 bothlongandlong longare 8 bytes and an 8-byte integer must readlong, while under ILP32/LLP64longis 4 and the same size lands onlong long. The widths come from #301.Three cases are decided, not left to fall out of the table:
signed char/unsigned char, never barechar— its signedness is implementation-defined, and kuna reserves thecharcore type for text.codebecomesvoid, socode *readsvoid *.doublewith no exact match spellslong double. No target has a 10-bytesizeof— the x86 cspecs record 10 as the x87 value width and annotate storage in a comment — so this is an approximation of storage, deliberately the same one the recompile prelude already makes, since the.cand the.hmust not disagree.Integer widths with no C type at all (3/5/6/7, 16-byte) keep
undefined<N>and are not widened:(undefined3)xis a 24-bit truncation and(unsigned int)xis not.Why the printer and not the factory
Renaming the interned core types was rejected on the merits, not just on churn: a core type's id is
hash_name(name); Ghidra-style identifiers derive from the first character of the type's name (float8is what makesfVar1); and the console's C-type parser resolves base types solely throughTypeFactory::find_by_name, with nounsigned/longkeywords in its grammar — 269<com> parse linescript lines across 87 corpus files feedint4/float8into exactly that path.Result
Invalid-C type tokens in the emitted output,
off→on:bounce.obj(the report)grepi386_pie_nlEvery survivor is an
undefined3/undefined7— a width no C type has, kept by design. Whole-binary sweeps oncombat.obj,libselinux.so.1,fmt_aarch64andmips_gp_le32leave no core-type name at all.Tests
The per-architecture claim is gated by comparison, not assertion:
ctypes_per_archdecompiles x86-64 and i386 in one test and pins that LP64 spells an 8-byte unsignedunsigned longand never reacheslong long, while ILP32 spells the same core typelong long. Getting that backwards is the single most likely way a size→name table is wrong, and it is invisible on either target alone. A fourth test asserts the headline contract directly — no Ghidra core-type name survives on any of four data models — plus a 9-case unit suite over LP64/ILP32/LLP64 and a two-passtests/stages/kuna-ctypes.xml.Gating, and a limit worth naming
Shipped catalog default is
off: 42 of the 675 datatest assertions and 43 stage assertions pin the Ghidra spellings, and the parity harness applies no mode — so both corpora are untouched, with no per-test opt-out and no baseline re-pin.Preset membership is what makes valid C the real default:
autoselectsaggressivefor anything under 500 KiB, i.e. forkuna decompile,decompile-all,decompile-project, the web front-end and the benchmark.Known limit, recorded in the DIV row rather than left to be discovered: a binary at or above 500 KiB gets
reliable, whose overrides are deliberately empty, so it keeps the Ghidra vocabulary.betaflight_STM32F405.elfat 533 KB is exactly such a case. Closing that means flipping the catalog default itself, which needs ~39 corpus opt-outs and is its own PR.Speed: pure rendering, no measurable cost — interleaved, minimum of 3 on
i386_pie_nl: on 1.641 s vs off 1.830 s (the on-arm is faster, i.e. below the noise floor).Gates
make test— 675/675 PARITY OKmake test-stages— PARITY OK (460/460)make rust-test— green locally, 311 targets (internal PRs skip the workspace suite)make check-spec— green;kuna catalog --check— OKCounters re-derived from the built artifact: live catalog 106,
catalog_bytecompat.rs106,kuna_phases/tests.rs106 / tiers (23, 47, 36) /},\n105 / live readers 33,phase_catalog.json106,docs/options.md106,xml.rscorpus 204.🤖 Generated with Claude Code
https://claude.ai/code/session_01JyfU1KXNMWieY7HkFx2YbN