Skip to content

[AUTOMATED] feat(p9): ctypes - emit valid per-architecture C type names instead of the Ghidra core-type vocabulary (DIV-75) - #302

Merged
mahaloz merged 1 commit into
mainfrom
feat/ctypes
Aug 16, 2026
Merged

[AUTOMATED] feat(p9): ctypes - emit valid per-architecture C type names instead of the Ghidra core-type vocabulary (DIV-75)#302
mahaloz merged 1 commit into
mainfrom
feat/ctypes

Conversation

@mahaloz

@mahaloz mahaloz commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

The defect

The emitted "C" is not C, and it mixes two vocabularies inside a single function:

void BounceClass::Init(unsigned int a0,...,float8 a5,float8 a6)
{
  int4 v1;            // eax
  float4 v2;          // stack - 0x2c
  unsigned int v3;    // esi     <-- relabelled
  float8 v4;          // mm0     <-- not
  ...

kuna interns its core types under the literal names uint1/int4/float8/float10/code — a verbatim port of upstream's no-<coretypes> fallback branch (sleigh_arch.cc:204), which the real Ghidra application never takes because its Java side supplies its own names over the wire. The printer prints those names.

The DIV-6 realtypes relabel covers only residual TYPE_UNKNOWN, so xunknown4 renders unsigned int while the genuine TYPE_UINT core type uint4 beside it does not. That one early-out is the entire cause of the mixture.

The fix

One speller, at the one chokepoint. declarator_parts already funnels declarations, casts (push_cast_type), prototypes, struct members, array element types and the --json type_to_c_string; p9_emit/kuna_ctypes.rs (core_type_spelling) extends its existing relabel step from TYPE_UNKNOWN to every core type. A user-defined or DWARF-recovered type is untouched — its name is already C.

The rule is a port of Ghidra's DataOrganizationImpl.getIntegerCTypeApproximation: match the type's size against the target's own declared widths, in declaration order, first hit wins. Never a hard-coded 2/4/8 table. Declaration order is exactly what makes it per-architecture — under LP64 both long and long long are 8 bytes and an 8-byte integer must read long, while under ILP32/LLP64 long is 4 and the same size lands on long long. The widths come from #301.

Three cases are decided, not left to fall out of the table:

  • A 1-byte integer is signed char / unsigned char, never bare char — its signedness is implementation-defined, and kuna reserves the char core type for text.
  • code becomes void, so code * reads void *.
  • Floating point is the one place an approximation is unavoidable: an exact width wins, but a width above double with no exact match spells long double. No target has a 10-byte sizeof — the x86 cspecs record 10 as the x87 value width and annotate storage in a comment — so this is an approximation of storage, deliberately the same one the recompile prelude already makes, since the .c and the .h must not disagree.

Integer widths with no C type at all (3/5/6/7, 16-byte) keep undefined<N> and are not widened: (undefined3)x is a 24-bit truncation and (unsigned int)x is not.

Why the printer and not the factory

Renaming the interned core types was rejected on the merits, not just on churn: a core type's id is hash_name(name); Ghidra-style identifiers derive from the first character of the type's name (float8 is what makes fVar1); and the console's C-type parser resolves base types solely through TypeFactory::find_by_name, with no unsigned/long keywords in its grammar — 269 <com> parse line script lines across 87 corpus files feed int4/float8 into exactly that path.

Result

void BounceClass::Init(unsigned int a0,...,double a5,double a6)
{
  int v1;             // eax
  float v2;           // stack - 0x2c
  unsigned int v3;    // esi
  double v4;          // mm0

Invalid-C type tokens in the emitted output, offon:

binary arch off on
bounce.obj (the report) i386 MSVC COFF 301 1
grep x86-64 5,153 11
i386_pie_nl i386 667 0

Every survivor is an undefined3/undefined7 — a width no C type has, kept by design. Whole-binary sweeps on combat.obj, libselinux.so.1, fmt_aarch64 and mips_gp_le32 leave no core-type name at all.

Tests

The per-architecture claim is gated by comparison, not assertion: ctypes_per_arch decompiles x86-64 and i386 in one test and pins that LP64 spells an 8-byte unsigned unsigned long and never reaches long long, while ILP32 spells the same core type long long. Getting that backwards is the single most likely way a size→name table is wrong, and it is invisible on either target alone. A fourth test asserts the headline contract directly — no Ghidra core-type name survives on any of four data models — plus a 9-case unit suite over LP64/ILP32/LLP64 and a two-pass tests/stages/kuna-ctypes.xml.

Gating, and a limit worth naming

Shipped catalog default is off: 42 of the 675 datatest assertions and 43 stage assertions pin the Ghidra spellings, and the parity harness applies no mode — so both corpora are untouched, with no per-test opt-out and no baseline re-pin.

Preset membership is what makes valid C the real default: auto selects aggressive for anything under 500 KiB, i.e. for kuna decompile, decompile-all, decompile-project, the web front-end and the benchmark.

Known limit, recorded in the DIV row rather than left to be discovered: a binary at or above 500 KiB gets reliable, whose overrides are deliberately empty, so it keeps the Ghidra vocabulary. betaflight_STM32F405.elf at 533 KB is exactly such a case. Closing that means flipping the catalog default itself, which needs ~39 corpus opt-outs and is its own PR.

Speed: pure rendering, no measurable cost — interleaved, minimum of 3 on i386_pie_nl: on 1.641 s vs off 1.830 s (the on-arm is faster, i.e. below the noise floor).

Gates

  • make test675/675 PARITY OK
  • make test-stagesPARITY OK (460/460)
  • make rust-test — green locally, 311 targets (internal PRs skip the workspace suite)
  • make check-spec — green; kuna catalog --check — OK

Counters re-derived from the built artifact: live catalog 106, catalog_bytecompat.rs 106, kuna_phases/tests.rs 106 / tiers (23, 47, 36) / },\n 105 / live readers 33, phase_catalog.json 106, docs/options.md 106, xml.rs corpus 204.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JyfU1KXNMWieY7HkFx2YbN

…es instead of the Ghidra core-type vocabulary (DIV-75)

kuna interns its core types under the literal names uint1/int4/float8/float10/
code -- a verbatim port of upstream's no-<coretypes> fallback branch, which the
real Ghidra application never takes because its Java side supplies its own names
over the wire. The printer prints those names, so none of them is valid C.

Worse, the output MIXES vocabularies: the DIV-6 `realtypes` relabel covers only
residual TYPE_UNKNOWN, so `xunknown4` renders `unsigned int` while the genuine
TYPE_UINT core type `uint4` beside it does not. One function declares
`unsigned int v3;` and `int4 v1;` in the same block, which is what was reported.

`p9_emit/kuna_ctypes.rs` extends the same single chokepoint (declarator_parts,
which push_cast_type, type_name_for_decl and the --json type_to_c_string all
route through) to every core type, matching the type's SIZE against the target's
own declared widths in declaration order, first hit wins -- the port of Ghidra's
DataOrganizationImpl.getIntegerCTypeApproximation. Declaration order is what
makes it per-architecture: under LP64 both `long` and `long long` are 8 bytes and
an 8-byte integer must read `long`, while under ILP32/LLP64 `long` is 4 and the
same size lands on `long long`.

Three cases are decided rather than left to fall out of the table: a 1-byte
integer is signed char/unsigned char, never bare `char` (implementation-defined
signedness, and kuna reserves `char` for text); `code` becomes `void`, so
`code *` reads `void *`; and a float wider than `double` with no exact match
spells `long double` -- an approximation of storage, since no target has a
10-byte sizeof and the x86 cspecs record 10 as the x87 VALUE width. Integer
widths with no C type at all (3/5/6/7, 16-byte) keep undefined<N> and are NOT
widened: (undefined3)x is a 24-bit truncation and (unsigned int)x is not.

Renaming the interned core types instead was rejected on the merits: a core
type's id is hash_name(name), Ghidra-style identifiers derive from the first
character of the type's name (float8 is what makes fVar1), and the console's
C-type parser resolves base types solely through TypeFactory::find_by_name,
which 269 corpus script lines feed int4/float8.

Invalid-C type tokens, off -> on: the reported i386 MSVC COFF bounce.obj
301 -> 1, x86-64 grep 5,153 -> 11, i386 i386_pie_nl 667 -> 0. Every survivor is
an undefined3/undefined7.

Gating: the shipped catalog default is off, because 42 datatest and 43 stage
assertions pin the Ghidra spellings and the parity harness applies no mode -- so
both corpora are untouched, with no per-test opt-out and no baseline re-pin.
Preset membership is what makes valid C the actual default rendering: `auto`
selects `aggressive` under 500 KiB. Known limit recorded in the DIV row: a binary
at or above 500 KiB gets `reliable` and keeps the old vocabulary until the
catalog default itself is flipped, which needs the corpus opt-outs and is its own
PR.

Speed: pure rendering, no measurable cost (interleaved, min of 3 on i386_pie_nl:
on 1.641 s vs off 1.830 s -- the on-arm is faster, i.e. below the noise floor).

Gates: make test 675/675 PARITY OK, make test-stages PARITY OK (460/460),
make check-spec green, kuna catalog --check OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JyfU1KXNMWieY7HkFx2YbN
@mahaloz
mahaloz merged commit 2b5117f into main Aug 16, 2026
9 checks passed
@mahaloz
mahaloz deleted the feat/ctypes branch August 16, 2026 00:15
mahaloz added a commit that referenced this pull request Aug 17, 2026
The branch was cut from #302 while main advanced to #316; the engine
PRs in between (spillargtrial #315, stackguard #306, snip-reads #307,
loadguardrange #308, ...) legitimately shift the ghidra-path output, so
CI's merge-ref run measured register leaks 108 vs the pinned 106. Rebase
onto origin/main and re-pin every value to the rebased tree:

  registers 106/64/60 -> 108/58/60, unique 32/2/8 -> 34/4/8,
  mangled 21/13/7 -> 21/6/7, c_lines 241/174/89 -> 243/128/89,
  diff ratios 0.643/0.898/0.811 -> 0.646/0.867/0.811 (band unchanged);
  placeholders/resolvable/traffic unchanged (49/25/17, 24/18/14,
  1477/1003).

The sub_3320 shrink (c_lines -26%, FS_OFFSET gone from its register
set, mangled 13 -> 6) is the stackguard/snip-reads work stripping the
canary sequence -- explainable, not suspicious. Determinism verified:
three consecutive runs of the pins test are measurement-identical.

Gates on the rebased tree: harness release + --include-ignored green,
make test 675/675 PARITY OK, make test-stages PARITY OK, check-spec OK,
kuna-ghidra dev-profile spot check green. The full workspace suite ran
green on this branch pre-rebase and every added commit is already-CI-
green main history, so it is covered by the branch + main CI runs.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC
mahaloz added a commit that referenced this pull request Aug 17, 2026
…the GUI-path quality gap (#317)

* feat(ghidra): ghidra-sim differential test harness — pin the GUI-path quality gap

Extract the decompile_at_e2e MockJava loopback into a shared module
(tests/ghidra_sim/mod.rs — AnswerSource-pluggable pump, wire builders,
session tracer, dual-<function> doc parser, a markup->C flattener that
replicates Java's getC() token cleaning (IllegalCharCppTransformer),
badness scanners, line-diff metric) and build ghidra-sim v1 on top
(tests/ghidra_sim/oracle.rs): a mock-Java answer source backed by kuna's
own analysis of real vendored ELFs — bootstrap_from_object for
bytes/labels, the real Sleigh re-encoded as wire <inst> docs for
getPcode, and a tspec GENERATED from the loaded Sleigh's
AddrSpaceManager so packed space indices agree by construction.
getMappedSymbols/getExternalRef answer EMPTY at a marked PHASE-3 SEAM.

tests/ghidra_sim_e2e.rs drives the full wire lifecycle (registerProgram
-> setAction -> decompileAt x3 -> flushNative -> repeat -> deregister)
over tests/bug-repro/faillog (sort/grep as an ignored breadth test),
asserts the response-document schema (name/entry echo, markup
opref/varref subset-of ast, 19-query legality + query-legal placement),
and PINS today's Phase-2 reality: per-function raw-register leaks
(106/64/60), Unique tokens (32/2/8), placeholders (49/25/17) of which
the loader already knows names for 24/18/14 (Phase 3 drives to 0),
getC()-mangled tokens (21/13/7, PR-C drives to 0), ghidra-vs-CLI line
diff ratios (0.64/0.90/0.81 floors), getPcode traffic 1477/1003, and
getMappedSymbols == 0 (Phase 3 flips to >=1). Pins move only with the
provider/emitter change that earns them.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC

* ci(ghidra): run the ghidra-sim harness in the gates job + make test-ghidra

Add a `gates`-job step running `cargo test -p kuna-ghidra --release --
--include-ignored` (with the standard specs-skip canary grep) after the
catalog checks: the workspace suite is skipped on internal PRs, and a
GUI-path regression is exactly what the ghidra-sim pins exist to catch
pre-merge. Cost: ~1-2 min of compile over the release deps `make
binaries` already built, ~2 s of test runtime. `make test-ghidra` is the
same run locally.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC

* docs(ghidra): live-smoke rig + testing-strategy section for the harness

integrations/ghidra/live-smoke/: a manual/dev pyghidra rig that swaps
DecompileProcessFactory.exepath to kuna_ghidra inside a real Ghidra,
decompiles the same functions with both cores, and writes a side-by-side
report with the same badness-scanner counts the in-tree harness pins
(README covers the offline-pyghidra setup and the getC()-vs-GUI-panel
rendering distinction). docs/ghidra-integration.md §11 rewritten around
the shipped harness: what it covers, how to run it, where the pins live.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC

* test(ghidra): harden the ghidra-sim pins per adversarial review

- Diff-ratio pins are now a BAND: ceilings [0.70,0.95,0.90] join the
  floors (a markup regression that makes the GUI text worse now fails
  instead of saturating), the vacuous `<= 1.0` assert is gone, and the
  flattened-C normalized line count is pinned per target (241/174/89) --
  the assert that actually catches a <break>-token collapse. The pin
  comment now decomposes the ratio (Phase-3 symbol gap + option-preset
  skew until setOptions is wired) so nobody chases 0 with symbol work.
- decompile_cli now routes through the SHARED per-function step
  (kuna_console::decompile_step::decompile_one, DIV-66) with the
  error-noreturn CALL_RETURN flow overrides built exactly as
  kuna_console::project does -- the previous direct drive call silently
  dropped them (latent: faillog has no error() sites, and every faillog
  pin re-measured IDENTICAL; sort/grep-class fixtures would have
  diverged from the real CLI).
- Assert strength: warnings frames pinned trim-empty on registerProgram
  and decompileAt (no substring blocklists); markup oprefs AND varrefs
  pinned non-empty per class; ast varnode refs collected from the
  <varnodes> child only (all Java's buildVarnodeRefs keys -- an op
  operand ref undeclared there must not launder the subset assert);
  the name echo compares against the sim's code_label (which consults
  label_overrides) rather than the raw program lookup.
- CI: the gates-job harness step tees its output to a log before the
  canary grep, so a FAILING pin still prints its diagnostics (the old
  command substitution aborted under bash -e before any echo).
- live-smoke docs/script: kuna_ghidra is not built by `make binaries`;
  point at `cargo build --release -p kuna-ghidra`.

Tests/CI/docs-only diff: the parity gates are untouched by construction.
`cargo test -p kuna-ghidra --release -- --include-ignored` fully green;
`make check-spec` green. No pinned value moved.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC

* test(ghidra): re-measure the faillog pins on main @ 813ee13 (rebase)

The branch was cut from #302 while main advanced to #316; the engine
PRs in between (spillargtrial #315, stackguard #306, snip-reads #307,
loadguardrange #308, ...) legitimately shift the ghidra-path output, so
CI's merge-ref run measured register leaks 108 vs the pinned 106. Rebase
onto origin/main and re-pin every value to the rebased tree:

  registers 106/64/60 -> 108/58/60, unique 32/2/8 -> 34/4/8,
  mangled 21/13/7 -> 21/6/7, c_lines 241/174/89 -> 243/128/89,
  diff ratios 0.643/0.898/0.811 -> 0.646/0.867/0.811 (band unchanged);
  placeholders/resolvable/traffic unchanged (49/25/17, 24/18/14,
  1477/1003).

The sub_3320 shrink (c_lines -26%, FS_OFFSET gone from its register
set, mangled 13 -> 6) is the stackguard/snip-reads work stripping the
canary sequence -- explainable, not suspicious. Determinism verified:
three consecutive runs of the pins test are measurement-identical.

Gates on the rebased tree: harness release + --include-ignored green,
make test 675/675 PARITY OK, make test-stages PARITY OK, check-spec OK,
kuna-ghidra dev-profile spot check green. The full workspace suite ran
green on this branch pre-rebase and every added commit is already-CI-
green main history, so it is covered by the branch + main CI runs.

[AUTOMATED]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AfGYKwUWvPYYj1Aw7tcLhC

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant