Release v0.4.0

Upgrading from v0.3.0

Two changes in this release alter files that oinkie generates. Neither affects
similarity scores, but both matter if you keep a working directory between runs.

The -B/--binary-type flag is gone. oinkie extract accepted it but never
read it, so passing -B llvm silently did Ghidra parsing anyway. Scripts that
pass the flag will now fail with an unknown-argument error; remove it. Backend
selection lives on oinkie lift --lifter-type, which is the option that has
always actually chosen a backend.

Birthmark file names have changed again. oinkie extract derives the output
name as {file_stem}_{hash}.json, and the hash prefix widened from 8 to 16 hex
characters (#30) so that the disambiguator carries 64 bits instead of 32. At 32
bits a collision became likely at roughly 77,000 files, and a collision was
silent — one binary's birthmark simply overwrote another's.

Names from v0.3.0 therefore do not match those from v0.4.0. On the first run
with --skip, every input is re-extracted. That is expected, not an error.
Older files remain valid inputs to compare and reaggregate; the only cost of
keeping them is that the same binary ends up present under two names.

Similarity CSVs record the pair in the correct order. The left record used
to hold the second birthmark and right the first (#29), so a --skip rerun
read the pair back in the opposite order from a fresh run. The score was never
affected — the metrics are symmetric — but results.csv could not be relied on
to say which file was on which side.

Only the writer changed. Similarity CSVs written by v0.3.0 or earlier still
carry the old ordering and will keep being read reversed, and mixing old and new
files in one directory means the interpretation varies per file. Regenerate the
directory to get consistent results.

Shell completions moved. They are now generated under assets/completions,
one directory per shell, with conventional file names (oinkie, oinkie.elv,
oinkie.fish, oinkie.ps1, _oinkie). Packaging scripts that referenced the
old completions directory need updating.

fc-* analyses produced a false positive. The symbol lookup compared the
call operand "(ram, 0x100000480, 8)" against a symbol table keyed
"0x100000480", so every call was discarded and every fc-* birthmark came out
empty (#40). Two empty sets are identical, so any two programs compared as a
perfect match through fc-freq-cosine, fc-set-dice, fc-freq-euclidean,
fc-set-jaccard, fc-seq-levenshtein, fc-seq-lcs, fc-set-simpson and
fc-freq-weightedjaccard.

Any conclusion drawn from an fc-* comparison on v0.3.0 or earlier should be
treated as unreliable and the comparison re-run. Birthmark files extracted with
fc-* are empty and carry no information; regenerate them.

Extracting an fc-* birthmark from a program that calls nothing is now an
error (#44). It used to succeed and write an empty birthmark, which is the
shape of the bug above: empty birthmarks compare as a perfect match, so the
output was worse than no output. oinkie extract -b fc-set on such a program
now stops with

bin/sample: no operation is a call, so every fc-* birthmark of it would be
empty -- and two empty birthmarks score as a perfect match. Either the program
really calls nothing, or oinkie's reader for ghidra-pcode does not recognise
that representation's call operations

The op-* families are unaffected, and so is any program that makes at least
one call — which, for a lifted binary, means every realistic input.

k-grams are named the same way everywhere, and it is the library's way
(#25). The command line had invented two spellings of its own, neither of them
the one the documentation and the file format use:

	v0.3.0	v0.4.0
extract -b	op-tri-gram-set	op-3gram-set
run -a	op3gram-set-dice	op-3gram-set-dice

Both came from hand-written ValueEnums whose command-line names were derived
from Rust identifiers, so OpTriGramSet became op-tri-gram-set and
Op3gramSetDice became op3gram-set-dice. The library has always parsed
op-3gram-set, and oinkie info has always printed op-3gram-set — which
-b then refused. Scripts passing a k-gram to -b or -a need updating;
fc-*, op-seq, op-set, op-freq and their analyses are unchanged.

Two things follow from deleting the hand-written lists. Any k now works:
--analysis op-12gram-set-jaccard and -b op-12gram-set run, where -a
stopped at k=6. And -b and -a now agree on what exists — -b reached
k=8 while -a reached k=6, so three k-gram families could be extracted but not
analysed in one step.

oinkie info and shell completion list up to k=8. That bound is only what gets
suggested; it is not what parses.

A half-written similarity file is no longer read as a score of zero.
compare --skip and run --skip reuse the CSVs an earlier run left behind. A
file holding the pair but no result, line — what an interrupted run leaves —
was accepted, and the missing score defaulted to 0.0: two programs reported
as having nothing in common, written into results.csv beside real scores,
with the run exiting successfully.

Such a file is now refused, naming it:

Error: Parse error: Result line not found in similarities/00000.csv

A --skip run over a directory containing one used to succeed and will now
stop. Delete the offending file, or drop --skip to recompute. Any
results.csv produced by a --skip run over an interrupted directory should
be treated as unreliable in the same way as the fc-* scores above: a zero
there is indistinguishable from a real one.

reaggregate is unaffected. It recomputes the score from the stored matrix
rather than reading it, and a truncated file has no matrix either — which it
already refuses.

A usage error is no longer announced twice. clap's own message begins
error: , and oinkie printed it behind Error: , so a mistyped flag came back
as Error: error: unexpected argument ... (#62). The prefix is gone from that
path. oinkie's own errors carry no prefix of their own and keep theirs:

$ oinkie run --bogus-flag
error: unexpected argument '--bogus-flag' found

$ oinkie extract no-such-file.json
Error: IO error for no-such-file.json: No such file or directory (os error 2)

Scripts matching on the old Error: error: text need updating. The exit codes
are unchanged: 1 for a usage error, 2 for anything else.

op-*gram-freq birthmarks can be extracted at all. Every non-empty one
failed at write time with

Error: hello_clang_3380b9841726541e.json: JSON error: key must be a string

A k-gram is a list of operations and a JSON object's keys are strings, so
serde_json refused the map outright (#59). op-1gram-freq through
op-8gram-freq — eight of the thirty birthmark types the CLI advertises,
and one of the three shapes a k-gram comes in — could not produce a file, and
with them twenty-four of its eighty analyses could not be reached through
extract + compare.

It looked as though only small k were affected. It was not: a program with
fewer than k operations yields an empty map, and an empty map has no key to
refuse. Any k fails on any program long enough to produce a k-gram.

The frequency map is now written as a list of [kgram, count] pairs, sorted,
so extracting the same program twice writes the same bytes:

{"KgramFreq": [[["CALL", "COPY"], 1], [["COPY", "COPY"], 1]]}

Birthmark files are written in a canonical order. A map and a set have no
order of their own, and oinkie wrote them in hash order — which comes from the
insertion history and the hasher rather than from what the birthmark holds
(#68). Two equal birthmarks could be written differently, and a rustc-hash
bump would relay every file for no reason.

Freq, Set and KgramSet are sorted on the way out now, joining KgramFreq.
Seq and KgramSeq are untouched and must be: their order is the program's,
and sorting them would not canonicalise the file but destroy the birthmark.

Nothing about any file's meaning changes, every existing file still reads, and
scores are unaffected — checked against a build of the previous commit across
nine families. What changes is the byte layout of newly written files, so a
checksum taken over a birthmark from an earlier version will not match one
taken now, and a --skip rerun still skips because it goes by the file
existing rather than by its contents.

A frequency stated more than once is refused, in both shapes. op-freq
and fc-freq used serde's derived map deserializer, which inserts in a loop,
so a birthmark saying {"COPY": 3, "COPY": 5} loaded as 5 and was scored with
nothing said (#66). It is refused now, naming the operation:

COPY: listed more than once; a birthmark names each operation once

Every repeat is refused, including {"COPY": 3, "COPY": 3}, whose counts
agree: a repeated key is a malformed object however the values fall, and
reading it would mean accepting a broken file whenever it was broken
consistently.

Only the frequency shapes changed. A Set repeating an element denotes the
same set and a Seq repeating one is a sequence doing its job — neither can
mean two things, and both still load. Nothing oinkie writes can produce a
repeat in any of them, since every map and set is serialized from a map or a
set, so no file it wrote is affected.

A k-gram listed more than once is refused rather than resolved. The list stands for a
map, so collecting it would let the last pair win: a file saying a k-gram
occurred 3 times and again 5 times would load as 5 and be scored, with nothing
said. Nothing oinkie writes can produce one, since the pairs come from a map.

No existing file changes. op-*gram-seq and op-*gram-set already wrote a
k-gram as a plain list and still do — which is why the map was changed rather
than the k-gram, since making Kgram itself serialize as a string would have
rewritten the two families that work. run -a op-Kgram-freq-* was never
affected: it compares in memory and never writes a birthmark, which is why the
whole thing went unnoticed.

compare stopped making one syscall per byte. Birthmark::try_from and
Program::try_from handed serde_json::from_reader a bare std::fs::File.
serde_json's IoRead takes one byte at a time, so an unbuffered file meant one
read syscall per byte of JSON (#51). Loading a 9 MB birthmark took 2.7 s;
it now takes 15 ms.

compare loads its inputs once per pair, so the loader runs O(n²) times.
Over three pairs of 9 MB birthmarks the whole command went from 22.0 s to
7.8 s, and the kernel time in it from 37.5 s to 0.08 s — the parsing was
never the cost. Output is byte-identical.

Nothing to change on your side, and compare is the only subcommand
affected. extract and run read their programs through AnyProgram::load,
which has read whole files since #45, and neither of them loads a birthmark
from disk. The other half of the fix is for library callers: Program::try_from
and Birthmark::try_from were both slow, and both are public.

An analysis name built from what oinkie info prints now parses. info
lists the algorithms as weighted-jaccard, but run --analysis only accepted
op-freq-weightedjaccard — a name assembled from what the tool itself showed
you was refused (#71). Both spellings work now, and so does
compare --algorithm weighted-jaccard, which was always the only one it took.

The other seven algorithms are single words and were already spelled the same
in both places.

The two halves of an analysis name are no longer split on the last hyphen.
Each hyphen is tried, rightmost first, and the split whose tail names an
algorithm and whose head is a birthmark type wins — so an algorithm added
later may be two words without breaking anything. Every name that parsed
before still parses, and a name whose birthmark half is wrong still reports
that half:

op-nonsense-jaccard  ->  op-nonsense: unknown birthmark type

Library API

These affect code using oinkie as a crate. The command-line interface is
unchanged by them.

AnalysisType::new now returns Result. It validates that the algorithm
operates on the birthmark's shape, which try_from already did and new did
not — so the check could be walked around by constructing the pair directly.
Callers add a ? or handle Error::IncompatibleAnalysis, a new variant.

Analysis names that pair an algorithm with the wrong shape are rejected.
Each algorithm computes over one representation and converts anything else
first, so op-seq-euclidean returned exactly what op-freq-euclidean returns
while its name promised a comparison over sequences, and op-set-levenshtein
scored 0.0 because no conversion exists. The error names the pairing that was
meant:

op-seq-euclidean: euclidean operates on frequency vectors; use op-freq-euclidean

In practice this only changes k-gram names: op-3gram-set-levenshtein and its
kind used to parse and now do not. Every pairing the CLI offers is canonical, so
nothing reachable through --analysis is affected.

More names parse than before. AnalysisType::try_from reads the algorithm
and hands the rest to BirthmarkType::try_from (#38), so the eight fc-*
combinations and the seven *-lcs ones now work, as does any k — previously
fifteen of the sixty-four combinations the CLI offers could not be parsed at all.

Program::new takes the representation. A lifted program now records
which intermediate representation its operations are written in, and the
constructor requires it rather than assuming one. Code building a Program
directly needs the extra argument; reading one from JSON does not, since the
field is defaulted for files written before it existed.

The argument is required rather than defaulted on purpose. A constructor that
filled it in would let a program lifted by one tool be labelled as another's,
which is the confusion the field exists to prevent.

Birthmarks record the representation they were extracted from, and a
comparison across two of them is refused. A birthmark is only meaningful
against another built the same way, and two lifters describe the same
instruction with different operations, so comparing across them would measure
the disagreement between the tools rather than anything about the programs
(#42, #43). Only Ghidra exists today, so nothing currently reachable is
refused.

Metadata gains an ir field, and the birthmark record in a similarity CSV
gains a seventh column for it. The column is appended rather than inserted, so
the positions earlier readers depend on are unchanged, and a six-column record
written by an earlier version still parses.

Op gains a required method, symbol_key. Any crate implementing Op for
its own lifter must add it. It renders the operation's first operand as the
program's symbol table keys it, so a call target can be resolved; returning
None when the operand cannot name a symbol is what keeps an unresolvable
target distinguishable from a lookup that found nothing. Ghidra's
implementation parses the operand through the existing Value type.

Op gains a second required method, is_call. Which operation is a call
was decided in the extractor by comparing the mnemonic against the literal
"CALL", which is P-Code's spelling; the Hex-Rays microcode and Binary Ninja's
LLIL name it differently and some name more than one operation, so any other
lifter would have matched nothing (#44). The question now belongs to the
operation, and so to the lifter that defines it.

The method has no default on purpose. A false default would let a new lifter
compile while recognising no call at all, which produces empty fc-*
birthmarks rather than an error — and empty birthmarks score as a perfect
match. Requiring it makes that a build failure. Ghidra answers for CALL and
CALLIND, but not CALLOTHER, which indexes a table of user-defined
operations rather than naming a function. Nothing changes in what Ghidra
extracts today: CALLIND reaches its target through a register, so no symbol
resolves for it.

Op loses code. It returned the numeric opcode and had no callers: the
birthmark is built from mnemonic, a string. Keeping it would have made every
future lifter invent a numbering for a value nothing reads (#45).

Which operation type reads a lifted file is now decided by the file.
Program<T> was generic, but nothing chose T — every caller wrote
Program<ghidra::Op>, so a file lifted by another tool would have been read
against P-Code's vocabulary. AnyProgram::load reads the ir field first and
picks the reader from it, and Extractor::extract_any and
Comparator::compare_any take what it returns (#45).

A representation with no reader is now refused by name:

foreign.json: no reader for ida-microcode; this build can read ghidra-pcode

Read as P-Code it would have failed anyway, since the opcode enum is closed —
but on whichever foreign opcode it happened to meet first, reported as an
unknown variant among seventy-five alternatives. That check is unchanged for
files that really are Ghidra's; an unknown opcode is still an error.

Program::new and the typed Extractor::extract_each and
Comparator::compare_programs are unchanged, so code that knows its operation
type keeps working.

lift::find_ghidra_home is gone; use LifterType::find_home. Discovery was
a Ghidra function re-exported as if it were the general one. It is now a method
on the backend, so each names its own environment variable and its own default
locations, and the error says which variable to set rather than naming
GHIDRA_HOME for whatever tool was asked for (#46). LifterType also gains
name and home_spec; home_spec is None for angr, which is imported as a
Python library and has no installation directory to point at.

New public items for asking whether a pairing is valid before building one:
Shape, BirthmarkType::shape, BirthmarkType::with_shape,
BirthmarkType::pairs_with and Algorithm::shape.

Error reports its cause. impl std::error::Error for Error was empty, so
source() returned None even for the seven variants holding a wrapped error
(#62). Each now returns it, and an error chain has the depth it should. Nothing
called source() before, because there was nothing to get.

Error is derived with thiserror rather than hand-writing Display, so the
message sits on the variant it belongs to. Every variant renders exactly as it
did, with one deliberate exception: Error::Clap no longer prefixes clap's
message with Clap error: , since clap's already begins error: . No #[from]
impls were added, so nothing converts implicitly through ? that did not
before.

New public items for listing the vocabulary rather than hand-writing it:
BirthmarkType::advertised, BirthmarkType::description,
AnalysisType::advertised_names and MAX_ADVERTISED_K; Algorithm::cli_name
became public. advertised_names pairs each birthmark with the algorithms that
operate on its shape, so a generated list cannot offer a pairing that
AnalysisType::new refuses. These exist for building a completion list, and
none of them bounds what try_from accepts.

oinkie lift now lifts one file at a time. It used to lift every input
concurrently with no bound. Pass -j/--jobs N to get the old behaviour, with N
naming how many files may be lifted at once (#55).

Lifting runs a whole decompiler process per file, and every input went to
rayon's global pool, whose size is the machine's core count -- ten JVMs at once
on a ten-core laptop, with no way to ask for fewer. It is also unsafe on a fresh
installation: Ghidra compiles its SLEIGH language definitions on first use and
caches them inside its own installation, so parallel lifts race to write the
same file and the loser reads a half-written one. analyzeHeadless exits
successfully regardless, so this arrives as a missing output rather than an
error (#54). Once the cache is built it cannot happen again, which is why -j
prints a notice rather than being refused:

notice: lifting up to 4 files at a time.
Ghidra builds its language cache on first use, inside its own installation,
and parallel lifts can corrupt a cache that has not been built yet --
reported as a missing output, not as an error.
If Ghidra was installed recently, lift one file without -j first.

Scripts that lift more than one file and depend on the old speed need -j
added. extract and compare are unchanged and remain parallel: they are
computation over local files, with no shared external state and no separate
process per input.

oinkie lift --intermediate is the lifter's working directory. It was
described as somewhere to keep "Ghidra project directories". Every headless
lifter needs a directory to run in, because that is where its script writes its
JSON; Ghidra additionally keeps its project there. The option, the flag and the
behaviour are unchanged — only what it is documented to mean, which now holds
for a lifter that is not Ghidra's.

--home and --script keep one help string each. They mean the same thing for
every backend — an installation directory, and a script in whatever language
that backend runs — and the part that has to be per-lifter is the error, which
now is.

Highlights

• #30 — widened the birthmark hash prefix to 64 bits, removing a silent
  overwrite that was reachable at corpus scale

• #29 — fixed the left/right inversion in similarity CSVs

• #40 — fc-* birthmarks resolve their symbols again; they were empty, so
  unrelated programs scored a perfect match through all eight of them

• #36 — lift -i with a relative path no longer resolves it twice into
  irs/irs, and the directory is created rather than having to exist

• #38 — analysis names are parsed in one place, and a pairing that cannot
  mean what it says is rejected with the canonical form named

• #44 — which operation is a call is now the lifter's answer rather than a
  literal "CALL" in the extractor, and a program in which nothing is a call is
  refused instead of yielding an empty fc-* birthmark

• #45 — the lifted file decides which operation type reads it, so the
  pipeline no longer assumes P-Code; a representation with no reader is refused
  by name

• #54 — the tests that run a real Ghidra no longer run at the same time as
  each other, so they cannot corrupt its language cache; two of them that
  differed only in their -i path are now one

• #71 — an analysis name built from what oinkie info prints parses;
  the split between the two halves is searched for rather than assumed to be
  the last hyphen, so weighted-jaccard works inside a name and a two-word
  algorithm added later needs no special care

• LLVM IR/BC is no longer named as a target. The tagline and the site
  described oinkie as a toolkit for "Ghidra Pcode, LLVM IR/BC, and Binary
  Ninja", but LLVM is in neither Ir nor LifterType and is not in the
  multi-lifter design work, which is about Binary Ninja and IDA Pro. The
  documentation says what is there and what is planned

• The publication list is current, and the same in both places. Three
  papers are added to the README's references, and the two that
  docs/content/academic.md was missing are added there with a description of
  what each contributes

• The release script no longer rewrites text that merely contains the old
  version. It replaced the bare number everywhere in README.md and the
  landing page, so cutting 0.3.0 to 0.4.0 would have turned a cited paper's
  NII CRID 1572824500.3.007232 into ...500.4.007232 — silently, in the
  released README (#75). Each replacement is anchored to the badge or link it
  belongs to

• The CLI reference matches the CLI. cli/README.md documented
  -B/--binary-type, which this release removes, and was missing lift -j
  and run -A; lift's lifter list still read ghidra, llvm, binary-ninja
  where it is ghidra, angr, ida-pro, binary-ninja; and the oinkie info
  sample in the site docs was from v0.2.0 and showed a section the command
  does not print. All of it is regenerated from the real output

• #68 — birthmark files are written in a canonical order, so the bytes are
  a function of what the birthmark holds rather than of hash order; sequences
  keep the program's order, which is theirs to keep

• #66 — a birthmark stating one operation's frequency twice is refused
  rather than silently read as the last of them, in op-freq and fc-freq as
  well as the k-gram shapes

• #59 — op-*gram-freq birthmarks can be written. Every non-empty one
  failed with "key must be a string", so eight of the thirty birthmark types
  the CLI advertises could not produce a file and twenty-four of its eighty
  analyses were unreachable; the frequency map is a list of [kgram, count]
  pairs now, and no existing file changes

• #51 — birthmark and program files are read whole and parsed from the
  bytes, rather than through serde_json::from_reader on an unbuffered
  File, which cost one syscall per byte. A 9 MB birthmark loads in 15 ms
  where it took 2.7 s, and a compare over three such pairs went from 22.0 s
  to 7.8 s

• --skip no longer reads a half-written similarity file as a score of
  zero. Found by covering the parser it lives in: the score defaulted when
  the result, line was absent, so an interrupted run's leftovers came back
  as a successful comparison finding nothing in common

• #62 — Error is derived rather than hand-written, and reports its
  cause: source() was None for every variant, including the seven wrapping
  a foreign error. A usage error is also no longer printed as
  Error: error: ...

• #28 — line coverage is 95.3%, up from 91.3%. Error's Display is
  covered for every variant, and the CLI's own parsers — the --skip reader
  for a stored comparison, and everything reaggregate loads — are tested
  against fixture files rather than only through end-to-end runs. Six of the
  new tests were checked by breaking the code they cover

• #25 — the CLI stopped keeping its own copy of the analysis vocabulary.
  Two hand-written ValueEnums — 64 variants with 64 mapping arms, and 31 more
  — are gone, and --analysis and --birthmark-type are parsed by the library
  that already parsed those names. The k ceiling went with them, the two
  options now agree with each other and with oinkie info, and the completion
  list is generated from the pairing rules, so it cannot offer a pairing that
  would be refused

• #24 — the tests no longer set GHIDRA_HOME and put it back afterwards.
  Two of them ran at once in the same binary and saw each other's writes, which
  is what made CI fail on a branch whose own checks were green; in edition 2024
  set_var is unsafe precisely because a concurrent read of the environment
  is undefined behaviour rather than merely a wrong answer. The search now takes
  its environment, and its test for whether a path exists, as parameters — so a
  test describes a machine instead of having to become one, and the case where
  nothing is installed can be asserted for the first time

• #55 — lift is serial by default, with -j/--jobs N to opt back into
  parallelism; the old default could corrupt Ghidra's language cache on a fresh
  installation, and the corruption was reported as a missing output

• #46 — home discovery is per-backend, and the path handling around a
  headless lifter — resolving before the working directory changes, running in
  its own directory, moving the result across file systems — is written once
  instead of once per tool. A lift that produced nothing now reports what the
  tool said, since Ghidra exits successfully even when its script throws

• The crate is now formatted with cargo fmt, and CI enforces it with
  cargo fmt --all -- --check

What's Changed

• fix: write the left CSV record from the first birthmark, not the second by @tamada in #29
• fix: widen the birthmark file name hash to 64 bits by @tamada in #30
• ci: take release notes from a file kept in the repository by @tamada in #32
• fix: remove the -B/--binary-type flag that was never read by @tamada in #34
• refactor: build completions from a separate gencomp binary by @tamada in #35
• fix: resolve the -i path before handing it to Ghidra by @tamada in #37
• refactor: delegate the birthmark half, and reject pairings that cannot mean what they say by @tamada in #39
• fix: resolve the symbol a call refers to, so fc-* is not empty by @tamada in #47
• feat: record which intermediate representation a lifted file holds by @tamada in #48
• feat: refuse to compare birthmarks from different representations by @tamada in #49
• feat: let the lifter say which operation is a call by @tamada in #50
• feat: let the lifted file choose the type that reads it by @tamada in #52
• refactor: write the headless lifter shape once, not once per tool by @tamada in #53
• feat: lift one file at a time unless asked for more by @tamada in #56
• test: stop the Ghidra tests from running at the same time by @tamada in #57
• test: hand the home search its environment instead of writing to ours by @tamada in #58
• refactor: let the library parse the names it already parses by @tamada in #60
• test: cover the error messages and the CLI's own parsers by @tamada in #61
• refactor: derive Error, and let it report its cause by @tamada in #63
• perf: read a JSON file whole instead of one syscall per byte by @tamada in #64
• fix: write a k-gram frequency as pairs, not as object keys by @tamada in #65
• fix: land #66 and #68, which merged into a dead end by @tamada in #70
• fix: find where an analysis name splits, rather than assume it by @tamada in #72
• docs: regenerate the CLI reference from the CLI by @tamada in #73
• fix: anchor the version replace to what it is replacing by @tamada in #76
• chore: update version to 0.4.0 by @tamada in #74

Full Changelog: v0.3.0...v0.4.0