Evaluation Corpus

twinflame's accuracy work is graded against real corpora, never eyeballed. This page documents how each corpus is produced; the measured results live on Accuracy Benchmarks. No APKs or DEX are committed (binaries stay out of git); only the reproducible recipe lives here. Build outputs go to a working area outside the git repo — the convention is a corpus/ directory alongside the repo checkout (../corpus/ relative to the repo root), which is not itself a git repo. The build scripts live in eval/corpus/.

What each corpus grades

Corpus	Kind	Oracle	Grades (use case)
BlackyHawky Clock	built here (OSS matrix)	mapping.txt	version change (UC1) + obfuscation axis
Fossify Contacts	built here (OSS matrix)	mapping.txt	version change (UC1) + obfuscation axis
CalculatorM3 (Compose)	built here (OSS matrix)	mapping.txt	version change (UC1) + obfuscation axis
private benchmark app (2 releases)	supplied binaries	none (unlabeled)	version change (UC1) — perf/accuracy at scale
large public app, 2 versions (Telegram)	supplied binaries	none (unlabeled)	scale + change classifier
two dex-dumps from memory	extracted DEX dirs	none (unlabeled)	cross-sample kinship (component match)

The buildable OSS apps are the calibration set: because we control the build we get a mapping.txt oracle for free, and we can rebuild the same source to measure the change-detection noise floor (see the change-detection roadmap, E-2). The two supplied-binary apps (no source, no oracle) are for performance + accuracy at scale, not calibration; the dex-dumps are the cross-sample kinship test (no bijection).

OSS calibration matrix — persistent build tree (built 2026-07-08)

The three OSS apps are now built once into a persistent, indexable tree so benchmarks never rebuild. Layout (lives in the non-git working area ../corpus/, i.e. outside this repo):

corpus/oss_matrix/<project>/<version>/<profile>/{app.apk,mapping.txt}

• projects / versions (adjacent releases = the realistic UC1 gap): clock 2.30→2.31 (com.best.deskclock), contacts 1.5.0→1.6.0 (org.fossify.contacts), calc v1.4.3→v1.5.2 (com.vagujhelyigergely.calculatorm3, Compose).
• profiles (R8 strength axis, applied by appending profiles/<p>.pro to app/proguard-rules.pro): lax (-dontoptimize, rename+shrink only), optimize (the app's shipped -optimize config), strict (-allowaccessmodification -repackageclasses '', full flattening).

3 projects × 2 versions × 3 profiles = 18 {app.apk, mapping.txt} pairs. This lets any run pick load pairs without rebuilding: cross-version at a fixed profile = UC1; cross-profile at a fixed version = the obfuscation/optimization axis; both together = worst case.

Supplied-binary apps (no source, no oracle) — corpus/supplied_apps/<app>/<version>/app.apk

Two-version pairs of real apps we don't build, used for performance + accuracy at scale, not calibration (no mapping.txt, so no P/R/F1). Same app.apk filename convention as the matrix:

app	versions	notes
cartera	1.8.2, 1.9.1	private benchmark app (cat.atm.cartera), R8-renamed, ~2.5-mo cross-toolchain gap
telegram	12.7.2, 12.8.3	large public app (~145 MB, ~40k classes/side); app code not name-obfuscated
ms_authenticator	6.2603.1485, 6.2606.4246	com.azure.authenticator, single universal APKs (10 dex each)
twitter	12.0.0, 12.5.0	very large (~210k classes). Downloaded as .apkm (APKMirror split bundle); app.apk is the extracted base.apk (the only DEX-carrying part — config splits arm64_v8a/en/xxhdpi have no code). The original .apkm is kept alongside for provenance.

Cross-sample kinship — corpus/dex_dump_sample/

Extracted DEX directories (dumped from memory), no APK/manifest. Not a version pair — the cross-sample kinship test (UC2: find shared implementation with no bijection). Loaded via load_dex on the directory. See refs.txt for the target components to match.

Reproduce: eval/corpus/build_matrix.sh (checks out each tag, delegates to build_profiles.sh for the three profiles). Toolchain used: JDK 21, Android SDK platforms 34–36, each app's own Gradle wrapper (9.6.1 / 8.13 / 8.9). CalculatorM3's release build needs a signing config; a throwaway keystore + keystore.properties (both gitignored) are staged in its checkout — R8/mapping.txt are unaffected, we only read DEX.

Parse-verified class counts (new loader, 2026-07-08) — confirm the profiles are genuinely distinct (optimization inlines/merges a large fraction away; layout obf barely moves class count):

project	version	lax	optimize	strict
clock	2.30	6047	2599	2599
clock	2.31	6079	2615	2615
contacts	1.5.0	11035	11035	11021
contacts	1.6.0	11061	11061	11044
calc	v1.4.3	4472	2199	2093
calc	v1.5.2	10244	5327	5136

Perf benchmarking of the current (post-Analysis-drop) loader across this matrix is done — published in BENCHMARKS.md (prepare + diff timings for every matrix cell and the supplied-binary pairs). The P/R/F1 re-run on the current loader is still pending — driver staged at eval/bench_matrix.py (UC1 + obfuscation-axis rows, mapping-join oracle). The historical accuracy results on Accuracy Benchmarks were measured 2026-07-05 on the previous loader and a subset of these builds; they stand as the baseline the new run is compared against (the loader rework was verified output-equivalent, so no material shift is expected).

Build recipe (rename-recovery corpus: one version, R8-off vs R8-on)

The existing scorer (python -m eval.cli donor.apk target.apk mapping.txt) wants a donor (R8-off, clear names) and a target (R8-on, obfuscated) of the same source, plus the target's real mapping.txt. A debug/release pair produces exactly this.

Prereqs: JDK 17+ (JAVA_HOME), Android SDK (ANDROID_HOME) with a platform + build-tools the app targets. Each project ships a Gradle wrapper (./gradlew) — no system Gradle needed.

Fossify Contacts (Kotlin, AGP/Gradle 9.x, flavors core/foss/gplay)

git clone https://github.com/FossifyOrg/Contacts.git && cd Contacts
git checkout <tag>                       # e.g. a release tag; omit for main
echo "sdk.dir=$ANDROID_HOME" > local.properties
./gradlew --no-daemon :app:assembleFossRelease :app:assembleFossDebug
# donor  (clear):  app/build/outputs/apk/foss/debug/app-foss-debug.apk
# target (obf):    app/build/outputs/apk/foss/release/app-foss-release-unsigned.apk
# oracle:          app/build/outputs/mapping/fossRelease/mapping.txt

Release falls back to unsigned with no keystore (fine — we only read DEX). Uses proguard-android-optimize.txt (full R8: shrink + optimize/inline + rename).

BlackyHawky Clock — https://github.com/BlackyHawky/Clock

OpenCamera — https://sourceforge.net/p/opencamera/code/ (git mirror / SF checkout)

Same shape: ./gradlew assembleRelease assembleDebug, adjust flavor task names per project (inspect app/build.gradle* buildTypes/productFlavors first; confirm release has minifyEnabled=true — if not, add a minifyEnabled=true release-copy variant).

Obfuscation profiles (strength axis)

One app version can be built under multiple R8 profiles (lax / optimize / strict) to grade the matcher across an obfuscation-strength axis — see Obfuscation Profiles for the profile definitions, the engine axis, and how they are graded.

Version-pair builds (build_xversion.sh)

eval/corpus/build_xversion.sh <repo> <tagA> <tagB> assembleFossRelease <out> [gradle_ver] builds two release tags (both R8-on) and stages app.apk + mapping.txt; eval.xversion grades matching by joining the two mapping.txt files on original class name (no connecting git history needed — shallow clones work). Results: the degradation curve.