Raise targetSdk 35 -> 36 (Android 16): Play now requires it. History, expected risks, and a safe migration playbook

[zo-sol]

Why this issue

Google Play now rejects our uploads with: "App must target Android 16 (API level 36) or higher." Our current standard is targetSdk 35 (compileSdk 35, minSdk 24, AGP 8.7.3). We have to raise the target again, and we want to do it the same way we did 28 -> 35: probe on hardware first, understand what the new SELinux domain changes, ship with a rollback story, and regression-test the exact bug classes that already bit us.

This issue is written to be self-contained: it retells the whole 28 -> 35 story so someone with zero prior context can execute the 36 bump safely.

Part 1. How we raised 28 -> 35 without losing the Linux engine

The Android app boots a proot-distro Ubuntu rootfs in app storage and runs node plus the official claude/codex CLIs inside it:

flowchart TD
    UI[WebView UI] -->|localhost HTTP 200| SM[ServerManager<br/>guest env + lifecycle]
    SM --> GE{{GuestExec seam}}
    GE --> DPE[DirectProotExec<br/>proot flags / binds / fake proc]
    DPE --> PROOT[proot from nativeLibDir<br/>Termux process_vm build]
    PROOT -->|maps guest ELF via its own loader mmap| NODE[node]
    NODE --> CLAUDE[claude CLI]
    NODE --> CODEX[codex CLI]

Loading

targetSdk 28 was originally pinned because Android 10+ applies W^X to the untrusted_app SELinux domain: an app targeting 29+ cannot execve() ELF from its own data dir. Our guest binaries live exactly there (/data/data/com.iqlabs.agentnet/files/rootfs/...), so raising the target looked like it would kill the engine.

What actually happened (PRs #109, #110):

• We built a ground-truth on-device probe instead of guessing. Verdict: we never direct-execve guest binaries. proot maps them with its own loader (mmap), and that path survives the targetSdk 35 W^X policy. Verified on the worst-case device (Solana Seeker: Android 16 plus arm64 pointer tagging). No linker-exec routing was needed. This is the same reason Play-Store Termux can run proot-distro at target 29+.
• We de-risked with a temporary legacy(28)/modern(35) flavor split plus the GuestExec seam, validated on hardware, then collapsed to a single targetSdk 35 config (Adopt targetSdk 35 as standard + Play release AAB pipeline #110).
• Along the way we fixed a pre-existing silent bug: tar hardlinks were extracted as host-absolute symlinks that dangle inside the guest, so claude died with spawn ENOENT (Add legacy/modern targetSdk flavors, extract guest-launch seam, fix claude hardlink extraction #109: TarExtractor guest-absolute fix, install marker bump v2 -> v3 for a one-time rootfs re-extract).

Constraints that made 35 work and must survive 36 untouched:

• Keep the Termux process_vm=yes proot build and its LD_LIBRARY_PATH libs. Stock proot reads guest memory with PTRACE_PEEKDATA, which the kernel rejects with EIO on arm64 top-byte-tagged addresses (Seeker), and the server silently never boots. Fixed in PR android: run sandbox on pointer-tagged devices (Solana Seeker) + mobile fixes #57; do not swap the proot build.
• Never enable MTE (android:memtagMode). Never set PROOT_NO_SECCOMP.
• Keep applicationId and signing identical so updates stay in-place and never wipe the user rootfs.

Part 2. What broke AFTER the bump (the silent deaths), and the lesson

Raising the target moves the app into a newer untrusted_app policy. The engine booted fine, and then tools started dying silently on side syscalls:

Bug	Symptom	Root cause	Fix
#112 -> #114	git clone always fails: "remote did not send all necessary objects"	first misdiagnosed as an Ubuntu 24.04 rseq regression; the rollback theory was disproven on device	interim dulwich clone shim
#115 -> #116	git loose objects silently lost (50 written, 0 survive), pnpm broken	the domain denies hardlinks; proot --link2symlink FAKED link() success (returns 0), defeating git's and pnpm's own rename/copy fallbacks	drop --link2symlink so link fails honestly and fallbacks fire; guest /etc/gitconfig core.createObject=rename as belt and suspenders
#117	node os.loadavg() / os.cpus() broken, inotify watchers dead	the domain denies read on /proc/loadavg, /proc/version, inotify sysctls, etc.	fake /proc files (.sysdata/) bind-mounted only where the real file is unreadable, tested by actually opening it (File.canRead() lies under SELinux)
#117	bun install produces 0 files, reports nothing	bun's default hardlink backend hits the same denial and does not fall back	PATH-ahead bun wrapper forcing --backend=copyfile
#122 cleanup	stale #114 git shim shadowed the fixed native git forever on old installs	shim was removed from fresh builds only	delete the shim on every launch (idempotent)

The one-line lesson: the exec wall was never the real danger. The real danger is a newer SELinux domain quietly changing side syscalls (link, /proc reads), plus layers that fake success and turn honest failures into silent data loss. A booting server and HTTP 200 prove almost nothing.

Two delivery channels exist for pushing fixes to already-installed devices:

1. Install marker bump: forces a one-time full rootfs re-extract. Heavy; only when files inside the bundled tar changed.
2. Every-launch idempotent writes in Installer.kt (gitconfig, .sysdata, wrappers, stale-file deletion). Cheap and instant; prefer this.

Part 3. The 36 plan

flowchart LR
    A[Bump toolchain<br/>AGP 8.9.1+ and compileSdk 36] --> B[targetSdk 36 build]
    B --> C{On-device probe:<br/>server boots?<br/>claude and codex answer?}
    C -->|denial found| D[Classify with the taxonomy below<br/>fix via Installer or DirectProotExec]
    C -->|clean| E[Silent-death regression suite]
    D --> E
    E -->|any silent loss| D
    E -->|all pass| F[QA matrix like issue 111<br/>Seeker + Galaxy + Pixel]
    F --> G[Ship single targetSdk 36<br/>same appId and signing]

Loading

Step 1: toolchain

• AGP 8.7.3 -> 8.9.1+ (compileSdk 36 support); keep Gradle 9.3 if compatible
• compileSdk = 36, targetSdk = 36 in app/build.gradle.kts; update the "targetSdk 35 is our standard" rationale comments in the same commit (house rule: update, never delete)

Step 2: re-ask the exec question on hardware

• Fix scripts/probe-modern-exec.sh: it still hardcodes the removed legacy/modern flavor tasks; point it at the single variant
• Run on Seeker (Android 16, pointer tagging) and at least one Galaxy: server HTTP 200, claude answers, codex answers
• Capture adb logcat -d | grep -iE "avc: denied|AgentNet" and classify any denial with the taxonomy below

Step 3: silent-death regression suite (what 28 -> 35 taught us)

Run these inside the guest via the in-app agent chat (NOT adb run-as; see the playbook for why):

• git clone https://github.com/expressjs/express && git -C express fsck: clean, loose object count matches
• pnpm install of a pinned small package: installed file count matches a known-good run
• bun add express: node_modules populated, require('express') works
• node -e "console.log(os.loadavg(), os.cpus().length)": real values, not zeros
• /proc sweep: for f in loadavg stat uptime version sys/kernel/cap_last_cap sys/fs/inotify/max_user_watches; do cat /proc/$f >/dev/null 2>&1 && echo OK $f || echo DENIED $f; done. Any NEW denial: add a row to FAKE_PROC in Installer.kt; that list plus the conditional bind in DirectProotExec is the entire fix
• hardlink honesty check: touch a; ln a b must fail visibly (EACCES), never fake success

Step 4: behavior changes specific to targetSdk 36

• Large screens ignore android:screenOrientation="portrait" at target 36: check the WebView UI in landscape and resized windows (foldable or tablet emulator profile)
• Predictive back is on by default at 36: verify back navigation (no Kotlin onBackPressed override exists today, so expected low risk)
• Edge-to-edge opt-out is fully removed at 36: verify insets (we already render under the 35 enforcement, expected no-op)
• Re-verify the lock-screen background turns feature (Keep agent turns running while the phone is locked (opt-in), with vibrating lock-screen approval alerts #119) under the dataSync FGS rules (the 6h timeout already applies at 35)
• 16 KB page size: verify the jniLibs (libproot.so and friends) have 16 KB aligned LOAD segments (APK Analyzer or llvm-objdump -p). This is a parallel Play upload gate, not a targetSdk behavior, but the same submission blocks on it

Step 5: ship

• Same applicationId and signing; confirm an in-place update preserves the rootfs (no data wipe prompt)
• Sideload APK and Play AAB both build the single 36 variant; keep the public app-debug.apk asset name so the download link survives
• Open a QA issue like QA: verify Android build after targetSdk 35 update (#110) #111 with the checklist above for cross-vendor coverage (Pixel, Samsung, one Android 13/14 device)

Debugging playbook (when something fails)

Denial taxonomy, read from adb logcat -d | grep -iE "avc|AgentNet":

Signal	Meaning	Direction
execute_no_trans on our binary	execve wall	exec routing / linker shim territory (we have never needed it)
execmod or execute on mmap	proot loader mmap blocked	proot-level fix; this would be the first real break of the loader-mmap path
EFAULT / PEEKDATA	pointer-tagging regression	check the process_vm proot build and its LD_LIBRARY_PATH are intact
ENOENT on a guest binary	dangling symlink or path bug	like the #109 claude fix, not an exec wall
tool "succeeds" but files are missing	a layer is faking success	instrument the syscall (strace or LD_PRELOAD shim) until one return value contradicts the filesystem; this is the #116 pattern
no AgentNet server log at all	server never started	earlier crash; read logcat from process start

Method rules, learned the hard way in #115:

1. Reproduce in the real SELinux domain. adb run-as runs in runas_app and does NOT reproduce untrusted_app bugs: a 56-clone run-as harness passed 100% while the in-app path failed 100%. Run probes as a true child of the app (paste commands into the in-app agent chat); use run-as only to stage files and read results back.
2. Read the tool's source first. We burned a full round on an O_TMPFILE theory that one look at git's object-file.c refuted.
3. One variable per probe. Save each result to a file in the guest, pull it with run-as, then change one thing.
4. Dig to the syscall. Stop only when you can name the single syscall whose return value contradicts the filesystem (for fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116: link returned 0, target did not exist).
5. Distrust any layer that reports success on behalf of the kernel (--link2symlink, File.canRead()). Verify with a real operation.

The full #115 probe kit (probe4-probe10 shell scripts, LD_PRELOAD C shims, staging instructions including zig cross-compilation for the aarch64 glibc guest) was removed from the tree after resolution but is fully recoverable from git: git show 2e87263:debug/issue-115/README.md.

Rollback story

targetSdk is a manifest attribute, not data. If 36 breaks a vendor we did not cover, shipping a 35 build again is an ordinary in-place update (same appId and signing); the rootfs and user data are untouched either way. What we can NOT easily roll back is an install marker bump (it re-extracts the rootfs), so prefer every-launch idempotent fixes for anything discovered after release.

[zo-sol]

Research + Step 1/2 prep done; device pass is the remaining gate

The big de-risking finding: raising targetSdk 35 -> 36 changes NOTHING in SELinux policy. There is no untrusted_app_36 (or _35) domain: AOSP seapp_contexts maps minTargetSdkVersion=34 and above to the same current untrusted_app domain (source). So the proot loader-mmap path, the hardlink denial (#116), and the /proc read denials (#117) are all byte-identical between a 35 and a 36 build on the same device. The 28 -> 35 crossing was the real one (it moved us across the 29+ W^X boundary); 35 -> 36 stays inside the same domain. The on-device run is now confirmation, not discovery.

What actually changes at targetSdk 36 (all verified against behavior-changes-16):

1. screenOrientation="portrait" is ignored on sw600dp+ screens. Handled: MainActivity now declares the temporary PROPERTY_COMPAT_ALLOW_RESTRICTED_RESIZABILITY opt-out (documented to die at targetSdk 37; the real fix is the responsive UI, JetBrains/IntelliJ plugin: reuse the localhost + webview surface via JCEF (and make the shared UI responsive for desktop) #128).
2. Edge-to-edge opt-out removed. No-op for us: we never used the opt-out, so the app has been edge-to-edge since the 35 bump.
3. Predictive back on by default. No-op: no back interception exists in the shell.
4. Local network permission (future) explicitly excludes loopback ranges, so the WebView -> 127.0.0.1 path is safe even when it ships.
5. No FGS/dataSync change (the 6h timeout came at 35), no WebView/cleartext change, no ART/JobScheduler change gated on 36.

Toolchain: AGP 8.7.3 -> 8.9.1 (the official minimum for compileSdk 36 per AGP releases); compileSdk/targetSdk = 36. Build verified locally: APK reports targetSdkVersion:'36' with the compat property present.

16 KB page gate: our arm64 jniLibs (libproot.so, libloader.so, libtalloc.so, libandroid-shmem.so) already have 16 KB-aligned ELF LOAD segments (Termux builds), so the Play check passes. Separate future risk, not a Play blocker: the rootfs glibc binaries (node/claude/codex) are invisible to Play but would need their own handling on actual 16 KB-page hardware.

Also added scripts/guest-regression.sh: the in-guest silent-death suite (hardlink honesty, git 50-loose-object survival, clone + fsck, pnpm/bun install reality checks, node os APIs, /proc sweep against FAKE_PROC). Every test asserts reported-success-matches-disk, per the #115 method; header documents the stage/run/collect flow (run IN-APP, never via run-as).

Remaining before ship: the on-device pass (Step 2/3 checklists above) on Seeker + one Galaxy, then push, then the #111-style cross-vendor QA issue. The change set is staged locally and lands once the device pass is green.

[zo-sol]

On-device validation: PASS (Seeker, Android 16, targetSdk 36)

Ran the Step 2/3 gates on the Solana Seeker (API 36) with the targetSdk-36 build installed as an in-place update over the previous versionCode-116 CI build (rootfs preserved, no data wipe).

Exec probe: server reached server ready (HTTP 200) seconds after launch; claude model list loaded (which spawns the claude CLI); codex app-server started. The full chain was visible in the process table under the app uid: libproot.so (with --copy-on-link and all eight #117 fake-/proc binds active) -> node index.js -> live claude turn + codex app-server. The only app-scoped avc denials were setattr on the sysfs root from proot's bind setup, which is harmless noise (not execve/execmod/execute_no_trans), exactly as the same-SELinux-domain research predicted.

Silent-death regression suite (scripts/guest-regression.sh, executed IN-APP in the real untrusted_app domain): 12 pass, 0 fail.

• hardlink honesty: link() succeeded AND the target is real, so the patched PRoot's copy-on-link is doing honest work at 36 (no fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116-class lie)
• git: 50/50 loose objects survive; expressjs/express network clone + git fsck clean
• node os APIs: cpus=8, loadavg real values (all FAKE_PROC binds working)
• /proc sweep: all eight covered paths readable; measured three uncovered denials (/proc/vmstat, /proc/sys/kernel/osrelease, /proc/sys/kernel/pid_max) with no consumer in our stack; same domain as 35 so these are not regressions, just newly recorded facts. If a tool ever needs one, the fix is a single new row in FAKE_PROC
• pnpm/bun: not present in the current rootfs, skipped (not a regression)

Claude and codex chats both answered normally in manual testing.

Shipped: ed1a5b4 (targetSdk 36 + AGP 8.9.1 + compat property + regression suite) and 4d108db (version 0.1.3) are on main. The sideload APK rebuilt via android-apk.yml run 124; the signed release AAB is building as android-release.yml run 9 (versionCode 9, versionName 0.1.3), which clears Play's API 36 gate. Remaining per this issue: cross-vendor QA (#111-style) on Pixel/Galaxy, which can proceed from the new android-latest APK.