Reverse Engineering and Contributing

Reverse Engineering libshell & Contributing

This tool is a from-scratch reverse-engineering of Meta's libshell.so (the Quest system-shell renderer + environment loader). This page explains how it was reversed, how to keep reversing it, and how to contribute — so anyone can pick up where the RE left off.

The big picture

• libshell.so (in com.oculus.vrshell) renders the home, loads environment packages, runs locomotion/physics, and decides which env to show. Everything the editor/cooker does is a faithful replica of a piece of it.
• The desktop tool is one C++/Vulkan binary: a libshell renderer replica + a scene editor + the V79/OPA → V205/HSL cooker that repackages old homes into installable APKs.
• Nothing is linked against Meta code — the formats (RENDMESH/MATL/RENDTXTR/RENDSHAD/HSTF/ ASMH, the PhysX SEBD collision, the HZANIM/ACL skeletal clips) were all reversed byte-by-byte.

Tooling for reversing libshell

You need a disassembler with a decompiler. Any of the three below works on the v206 libshell.so (a ~44 MB aarch64 ELF). Pick IDA or Ghidra (Ghidra is free and fully capable); the capstone script is a no-GUI extra.

Get the binary first

libshell.so ships inside com.oculus.vrshell. Pull it off a headset (Developer Mode + adb):

adb shell pm path com.oculus.vrshell                 # find the APK path
adb pull <that_path>/base.apk  vrshell.apk
unzip -o vrshell.apk 'lib/arm64*/libshell.so'        # -> lib/arm64-v8a/libshell.so

(or adb pull it straight from the app's extracted native-lib dir). It's a stripped release build — no symbols — which is why the string → xref → function loop below is the whole game.

A. IDA Pro (full walkthrough)

1. Load it. File → Open → libshell.so. IDA detects ELF / ARM64 (AArch64); accept the defaults and let auto-analysis finish (bottom-left "AU" idle; it takes a few minutes on 44 MB). The .text virtual base is 0x9c98a0 — worth pinning so your notes/capstone addresses line up.
2. The windows you live in (all under View → Open subviews, but learn the shortcuts):
   • Strings — Shift+F12. Your #1 entry point. Filter for RENDMESH, ASMH, MeshDefinition, falling back to device default, etc.
   • Functions — the docked left panel (Ctrl+F inside it to filter by name/address).
   • Names — Shift+F4 — every named location; filter here too.
   • IDA View / Disassembly and the Pseudocode (decompiler) tabs.
3. Navigating — the six keys that matter:
   • G → jump to an address (paste 0x…).
   • X (or Ctrl+X) on any label/name/data → list cross-references TO it (who reads this string / calls this function). Double-click an xref to land inside the caller.
   • Enter → follow the thing under the cursor; Esc → jump back (like a browser).
   • F5 → decompile the current function to C-like pseudocode (needs the Hex-Rays AArch64 decompiler; without it, read the disassembly — same logic, more verbose).
   • Space → toggle graph ↔ linear disassembly.
4. Annotate as you go — this is the actual work:
   • N → rename the function/variable/label under the cursor (sub_DD2204 → logWrite, a2 → assetHeader).
   • Y → set the type of a variable (e.g. make a1 a MeshDefinition*).
   • ; → a comment on that line (repeatable comment: : ). Write the evidence ("FourCc == 'ASMH'", "nonzero Result → env rejected").
   • Structs — Shift+F1 (Local Types) → declare a C struct for an asset header, then Y it onto the pointer. Now every [x0, #0x34] shows as ->width instead of a raw offset. This is how you turn a format into a readable header.
5. Run the loop (see "Getting started" below): pick a string → X to its user → F5 → read → rename → follow the offsets it reads. Repeat outward.

B. Ghidra (free — same power, different keys)

1. New project → File → Import File → libshell.so → Analyze (accept defaults). It runs auto-analysis on import.
2. Windows/keys:
   • Defined Strings: Window → Defined Strings (or Search → For Strings…).
   • References: right-click a string/function → References → Show References to.
   • Decompiler: the pane is open by default next to the listing — no license needed.
   • Go to address: G. Rename: L. Retype/edit function signature: Ctrl+L / F on a var. Comment: ;. Structs: the Data Type Manager (create a struct, then apply it to a variable). Back/forward: Alt+← / Alt+→.
3. Same loop, same output.

C. Offline capstone script (no GUI, no license)

A ~120-line Python analyzer when you just want scriptable string-xrefs (this project used it heavily):

• Parse the ELF section headers → .text (vaddr base 0x9c98a0), .rodata, .data.rel.ro.
• Index every C string in .rodata/.data.rel.ro → vaddr → string.
• Build an ADRP+ADD / ADRP+LDR literal-address index: scan .text words, track the last ADRP page per register, and when an ADD/LDR consumes it, record computed_addr → code_addr. That gives you string xrefs with no disassembler.
• Build a BL call-graph (0x94000000 mask, sign-extend imm26<<2) → target → callers.
• Cache the maps (pickle); then str <regex>, xref <addr>, callers <addr>, dis <addr>.

The maintainer also runs an IDA-automation bridge to script these queries in bulk — but you do not need it; everything above is the plain IDA/Ghidra GUI on a file you pulled yourself.

Rule of thumb: strings → xrefs → the function. Never scan the whole 32 MB .text blind.

Getting started — your first hour in IDA / Ghidra

You do not need to understand libshell to start. Pick one concrete question ("why does my env's floor texture look wrong?", "what byte says a mesh is skinned?") and chase it. Here's the loop that cracked every format in this tool.

0. Get the binary + set the base

• Pull libshell.so off a headset (or your APK): adb pull from com.oculus.vrshell's native libs (/data/app/.../lib/arm64/libshell.so), or unzip the vrshell APK. It's a 44 MB aarch64 ELF.
• Load it in IDA Pro (aarch64) or Ghidra (free — File → Import, then Analysis → Auto Analyze). Let auto-analysis finish (it takes a few minutes; the file is huge).
• The .text vaddr base is 0x9c98a0 — note it so capstone addresses and the disassembler agree.

1. Start from a STRING, never from an address

The single best entry point is a log string or a FourCC. Both tools have a strings view:

• IDA: Shift+F12 (Strings window) → search. Or the MCP search_text / find_regex.
• Ghidra: Search → For Strings…, or Window → Defined Strings.

Search for something you care about, e.g. RENDMESH, ASMH, MeshDefinition::fix, falling back to device default, Asset ptr error. Double-click the string → look at its cross-references (IDA: Ctrl+X on the string / its data address; Ghidra: right-click → References → Show References to). That xref lands you inside the function that uses it — which is the function you actually want.

2. Read the function (decompile), rename as you go

• IDA: F5 for pseudocode. Ghidra: the Decompile pane is open by default.
• Rename everything you identify immediately (IDA N, Ghidra L): the function, its args, the struct fields. sub_DD2204 → logWrite, a2 → assetHeader, +52 → width. Future-you (and the next contributor) reads names, not offsets. This is the highest-leverage habit in the whole process.
• Add comments (; in IDA, ; in Ghidra) with the evidence — "FourCc check == 'ASMH'", "returns Result; nonzero → env rejected".

3. Follow the data, not just the code

To crack a format (the useful RE here), find where a header pointer is read at fixed offsets:

• Look for LDR w?, [x?, #0x34]-style loads off the asset pointer → those offsets are the struct layout. Cross-check two or three call sites; if they all read +52 as a 16-bit value used as a width, that's your field.
• FourCC/magic checks (CMP against an immediate like 0x484D5341 = "ASMH" little-endian) tell you what a loader expects. A version compare right after the magic is the version gate.
• When a check fails and the code builds an error Result and returns up the stack → that path is fatal (the asset/env is rejected). A check that only logs and continues is non-fatal. Knowing which is which is most of the battle (see the verified gate map below).

4. Confirm on a real asset (ground truth)

RE is a hypothesis; a real file confirms it. Dump the same bytes from a real Meta env (unzip the scene.zip, hexdump the RENDTXTR/RENDMESH header) and check your offsets predict the real values. The cooker's [COOK-VERIFY] oracle exists exactly for this — it checks every mesh against Meta's own fbVerifyAgainst logic at cook time.

What to track / write down

Keep a running note (a .md per subsystem works well) with, for each thing you crack:

• the function address and the name you gave it,
• the string/FourCC that led you there,
• the struct offsets and their meaning (+44 mips (u8), +47 fmt, +50 h, +52 w),
• whether a gate is fatal or non-fatal, and the exact reject message,
• one line of evidence (the compare/branch you saw), so it's reproducible.

That's literally how the map below was built. When you crack something new, add it here and open a PR — documentation of a format is as valuable as code.

Frida (optional, dynamic)

Static RE tells you the layout; Frida confirms behaviour live by hooking __android_log_write, open, or a specific sub_*. ⚠️ On this Quest, do not run a persistent frida-server — it conflicts with Zygisk and SIGSEGVs vrshell. Use short-lived spawn hooks and avoid hot-path functions.

How the environment load actually works (verified map)

The env-load dispatch lives around sub @0x1406a5c. It branches on the footprint path: cbz on the footprint string @0x1406c08 → empty footprint ⇒ load as Environment (0x1406d08); otherwise as Footprint. There's also an as Vista branch. The fatal gate is postInitAsset failure (@0xee86f0) — it builds an error Result and propagates it up → the env is rejected → fallback to the device default (nuxd).

Everything logged goes through sub_DD2204 → __android_log_write (tag defaults to "Clay", or a per-subsystem tag like "Shell" / "[SEO]"). The logger only emits when logger.minLevel <= msgLevel, and debug.logLevel (a shell-settable system property) sets that threshold.

Gate	Addr	Fatal?
assets.manifest FourCc "ASMH" + version 2	0x1e115f4 / 0x1e1206c	FATAL (root of asset resolution)
ClayPackage::Header magic "FBCLAYPACKAGE01"	0x20c10a8	fatal for that loader — not our format (we ship loose RENDMESH+ASMH)
checkShellEnvironmentRequirements (SublevelComponent)	0x1488224	NON-fatal — caller @0xadc550 does mov w20,#1 and ignores the result
Asset version newer than defined	0x2743880	NON-fatal — pure logger
MeshDefinition::fix (lods/parts/vertexStreams/elements/bounds/screenSize/8-align)	—	FATAL → asset load fails
MaterialDefinition::fix (constantValues/textureValues)	—	FATAL
Material constant-buffer size mismatch	0x17a5004	NON-fatal — logs + ret
Environment failed to load … falling back to device default	0x2091c6	the "rejected → nuxd" marker
Asset ptr error … reason: %s	0xae208	the human-readable reject reason

The cooker satisfies every fatal gate by construction: ASMH v2 manifest, meshes verified against a Meta oracle (fbVerifyAgainst, incl. the V205.2 8-byte hash alignment @0xeba93c and non-zero LOD screenSize), materials patched from real Meta MATL templates (so constantValues/textureValues are inherited-valid), and every referenced asset shipped in the same scene.zip (self-contained keyForPath refs → nothing missing → postInitAsset never fails).

Diagnosing on-device — with NO root

adb runs as the shell user, which holds READ_LOGS, and debug.logLevel is a debug property shell may set. So on a completely unrooted Quest you can:

adb shell setprop debug.logLevel Verbose   # full verbosity, no su
adb logcat -c                              # clear
adb shell am force-stop com.oculus.vrshell # reload the home
adb logcat -d -v brief                     # read WHY it loaded / fell back

The editor's Logcat tab does exactly this live (auto-verbose, auto-start, Reload home, Export a shareable report) — see Diagnosing a Rejected Home.

Contributing

• Build: cmake -S <project> -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DHSR_HAVE_PHYSX=ON then cmake --build build. PhysX is optional (OFF builds on Linux/macOS; the cooker falls back to the device-compatible ColliderBox grid). build_linux.sh is the one-command Linux build.
• Layout (relative to the project dir):
  • src/render/ — the Vulkan libshell replica (SPIR-V, materials, IBL, skinning).
  • src/loaders/ — V79 .gltf.ovrscene, .opa official homes, SceneLoader (HSR/haven).
  • src/cook/hsl_cooker.h — the whole V203/HSL cook (RENDMESH/MATL/…/ASMH + PhysX SEBD + spliceAPK).
  • src/io/ — the Blender-round-trip glTF import/export.
  • src/ui/editor.h — the editor + the device install / Logcat / diagnostics.
• The cook-time verifier is your friend: every mesh is checked against the Meta oracle at cook time and prints [COOK-VERIFY]. If it prints no PROBLEM, the asset will pass libshell on device.
• Adding a feature: cook features are editor UI toggles that set an env var read by runCook (not global flags). Render fixes are iterated in the desktop preview first (a fast replica), then cooked to the device and confirmed there — the Quest's libshell is the real renderer and the final authority; the preview is a close proxy that can't always reproduce it.
• When you get stuck on device, don't guess the format — read libshell's exact logic with the capstone/IDA workflow above, then match it. That's how every format in here was cracked.