Architecture and Design

Architecture and Design

This page mirrors the Safety model, Exit codes, Programmatic use, and Design overview sections of the README.

Safety model

RE-Unpacker takes adversarial input seriously -- archives dropped on RE rigs are often malicious.

• No shell=True. Every extractor invocation is an argv list. No construction of command strings from filenames.
• Per-extractor timeout. Default 1800s. On timeout, SIGTERM goes to the process group; SIGKILL follows 5s later if the leader hasn't exited. Raises ExtractorTimeout → recorded and next extractor tried.
• Bounded subprocess output capture. 1 MiB per stream; truncation is marked in the manifest with stdout_truncated / stderr_truncated.
• Path-traversal audit after every extraction. Every extracted symlink is resolved and compared to the output root. Escaping symlinks are replaced with a placeholder *.escaping_symlink.txt file recording the original target (so an analyst still sees what the archive tried to do). Escaping regular files are moved to <output_root>/_quarantine/. Counters are surfaced in stats.symlinks_neutralized and stats.quarantined_paths.
• Output-size ceiling (preventive on POSIX). Every extraction child runs under RLIMIT_FSIZE sized to --max-extracted-size, so a single-file decompression bomb (a tiny .gz that would expand to hundreds of GB, an xz/zip bomb) is stopped by the kernel mid-write rather than only after it lands. On Windows the stdlib has no clean RLIMIT_FSIZE analogue, so this cap is a no-op there and the post-extraction checks below are the safety net.
• Quota tracker (detective, run-wide). After each extraction step, the produced byte count and file count are measured and checked against the per-archive and run-wide ceilings (--max-extracted-size, --max-total-size, --max-files). Tripping one raises SafetyLimitExceeded → exit code 2, partial output and manifest preserved. This backstops the many-small-files case and the total-size case (and is the primary guard on Windows).
• UPX always operates on a copy. Source is never mutated.
• AppImage extraction never executes the binary with install privileges. When possible it bypasses execution entirely via unsquashfs -o <offset> against the SquashFS tail; when it falls back to --appimage-extract, the fallback runs in a tempdir with a freshly-copied file.

The deliberately-out-of-scope items are symmetrical-cryptography password recovery for password-protected archives, and recursion into file formats requiring OS-kernel-level mounting (loop-mount ISOs via mount -o loop). Both carry risks that outweigh their value for the RE-triage use case.

---

Exit codes

Code	Meaning
0	Run completed. Any per-file errors are recorded in the manifest -- the run still succeeded.
1	Input path invalid or unreadable; no run attempted
2	SafetyLimitExceeded tripped mid-run; partial output preserved
3	--tools-check mode: one or more known tools are missing
4	Unexpected fatal error (bug). Please file an issue.
5	Privilege required: --install, --uninstall, or --repair invoked without root
6	Package manager error: apt failed during install / remove / reinstall

---

Programmatic use

Most users will stick with the CLI, but the package is import-friendly:

from re_unpacker import main as cli_main

rc = cli_main(["./sample.deb", "-o", "./out", "--log-level", "WARNING"])
# Rc is the same integer the CLI would have exited with.

For direct use of the orchestrator (e.g. to embed RE-Unpacker inside a larger pipeline), see re_unpacker.cli._run_normal -- it's the canonical construction pattern for logger → tools → manifest → quota → orchestrator.

---

Design overview

Three-layer file-type detection

1. Magic bytes read from the file head (and, for formats like DMG, from the tail). Offset-aware; handles ISO-9660 at offset 32769.
2. file(1) via libmagic -- used for PE sub-type disambiguation (NSIS vs InnoSetup vs InstallShield vs WiX Burn) and for formats the magic table doesn't cover.
3. Extension -- used only as a tertiary tiebreaker (e.g. disambiguating .jar/.apk/.whl among plain ZIPs, or confirming an OLE2 compound doc is specifically a .msi).

Each detection carries a signals list in the manifest so you can see exactly why a file was classified the way it was:

"signals": ["magic:GZIP", "file_desc:gzip compressed data, from Unix, …",
            "mime:application/gzip", "refine:tar_composite:.tar.gz", "ext:.tar.gz"]

Extractor registry

Every extractor subclasses re_unpacker.extractors.base.Extractor and declares:

• handles_kinds: frozenset[FileKind] -- what it can open
• required_tools: tuple[str, ...] -- what binaries must be on PATH
• priority: int -- higher wins when multiple extractors handle the same kind
• is_secondary: bool -- True for resource / section dumpers that run alongside the primary

The registry builds two dispatch maps (primary, secondary) at startup. The orchestrator pulls the primary list for a detected kind, tries each in priority order until one succeeds (or all raise ExtractorNotApplicable), then runs every applicable secondary extractor regardless.

Dispatch chain semantics

The key distinction is ExtractorNotApplicable vs ExtractorFailure:

• ExtractorNotApplicable: the extractor looked at the file and decided it's not the right job (UPX sees no magic; binwalk returns rc=3 "no signatures"). Orchestrator catches silently and tries the next extractor. Not recorded as a manifest error.
• ExtractorFailure: the extractor tried and failed (non-zero exit, malformed output). Recorded as a manifest error; orchestrator tries the next extractor.

This is why a run on a bare ELF binary produces errors=0 even though UPX and binwalk were both attempted and declined.

Recursion engine

BFS work queue of (path, depth, source_archive, source_archive_sha256, rel_path). Dedup is by SHA-256 of the file contents -- a byte-identical archive appearing twice in the input is extracted once. Worker threads (-j N) share the queue, the dedup set, and the manifest; all access is lock-protected. Manifest writes are line-buffered JSONL so an interrupted run still has a valid partial record.

---