Mach-O: Do not back a segment that occupies no memory

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Unstripped Mach-O executables that keep their debug info in a __DWARF segment rather than a separate .dSYM fail to load with ValueError: Address 0x... is already backed!. __DWARF has vmsize 0, so it occupies no address space and the linker gives __LINKEDIT the same vmaddr, but _load_segment sized the backer from filesize alone and dropped hundreds of kilobytes of debug info exactly where __LINKEDIT belongs.

A segment maps vmsize bytes and the file contributes at most that many, which is this backend's invariant to keep: leave a segment that maps nothing unbacked, and clamp the rest to memsize.

Reproduced on repomapper from the Homebrew reposurgeon 5.9 arm64_sonoma bottle. The regression splices the same segment shape into an existing fixture, so it needs no new binary.

Validation: #729 (comment)

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head fca4973e48395dcce9aa20aaf233f6b46aa9cb7c against baseline b58ea02a446106647cdaae32bdf91b7062404cc1. Python 3.12.13.

Reproducer, a public Homebrew bottle:

TOKEN=$(curl -s "https://ghcr.io/token?service=ghcr.io&scope=repository:homebrew/core/reposurgeon:pull" \
  | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])")
curl -sL -H "Authorization: Bearer $TOKEN" \
  https://ghcr.io/v2/homebrew/core/reposurgeon/blobs/sha256:0e181830306d8e99c3501ebbe88236a93a261b7c8955c79e244c6bf104381f03 \
  | tar -xz reposurgeon/5.9/bin/repomapper
python -c "import cle; cle.Loader('reposurgeon/5.9/bin/repomapper', auto_load_libs=False)"

That file is sha256 f24c22b752cddf968619512f1784b1c66d30d1758ec5d75d79210f633ff6da93, an arm64 MH_EXECUTE with __DWARF at vmaddr 0x100254000, vmsize 0, filesize 0xe45cd immediately in front of __LINKEDIT at the same vmaddr. On the baseline the load raises ValueError: Address 0x254000 is already backed! from cle/backends/macho/macho.py:1054 into cle/memory.py:248. On this head it loads, and loader.memory.load(0x100254000, 16) returns 112280e50e512288e50e512290e50e51, which is __LINKEDIT's file content at offset 0x310000 rather than the 5a4c4942 (ZLIB) compressed-debug header at __DWARF's offset 0x228000.

• Regression: pytest tests/test_macho.py — 10 passed. With only the macho.py hunk reverted, test_zero_vmsize_segment fails with ValueError: Address 0x2000 is already backed! and test_filesize_larger_than_vmsize fails on a backer 4096 bytes longer than the segment maps.
• Full suite: pytest tests/ — 204 passed, 9 skipped on head; 202 passed, 9 skipped on the baseline. The skips are pre-existing unittest.skip("TODO") markers in tests/test_macho_bindinghelper.py.
• Lint/type: pylint and pyright against the merge base — cle/backends/macho/macho.py 10.00 unchanged, pyright badness 0.1723 -> 0.1713; tests/test_macho.py 9.90 -> 9.93, badness 0.080 -> 0.055.
• Hooks: pre-commit run --all-files — 22 hooks pass, 2 skipped for having no matching files, no file rewritten.
• Fixture sweep: all 22 Mach-O files under angr/binaries tests/ loaded on both revisions with cle.Loader(..., auto_load_libs=False, main_opts={"backend": "mach-o"}). Backer start addresses, lengths and sha256, segment tables, and min_addr/max_addr are identical on both; tests/aarch64/IPwnKit.macho.kext raises CLECompatibilityError: Unsupported Mach-O file type: 11 on both. None of them has a vmsize 0 segment, so the change fires on none of them.
• Corpus sweep: 47,664 Mach-O objects from a corpus of public macOS, iOS and UNIX package builds, parsed at the header level. 32 have a segment with vmsize 0 carrying file content, all of them __DWARF sharing __LINKEDIT's vmaddr; that matches the number of Mach-O units that failed at this raise site in the sweep that found the bug. None has filesize > vmsize > 0 and none has a segment whose file range runs past EOF.

Caveats:

• The min(filesize, memsize) clamp has no observed failing input of its own. It is the general form of the invariant the vmsize 0 skip depends on — with the clamp in place, the zero case falls out as an empty read instead of being a separate special case — and the second regression exercises it on a synthesized segment.
• Two further binaries with the same segment shape were checked. pulumi-language-bun from the Homebrew pulumi 3.254.0 arm64_sonoma bottle (sha256 c7407b5e97ecf140abb7264fc07d28cfade129424b8a91c4ff522854e0250a32) now loads. repocutter from the reposurgeon 5.9 sonoma bottle (sha256 ca6b9453771c0e091fbaa61fe5ff55ed1b0ba39f0ade99886ef21883dff120c5) gets past this failure and then hits CLECompatibilityError in _load_lc_unixthread, an unrelated defect the ValueError was masking. That x86_64 binary also shows MachO.__init__ hardcoding linked_base = 2**32, which is wrong when __PAGEZERO is not 4 GB; neither is addressed here.
• Other backends reach the same add_backer raise site for unrelated reasons; only Mach-O segment sizing changes here.
• add_backer rejects a new backer only when its start address is already backed, so an oversized backer starting below an existing one is still accepted silently. That is a Clemory question and is left alone.
• Sections of a vmsize 0 segment are still registered, because sections_by_ordinal indices are what symbol entries refer to.
• The gate covered cle only; angr was not run against this head.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_729

[ltfish]

LGTM. Thanks!