ELF: Survive a GNU hash table that declares no buckets

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

An ELF whose GNU hash table declares no buckets aborts the whole load. On binaries/tests/x86_64/gnu_hash_resiliency_0, the fixture in the linked binaries pull request — test_killing_ref with the 96 bytes of its DT_HASH and DT_GNU_HASH tables zeroed, the way some stripping and obfuscation tools leave a binary:

  File "cle/backends/elf/elf.py", line 1079, in __register_dyn
    num_symbols = seg_readelf.num_symbols()  # this is not actually reliable
  File "elftools/elf/dynamic.py", line 321, in _num_symbols
    return gnu_hash_section.get_number_of_symbols()
  File "elftools/elf/hash.py", line 160, in get_number_of_symbols
    max_idx = max(self.params['buckets'])
ValueError: max() iterable argument is empty

Nothing else about such a file need be broken. Its symbol table, relocations and version tables can be entirely intact, and both hash tables only accelerate lookup by name, so cle refuses a file it could read.

Root cause

ELF.__register_dyn calls seg_readelf.num_symbols() unguarded. pyelftools sizes DT_SYMTAB from whichever hash table it finds, and GnuHashSection.get_number_of_symbols does max(self.params['buckets']) on an array that is empty here. The exception escapes __register_dyn, __register_segments and ELF.__init__, so a malformed lookup accelerator costs the entire load.

The same shape reaches cle's own table one step later: GNUHashTable.get does h % self.nbuckets, which is ZeroDivisionError when nbuckets == 0. ELFHashTable.get already returned no match for that case, so the two siblings disagreed.

Fix

ELF.__num_dynamic_symbols catches ELFError and ValueError from num_symbols() and falls back to the bound pyelftools itself uses for a file with no hash table at all: the nearest dynamic pointer above DT_SYMTAB, restricted to tags that really do hold addresses rather than sizes or string-table offsets. GNUHashTable.get returns no match when nbuckets == 0. Same file, this head:

gnu_hash_resiliency_0: hash table does not give a dynamic symbol count (max() iterable argument is empty)
loaded : <ELF Object gnu_hash_resiliency_0, maps [0x400000:0x404017]>
entry 0x401070   symbols 56   sections 30   relocations 11   PLT 4   deps ['libc.so.6']

The fallback bound is exact rather than merely plausible, and checkable from the file: DT_SYMTAB is 0x3d0 and DT_STRTAB is 0x4a8, so at DT_SYMENT 24 the table holds 9 entries with no remainder, and independently DT_VERSYM 0x5fe plus two bytes per symbol for 9 symbols reaches 0x610, which is exactly DT_VERNEED.

Testing

tests/test_gnu_hash_resiliency.py::test_zeroed_hash_tables loads the corrupted fixture and the uncorrupted test_killing_ref and requires the same entry, the same imports, the same relocation count and the same (name, relative_addr) symbol set. It fails on the baseline with the ValueError above and passes here; pytest tests reports 240 passed and 9 skipped against 239 and 9 on the baseline.

No linker emits an empty bucket array, checked positively: across 8,424 ELF files from a distribution's package closure the smallest nbuckets is 1, and a shared object exporting nothing links to nbuckets = 1 under GNU ld 2.46 and under LLD 21.1.8 alike. Every ELF in angr/binaries was loaded on both revisions and compared on backend, entry, dependencies and digests of the symbol, section, segment, relocation and PLT tables: 831 files, 0 differences.

Validation: #792 (comment)

sync: angr/binaries#200

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 43611edc172fe797df7b2c204c39d0b29816bd01 against baseline 46a37333f4f59b0facf8774ee743ebc4cc074e9b, with the fixture at 6c822960527ab674459496dc0af9b698f60c422b (baseline 8646be4eafa4f1fc285d787fb2b73426a5e11d19).

The fallback bound is exact, not merely plausible, and the fixture lets you check that without any file you do not have. Its DT_SYMTAB is 0x3d0 and its DT_STRTAB is 0x4a8, so at DT_SYMENT 24 the dynamic symbol table holds exactly 9 entries with no remainder. Independently, DT_VERSYM sits at 0x5fe and carries two bytes per symbol, so 9 symbols end at 0x610 — which is exactly DT_VERNEED. Two derivations that share no assumption agree on the count the fix recovers, and the loader it produces from the corrupted file is indistinguishable from the one built from the uncorrupted original.

• Regression: pytest tests/test_gnu_hash_resiliency.py — on baseline it fails with ValueError: max() iterable argument is empty raised at elftools/elf/hash.py:160 get_number_of_symbols, reached from cle/backends/elf/elf.py:1079 __register_dyn by way of DynamicSegment.num_symbols(); on head it passes
• Focused: the fixture and the uncorrupted test_killing_ref load to the same entry point, the same 56 symbols at the same relative addresses, the same 30 sections, the same 11 relocations, the same 4 PLT entries and the same single dependency
• Full suite: pytest tests — 239 passed, 9 skipped on baseline; 240 passed, 9 skipped on head, the extra one being the new test
• Lint/type: merge-base comparison of every changed file — pylint 10.00 unchanged on both changed modules and 10.00 on the new test; pyright badness unchanged or lower on all three
• Hooks: pre-commit run --all-files — every hook passes over the whole checkout
• Test inputs: no binaries or assembled containers outside angr/binaries
• Workspace gate: cle and angr/binaries, run in a per-feature instance carrying only these two branches — passed
• Fixture: rerunning tests_src/gnu_hash_resiliency/build_gnu_hash_resiliency.py on the committed input reproduces the committed fixture byte for byte

No linker can produce this input, and that was checked positively rather than only by absence. Across 8,424 ELF files from a distribution's package closure, 8,332 of which carry DT_GNU_HASH, the smallest nbuckets is 1 and none makes num_symbols() raise. Linking a shared object that exports nothing ({ local: *; };, --hash-style=gnu) with GNU ld 2.46 and again with LLD 21.1.8 yields nbuckets = 1 and bloom_size = 1 from both; binutils special-cases a table with nothing to hash. An empty bucket array is a corruption signature, never a build artifact, which is why the fixture is a real linker's output with the corruption applied rather than an assembled header.

The second hunk covers the same shape one step later. GNUHashTable.get takes h % self.nbuckets, so an empty bucket array with a non-empty bloom filter raises ZeroDivisionError there instead. ELFHashTable.get already returns no match when nbuckets == 0; the two siblings now agree.

Loading is otherwise unchanged. Every ELF in angr/binaries was loaded twice, once per revision, and the runs compared on backend, architecture, entry, linking, dependencies, object count, and digests of the full symbol, section, segment, relocation and PLT tables: 831 files paired, 0 regressions, 0 differences of any kind. A further non-redistributable set of 794 files was compared the same way: 0 regressions, 0 differences, and 1 file that previously lost its entire load and now loads and decompiles.

The workspace gate ran on these two branches in isolation and passed. It ran in a per-feature instance holding cle at 43611ed and angr/binaries at 6c82296, each one commit on top of master with no other branch's work in the tree: the workspace checks, the test-input check, every configured pre-commit hook, the per-feature-instance scenarios, and the cle suite, which reported 240 passed and 9 skipped. Suites for repositories that instance has not adopted — archinfo, pypcode, pyvex, pysoot, claripy, angr and angr-management — were skipped by design; hosted CI covers them. The paired runs against master reported above isolate this diff, and the gate covers the workspace around it.

One coverage gap, stated rather than left to be discovered. The paired load comparison does not reuse this workspace's decompilation harness: that harness decompiles every function of a binary, which is not affordable across 1,625 files, and its comparison tool keys on per-function results a load-only record does not have, so a load-only probe was written for it.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_792

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Full loader output for binaries/tests/x86_64/gnu_hash_resiliency_0, the fixture added by the linked binaries pull request, before and after this change. The reproducer is cle.Loader(path, auto_load_libs=False) and nothing else; the file is test_killing_ref with the 96 bytes of its DT_HASH and DT_GNU_HASH tables zeroed, so its .gnu.hash reads nbuckets=0, symoffset=0, bloom_size=0, buckets=[].

Before — the load aborts in pyelftools while sizing the dynamic symbol table from an empty bucket array, and the object is lost entirely:

cle master (d2ecea0)

binary : binaries/tests/x86_64/gnu_hash_resiliency_0
cle    : master d2ecea0

Traceback (most recent call last):
  File "./repro-cle792.py", line 9, in <module>
    ld = cle.Loader(path, auto_load_libs=False)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cle/cle/loader.py", line 187, in __init__
    self.initial_load_objects = self._internal_load(
                                ^^^^^^^^^^^^^^^^^^^^
  File "cle/cle/loader.py", line 805, in _internal_load
    obj = self._load_object_isolated(main_spec)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cle/cle/loader.py", line 1017, in _load_object_isolated
    result = backend_cls(binary, binary_stream, is_main_bin=self._main_object is None, loader=self, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cle/cle/backends/elf/elf.py", line 202, in __init__
    self.__register_segments()
  File "cle/cle/backends/elf/elf.py", line 1020, in __register_segments
    self.__register_dyn(seg)
  File "cle/cle/backends/elf/elf.py", line 1079, in __register_dyn
    num_symbols = seg_readelf.num_symbols()  # this is not actually reliable
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "site-packages/elftools/elf/dynamic.py", line 312, in num_symbols
    return self._num_symbols
           ^^^^^^^^^^^^^^^^^
  File "python3.12/functools.py", line 998, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "site-packages/elftools/elf/dynamic.py", line 321, in _num_symbols
    return gnu_hash_section.get_number_of_symbols()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "site-packages/elftools/elf/hash.py", line 160, in get_number_of_symbols
    max_idx = max(self.params['buckets'])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: max() iterable argument is empty

After — the fallback bound takes over, one warning is logged, and the object loads identically to the uncorrupted test_killing_ref (56 symbols, 30 sections, 11 relocations, 4 PLT entries, one dependency on each):

with this change (43611ed)

binary : binaries/tests/x86_64/gnu_hash_resiliency_0
cle    : this change 43611ed

WARNING | cle.backends.elf.elf | binaries/tests/x86_64/gnu_hash_resiliency_0: hash table does not give a dynamic symbol count (max() iterable argument is empty)
loaded           : <ELF Object gnu_hash_resiliency_0, maps [0x400000:0x404017]>
entry            : 0x401070
symbols          : 56
sections         : 30
relocations      : 11
PLT entries      : 4
dependencies     : ['libc.so.6']