COFF: place a section at an address its alignment allows - #804
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Replaces the record for the previous head The resolution puts the section with no file bytes and the section whose offset does not satisfy
Suites
CI prediction, written before the first job on this head finishedEvery check should pass: COFF A/B, 163 objectsThe figures below are the ones measured for head The header scan says why: of those 7,968 section headers, 92 have no bytes in the file and every Two frozen cle trees, both 163 objects from seeded draws over five collections of a private corpus of compiler output: MSVC
Reproducer provenance
Limits
|
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Complete section map, function symbols and CFG for Before — every cle#764 headAfter — the three with this changeThe recovered code is the source's, first eight instructions of each: disassembly |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_804 |
43bc7d0 to
ce0d435
Compare
The backend maps the object at its own file offsets and gives each section vaddr = PointerToRawData. A section holding no bytes in the file states that field as 0 -- that is what .bss is -- while SizeOfRawData still states its length, so it lands on the file header and, once it is longer than the header and section table, over the sections that follow. .text begins at file offset 0x104 in a six-section mingw object, so a 0x1300-byte .bss covers its first 0x11fc bytes. find_section_containing() then answers .bss for real code, and CFGFast._generate_cfgnode drops any block whose section is not executable, so those functions are never recovered; uninitialized data reads as the file header rather than zeros, and every .bss symbol is given an address inside the code. A section that states PointerToRawData 0 and marks itself IMAGE_SCN_CNT_UNINITIALIZED_DATA now gets zero-filled space of its own past the image, at the alignment its IMAGE_SCN_ALIGN_* states. Relocation patch offsets and symbol addresses read the same layout, so they follow it. The flag is the condition rather than the zero pointer alone, because the zero pointer alone is what a file controls: a 120-byte object can state PointerToRawData 0 with SizeOfRawData 0x4000000 on a section marked code, and zero-filling that is 64 MiB of allocation bought with one header field. Across 1,480 sections with no bytes in the file, in 64 distinct shapes, every one sets the flag, so requiring it costs nothing real. A section without it keeps the address its header states, which is what master does with it. MAX_IMAGE_SIZE bounds what the flag still admits. SizeOfRawData is 32 bits wide and a section that does set the flag can still state close to 4 GiB, so past 0x10000000 the section is placed and reports its stated size but no zero fill is allocated for it and a warning names it -- the outcome pe.py reaches through max_virtual_address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backend maps an object at its own file offsets and gives each section vaddr = PointerToRawData. That offset is only as aligned as the file's packing leaves it, and MSVC packs raw data with no padding between sections, so a section commonly begins where the IMAGE_SCN_ALIGN_* in its own header does not allow. All eight .text$mn sections in tests/x86/fauxware.obj state 16-byte alignment and six of them start at a file offset that is not a multiple of 16. x86 and AMD64 never notice, because their instructions have no alignment requirement. ARM64 and ARMNT, whose machine types #724 adds, do: a .text$mn placed on an odd address holds no instruction anything can decode, every function symbol in it lifts to a zero-length block, and CFGFast recovers nothing from it. Over 66 ARM64 COFF objects, 298 of 880 function symbols went unrecovered and 16 objects recovered none of their own; with the sections placed where their headers ask, all 880 are recovered. A section whose file offset does not satisfy its alignment now gets space of its own past the image with its bytes copied in, from the same cursor and under the same MAX_IMAGE_SIZE ceiling as the section marked IMAGE_SCN_CNT_UNINITIALIZED_DATA that already goes there. Where that ceiling would stop the move, the section keeps its file offset, which is where its bytes are. A header that states no alignment states no requirement and its section is left where it is; the 16 _section_alignment returns for that case is only where to put a section that has to be placed somewhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ce0d435 to
f75cec6
Compare
Coff._add_relocs took the patch address as section.PointerToRawData plus reloc.VirtualAddress and registered a relocation there without checking it. Neither bound was tested, and the two fail differently. A field past the end of the file crashes. The backend maps the object as one backer covering the file, so CoffRelocationDIR32.value asks Clemory for four bytes at an address nothing maps, and cle.Loader(..., perform_relocations=True) raises KeyError out of Clemory.load. A field merely past the end of its own section does not crash, and that is the worse half. Every offset in the file is mapped, so the store lands wherever the arithmetic points -- another section's raw data, the relocation table, the symbol table -- and the load returns normally with those bytes rewritten. Check both bounds where the relocation is registered rather than in relocate(). A relocation that cannot be applied should not reach self.relocs at all: it is handed to the symbol resolver, it can produce an extern symbol for a field that will never be written, and it is visible to every consumer that iterates an object's relocations. It is also where the PE backend drops a section whose raw data the file does not hold. The field's width comes from struct.calcsize on the relocation class's PACK_FORMAT -- four bytes normally, eight for ADDR64, two for SECTION -- so a four-byte field starting on the last byte of a section is out of bounds, which a bound on the start offset alone would miss. PACK_FORMAT is declared on CoffRelocation rather than on Relocation, so RELOC_CLASSES is annotated with the class it actually holds. This leaves the section mapping loop alone. Bounding a section's raw data by the size of the file is #806; the two compose, because _add_relocs walks self._coff.sections itself and would still register the relocations of a section that loop has skipped. Two details keep this bound correct against the other open COFF branches, and change nothing on this one. The section comes out of self._coff.sections by index rather than off the loop variable. Both name the same object here, by the definition of enumerate. #764 rewrites this loop to walk indices and drops the variable, and the two branches merge with no textual conflict, so with both applied and the loop variable read _add_relocs raises NameError on the first relocation of a supported type. Of the five COFF objects angr/binaries tracks that this backend loads, four carry such a relocation and stop loading; the fifth has none. #804 is stacked on #764 and carries the same rewrite. The file-size half of the bound is taken against self._image_vmem, the bytes the backend maps, rather than against self._data. Here the two are the same object: _image_vmem is assigned from _data in __init__, never rebound, and cle defines no subclass of Coff. #804 places a section whose file offset does not satisfy its alignment past the end of the file and extends the image to cover it, so a relocation into a moved section is past len(self._data) and inside the image, and bounding on the file would skip it. With both applied and the file used, x86/fauxware.obj keeps 177 of its 225 relocations and x86_64/fauxware.obj 66 of 126, and the test below asserting 225 fails.
cle's tests on master now load tests/aarch64/langdetect_go.macho and tests/aarch64/relocatable_object.macho, which #193 and #224 added after this branch was cut. angr/cle#764 and angr/cle#804 name this pull request in their sync: lines, so CI checks this branch out instead of master; once either is rebased onto current cle master those two files would be missing and the macOS job would fail. Merging master in supplies them and leaves this branch's own three objects and build script untouched.
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
An MSVC-built ARM64 COFF object's code is placed where no instruction can be decoded, so
CFGFastrecovers nothing from it.zlib-ng-static.dir\Release\insert_string_roll.obj(sha256
41eb600a40a3fd47b32c77ad3cb51cceabce5a86389924bfdb2f5749f7844eba), a member oflib/zlibstatic-ng.libin the officialzlib-ng 2.3.3 Windows ARM64 release,
has three
.text$mnsections and three function symbols. Loaded with the ARM64 machinetype that #724 adds:
Across 66 ARM64 COFF objects from four corpus collections, 298 of 880
function symbols lift to a zero-length block, 16 objects recover none of their
own, and CFGFast covers 73% of their executable bytes.
Root cause
The backend maps an object at its own file offsets and gives every section
vaddr = PointerToRawData. A file offset is only as aligned as the file's packing leavesit, and MSVC packs raw data with no padding between sections, so a section commonly begins
where the
IMAGE_SCN_ALIGN_*in its own header does not allow. The trackedbinaries/tests/x86/fauxware.objshows the same thing: all eight of its.text$mnsections state 16-byte alignment and six of them start at a file offset that is not a
multiple of 16.
x86 and AMD64 never notice, because
instruction_alignmentis 1 for both.AARCH64states 4 and
ARMEL2, so a.text$mnon an odd address holds nothing that decodes,project.factory.block()returns a zero-length block at every function symbol in it, andCFGFasthas no block to start from.Fix
A section whose file offset does not satisfy the alignment its own header states now gets
space of its own past the image, with its bytes copied in. A section whose header states no
alignment states no requirement and is left alone; 16 stays the default only for choosing
an address for a section that has to be placed somewhere.
Testing
tests/test_coff.py::TestCoff::test_sections_are_placed_at_the_alignment_they_stateasserts that each of the eight
.text$mnsections inbinaries/tests/x86/fauxware.objisplaced on a 16-byte boundary and holds the bytes at its file offset; the alignment
assertion fails on the merge base. Over 163 COFF objects the ARM64 function symbols
recovered go from 582 of 880 to 880 of 880, while every x86, AMD64 and ARMNT
object recovers exactly the same symbols on both sides. Asserting the ARM64 half here
needs the machine type from #724, so this repository cannot cover it until that
merges.
This is built on #764. Merge angr/binaries#184 first.
Validation: #804 (comment)
sync: angr/binaries#184
session: sharpen