Research Report Idclhash

IDCL Header +0x08 — Solved (derived from code, not a hash sweep)

Result: +0x08 is fully computable from the container itself. No Manifest-, Mount-, or Package Identity value is required. A self-built chunk is now possible — hardlinking a Retail chunk is no longer necessary.

Verified: 28/28 Retail and DLC archives, both header fields (+0x08 and +0x20) reproduced byte-for-byte.

The Formula

K = 0x9DDFEA08EB382D69
M = (1 << 64) - 1

h = farmhash64(header[0x10:0x78])     # FarmHash Fingerprint64, 104 Bytes
a = ((version ^ h) * K) & M           # version = header[0x04], u32
a ^= a >> 47
b = ((a ^ h) * K) & M
b ^= b >> 44
b = (b * K) & M
b ^= b >> 41
header_0x08 = (b * K) & M

+0x20 (already known, here as a positive control) is the same farmhash64 — without Finalizer — over the metadata of addr_entries (always 120) up to and including the final IDCL marker.

Order when building a chunk (important, fields depend on each other):

1. Complete the metadata block (including 4-byte IDCL marker at the end)
2. header[0x20] = farmhash64(metadata)
3. Fill the rest of the header
4. header[0x08] = header_hash(header, version) — hashes [0x10:0x78), so including the just-set +0x20

Reference implementation: tools/probe_idclhash.py (header_hash(), metadata_size()). Uses farmhash64 from tools/probe_tnc_hash.py unchanged.

Where the Code Is

The entry point was not the IDCL-Immediate — that does not exist in .text at all. The engine compares against a global in .data (RVA 0x2EF0F70, containing 49 44 43 4C). The function was found instead via RIP-relative Xrefs to the error strings in .rdata:

String (RVA)	Text	Xref in .text
0x26D08D8	container %s has invalid magic has: %x, wants %x	0xACB9BB
0x26D0A90	container %s is out of date version: has: %d, wants %d-%d	0xACC3AD
0x26D0AD0	container %s has corrupt header: hash: %llx, wants %llx	0xACBABC
0x26D0A30	...Container corrupted, end of metadata magic mismatch	0xACBC08
0x26D0FE0	container %s has corrupt metadata: hash: %llx, wants %llx	0xACBC33

All five lie in one function: idResourceStorage::OpenContainer, .text RVA 0xACB780–0xACC441 (boundaries from the .pdata-RUNTIME_FUNCTION table, not guessed).

The +0x08 Read Path, Disassembly

acb993 mov rax,[rsi] ; idFile-vtable acb996 mov r8d,0x10 ; Read 16 bytes acb99c lea rdx,[rsp+0x58] acb9a4 call [rax+0x28] ; Read -> [rsp+0x58]=magic, +0x5c=version, +0x60=header[0x08] acb9ac mov r9d,[rip+0x24255bd] ; -> .data 0x2EF0F70 = 'IDCL' acb9b3 cmp r8d,r9d ; Magic acba38 lea eax,[r9-0xb] acba3c cmp eax,1 ; Version in {11,12}, otherwise "out of date" acba4b cmp r9d,0xc acba4f jb 0xacbaf2 ; v11 path (see below) acba55 mov r8d,0x68 ; Read 104 bytes acba5b lea rdx,[rbp-0x80] ; == header[0x10] acba5f call [rax+0x28] acba62 mov edx,0x68 acba67 lea rcx,[rbp-0x80] acba6b call 0x1363b80 ; h = FarmHash64(header[0x10:0x78]) acba70 mov edx,[rsp+0x5c] ; version acba74 xor rdx,rax ; version ^ h acba77 movabs r9,0x9ddfea08eb382d69 acba81 imul rdx,r9 ; a = (version^h)*K acba85 mov rcx,rdx acba88 shr rcx,0x2f acba8c xor rdx,rcx ; a ^= a>>47 acba8f xor rdx,rax ; a ^ h acba92 imul rdx,r9 ; b = (a^h)*K acba96 mov r8,rdx acba99 shr r8,0x2c acba9d xor r8,rdx ; b ^= b>>44 acbaa0 imul r8,r9 ; b *= K acbaa4 mov rax,r8 acbaa7 shr rax,0x29 acbaab xor r8,rax ; b ^= b>>41 acbaae imul r8,r9 ; b *= K acbaba mov r9,[rsp+0x60] ; header[0x08] from file acbaba cmp r8,r9 acbaba je ok -> Log "corrupt header", file->Close(1), Mount fails

The comparison target is literally the u64 at file offset 0x08; the hashed region is literally [0x10,0x78), because the second Read fetches exactly 0x68 bytes starting from the file pointer 0x10.

+0x20 in the same code (Positive control within the binary itself)

acbbee  mov  rcx,[rsp+0x50]       ; metaSize
acbbf3  mov  r8,[rsp+0x48]        ; metaBuf
acbbf8  mov  eax,[r8+rcx-4]       ; last 4 bytes
acbbfd  cmp  eax,[rip+0x242536d]  ; -> 0x2EF0F70 'IDCL'   (Endmarker)
acbc15  cmp  dword [rsp+0x5c],0xc
acbc22  call 0x1363b80            ; same hash function
acbc27  mov  r9,[rbp-0x70]        ; == header[0x20]
acbc2b  cmp  rax,r9

This confirms that 0x1363b80 (exact .pdata function start 0x1363B80–0x1363C88, 11 direct callers) is uniquely FarmHash Fingerprint64 — because +0x20 is independently known as a FarmHash over the metadata, and our Python version reproduces it 28/28.

Metadata length, translated back from 0xACBB0E–0xACBB47

meta = 144*num_entries + 32*num_deps + 4*num_dep_indexes
     + 8*num_string_indexes + strings_size + 4

(the +4 is the 'IDCL' endmarker; header[0x38], [0x3C], [0x44] go in with factors 4/8/1, are all 0 in all Retail files). Matches exactly with the layout in wolfsdk/idcl.py.

Cross-check in the Writer

idResourceStorageGenerator::WriteContainer, .text 0xAC9BD0–0xACADD4, computes the same two values during writing:

acaa02  call 0x1363b80            ; FarmHash64(metadata) -> Header field +0x20
acaabc  lea  rcx,[rbp+0x90]       ; Header-Body, corresponds to header[0x10]
acaab7  mov  edx,0x68
acaac3  call 0x1363b80
acaacb  xor  rdx,0xc              ; version as literal 12
acaacf  movabs r8,0x9ddfea08eb382d69
        ... identical 47/44/41 chain ...
acab0a  mov  [rbp+0x40],rcx       ; -> Header field +0x08

Reader and Writer match instruction for instruction. This is the strongest evidence that nothing external influences this: the value is generated during writing from exactly the same bytes.

Controls

Positive control — re_probes/idclhash/derive_retail.txt

python tools/probe_idclhash.py
...
+0x08 28/28   +0x20 28/28

28 files / 27 distinct archives (16 base/, 8 patches, 3 DLC chunks, chunk_22 appears twice), with meta ranging from 7,584 to 14,478,124 bytes. The known field +0x20 is reproduced in all 28 cases — the tooling prerequisite is thus satisfied before any claim regarding +0x08 is made.

Negative controls — re_probes/idclhash/negative_controls.txt

Nine variants, each constructed such that they should have passed if the derivation were incorrect. All nine fail:

wrong base +16 (shifted header)                         failed as expected
hash over header[0x00:0x78] instead of [0x10:0x78]      failed as expected
hash over header[0x18:0x78]                             failed as expected
hash over header[0x10:0x70]                             failed as expected
finalizer without version xor                           failed as expected
plain FarmHash64(header[0x10:0x78]) without finalizer   failed as expected
metadata hash over meta[:-4] (marker excluded)          failed as expected
+0x20 unchanged after flipping one metadata byte        failed as expected
+0x08 unchanged after flipping header+0x20              failed as expected

python tools/probe_idclhash.py --negative yields an exit code ≠ 0 if any of these unexpectedly pass.

The old "counterexample" from hash08.md — resolved, not refuted

hash08.md observed that chunk_26_hash1 and chunk_26_oracle share the same stored +0x08 (3533df45c7f9b1d2) despite differing at +0x20, concluding: no container checksum. The observation is correct, but the conclusion is not — both files carry a copied, incorrect value for the file. Our derivation (re_probes/idclhash/custom_chunks.txt):

chunk_26_hash1          +0x08 3533df45c7f9b1d2/3533df45c7f9b1d2 OK   +0x20 BAD
chunk_26_hash2          +0x08 6e9d625602ddbc9e/6e9d625602ddbc9e OK   +0x20 OK
chunk_26_oracle         +0x08 3533df45c7f9b1d2/6e9d625602ddbc9e BAD  +0x20 OK
chunk_26_mountprobe     +0x08 ce1698c00e1cba48/3533df45c7f9b1d2 BAD  +0x20 BAD
chunk_26_mountprobe_v2  +0x08 ce1698c00e1cba48/6e9d625602ddbc9e BAD  +0x20 OK

Notable: for chunk_26_oracle we compute 6e9d625602ddbc9e — exactly the value noted in hash08.md as "the alternative not generated by any candidate". And ce1698c00e1cba48 (stored in both mountprobe files) is the +0x08 of gameresources.resources, i.e., the copied retail value. chunk_26_hash2 is already fully consistent.

Edge cases / other code implications

• Version 11 (0xACBAF2) has no header hash: only 0x60 bytes are read starting at 0x18; the u64 at 0x08 is shifted to header[0x10]; the header size is 0x70 instead of 0x78. The +0x20 check is also restricted to v12 via cmp [rsp+0x5c],0xc; jb. Retail is consistently v12.
• Accepted versions are 11 and 12 (lea eax,[r9-0xb]; cmp eax,1; ja).
• An incorrect +0x08 causes a hard mount error (warning + file->Close, return error), not an int3. An incorrect metadata endmarker is fatal instead (0x12386B0 + int3).
• The magic constant resides as a global in .data (0x2EF0F70), not as an immediate — therefore a byte scan for IDCL in .text finds nothing.

Artifacts

• tools/probe_idclhash.py — derivation + positive/negative controls (Py 3.11, stdlib; imports farmhash64 from tools/probe_tnc_hash.py)
• re_probes/idclhash/disasm_opencontainer.txt — disassembly evidence (Reader + Writer), generated with vendored capstone
• re_probes/idclhash/derive_retail.txt — 28/28
• re_probes/idclhash/negative_controls.txt — 9/9 fail as intended
• re_probes/idclhash/custom_chunks.txt — resolution of the old counterexample

Open points for others

• wolfsdk/idcl.py (foreign ownership) still contains in its docstring the statements "header +0x08 remains unknown and must likewise be preserved" and "The engine validates +0x08 when it mounts a container, so a freshly built container is not loadable until that value is calculated" — this has now been resolved and should be updated by the owner. A Writer must recalculate +0x20 and then +0x08 instead of receiving them.
• Remaining unresolved (not part of this task): entry+0x50 data_hash and entry+0x60 hash_0x60. For entries with unchanged payloads, they can simply be copied; they flow as bytes into the metadata and thus into +0x20 into +0x08, thereby not breaking the chain.