MAP_COMPRESSED Container Format

MAP_COMPRESSED Container Format

Status: 🟡 Partially cracked — the flat tile table is ✅ cracked and validated at scale; the geo-index prefix is now ✅ FULLY cracked — a real, on-disk k-d tree (not the R-tree hypothesized earlier from a different firmware trail — see the reconciliation note below), via DBAL.OUT disassembly + decompilation, with the split coordinate's own scale resolved (the disc-wide /100000-degree convention) and a real, fast, validated point-lookup/bbox-index now wired into the viewer.

The container used by eeuz.mp0, .mg1–.mg4 (files.cfg compressionType 2). See Tile Geometry and Topology for what's actually inside a tile.

Header

80 bytes, a different template from the plain SIEMENS header used elsewhere: (*$SIEMENS&^) + Copyright 1998 + version strings, no padding after — directory data starts immediately at byte 80.

Directory region (byte 80 → first tile)

Two sub-parts:

1. Geo-index prefix — ✅ FULLY CRACKED (a real k-d tree, edge's scale resolved)

Byte 80 → table_start (~87.7KB for mg4's 6,110 tiles, ~14.4 bytes/tile).

An earlier session found real evidence of an R-tree — mangled C++ symbols directly in the firmware's own FHDD6.FLI symbol table (see Firmware Reversing): MDCacheTIRTree, findNode(MapRect), enlargeMBR, splitNodeElements, intersectsRecursive — but could never reach the actual node/MBR byte layout (every nearby function turned out to be a thin wrapper whose real work happens past FHDD6.FLI's own 9MB+ disassembly wall).

A later session cracked the ACTUAL on-disk structure of this exact region, via a completely different route — DBAL.OUT disassembly (the same small, unlinked, no-disassembly-wall object file that cracked eeu.iof/eeu.rd), and it's a k-d tree, not an R-tree. db_kd_search, kd_pdecend/ kd_rdecend, db_page_kd_subtree, and db_upack_kd_top_node/ db_upack_kd_sub_node all decompile fully — unlike eeu.rd/eeu.iof's own field extraction, these functions have NO external/black-box calls, so their real logic was traced instruction-by-instruction, not guessed. These are very likely 2 genuinely different subsystems — MDCache ("Map Data Cache") plausibly names a higher-level, in-memory/UI-side spatial cache for live rendering, while db_kd_search is DBAL's own low-level mechanism for resolving a query rectangle against the PARCELS actually stored in this specific file. Not reconciled further than that; both findings stand as real, from independent evidence.

The structure, each piece validated separately:

byte 80 + [14-byte unidentified prefix] →
28-byte mapHeader:
    min_long, min_lat, max_long, max_lat   int32 LE each (16 bytes)
    layer                                   uint16 LE
    pcl_cnt                                 uint32 LE
    kd_cnt                                  uint32 LE
    kd_depth                                uint16 LE
→ 4-byte maxPclId (uint32 LE) -- named in eeu.mod's own schema, sits
  right after mapHeader and before the node array; an earlier session's
  reader skipped mapHeader but NOT this field, an easy-to-miss 4-byte
  bug that looked exactly like a real structural wall (see below) →
→ kd_cnt × 13-byte interleaved node records:
    edge        int32 LE   -- the real split coordinate, raw/100000 = degrees
    lo_index    int32 LE   -- dual-purpose, see below
    hi_index    int32 LE   -- dual-purpose, see below
    kd_info     1 byte     -- 7 packed flag bits, see below

• mapHeader's own position was found empirically (no known fixed- offset formula yet) by scanning for the first pcl_cnt-shaped value that EXACTLY matches read_directory(path)["num_tiles"] — on mg4: found at relative offset 14, pcl_cnt = 6,110, an exact match to the file's own independently-known real tile count. kd_cnt = 6,744, kd_depth = 22.
• The total region size is independently validated a 2nd way: 28 (mapHeader) + kd_cnt × 13 predicts the node array's end to within 4 bytes out of 87,718 (99.995%) against table_start (already established independently, from the flat tile table below) — 2 unrelated numeric signals landing almost exactly, not a coincidence.
• kd_info's 7 real bits, traced directly from kd_pdecend's own branch logic: cut(0), lo_is_subtree(1), lo_empty(2), lo_leaf(3), [unused(4)], hi_is_subtree(5), hi_empty(6), hi_leaf(7) — matches eeu.mod's own declared bit offsets for the 4 it separately names (2, 3, 6, 7) exactly; bits 1 and 5 aren't named in eeu.mod's schema but are real, confirmed by the code.
• lo_index/hi_index are dual-purpose, confirmed by tracing both branches: when neither the corresponding _leaf nor _subtree flag is set, the FULL 32-bit value is a plain recursive index into the SAME node array (empirically: hi_index lands in [0, kd_cnt) 100% of the time). When the _is_subtree flag is set instead, only the LOW BYTE is used — as a cache-slot id passed into db_page_kd_subtree, whose own decompiled code is a real LRU subtree cache (oslib_MutexGet(&s_KdCache_Lock), find_oldest_ subtree, a fixed-size slot table) — i.e. the design supports most of the tree living on disk as separate, lazily-loaded SUBTREES, not one flat array. In practice, NOT observed on any real layer tested (mg1-mg4, mp0) — every node resolves via the plain top-level array with 100% leaf coverage, so _is_subtree paging isn't actually exercised by this disc's own data, whatever headroom it gives larger theoretical datasets.
• edge is confirmed to be the real split coordinate, in the SAME /100000-degree convention used everywhere else on this disc. During descent, kd_pdecend copies the CURRENT query rectangle into both a "lo" and "hi" candidate rectangle, then — depending on cut (which axis) — overwrites ONE shared boundary field of each with edge (the hi rectangle's lower bound becomes edge+1; the lo rectangle's upper bound becomes edge) — textbook k-d tree splitting, confirmed by tracing the real code.

A previous session concluded edge's scale was a PROVEN HARD WALL — the same class as eeu.rd/eeu.iof's external schema-reader limit. That conclusion was wrong, and the error is instructive. The node array's assumed start offset (mapHeader's own 28 bytes, nothing more) was missing one field: eeu.mod's own schema (the disc's real data dictionary — see eeu.mod) lists a 4-byte maxPclId sitting between mapHeader and the kd_node table. Reading node 0 from 4 bytes too early meant EVERY field after the first one was reading garbage — lo_index references landing in the millions, nodes disconnected from each other, edge values that didn't match any plausible scale — which looked exactly like hitting the same external- reader wall previously believed proven for eeu.rd/eeu.iof's own field extraction. That belief turned out to be wrong too, resolved in the SAME session (see DBAL.OUT Full Decompilation's own correction): db_upack_kd_sub_node/_top_node/_sub_info/_top_info do go through mapView_o/getDataItem__C7DBFViewPCc and an opaque vtable call, but that call's real target is a real, DEFINED, compiled function sitting in DBAL.OUT's own .text — Ghidra's pseudocode just can't show it by name (that's what indirect/virtual dispatch always renders as), but .rela.text's relocations against the concrete class's own _vt$ClassName symbol resolve it exactly. The file contains the raw unpack logic after all — a full, real DBFInteger/ DBFIntegerSpecial/DBFBitComponent/... class hierarchy (621 DBF* symbols), including the actual generic integer codec (setValPtr__17DBFIntegerSpecialPcT1b, 9 real encoding schemes) and the bit-field shift/mask logic. None of that was needed to fix THIS specific bug (a plain 4-byte offset gap, found via eeu.mod's schema, not via the reader's code), but it means the "external reader" framing across this whole project was itself mistaken, not just inapplicable here — see the DBAL.OUT decompilation page for the full writeup.

Fixed and validated end-to-end, once the offset was corrected (mapHeader(28) + maxPclId(4) before the node array, not just 28):

• A full top-down traversal from node 0 now visits all 6,744 nodes and reaches all 6,110 leaves exactly once on mg4 — a perfect, zero-overlap partition of mapHeader's own range (previously: 1-2 nodes reachable, thousands of disconnected fragments).
• Treating edge/lo_index/hi_index as raw / 100000 degrees and reconstructing each leaf's box (root range narrowed by every ancestor's edge cut) against build_bbox_index()'s independently- decompressed real content boxes for the same tile id: median error ~0.003–0.013° (effectively exact), 90.2% full containment at a 0.02° tolerance. The remaining gap is build_bbox_index()'s own already- documented proxy imprecision (content doesn't always reach a tile's true edge), not a k-d tree decode error.
• A real point-lookup (walking kd_pdecend's exact single-path branch logic: compare the query coordinate on axis cut against edge, ≤ descends lo, > descends hi) returns, for 200 points sampled at the center of their own tree-derived box, the SAME tile id 200/200 times, and matches build_geo_index()'s own anchor-based tile for 262/300 random real anchors (the gap being anchor-point noise near tile boundaries, not a lookup error).
• A structural bbox index built from the tree (no tile decompression at all) reaches 100% of tiles on every layer size tested — mg4 6,110/6,110, mg3 16,503/16,503, mg2 34,209/34,209, mg1 63,996/63,996, and mp0 194,705/194,705 — vs. content-decode's partial coverage (e.g. mg4 5,935/6,110, mp0 194,650/194,705) and at a fraction of the cost (~2s combined for all 4 FAST_LAYERS, under 1s for mp0, vs. the content-decode path's "several minutes" for mp0 alone).

This is now a real, fast, firmware-accurate replacement for build_geo_index()/build_bbox_index()'s brute-force full-file decompression — see "Practical geo-index workaround" below for how the viewer now uses it, and the function docstrings for the full numbers. build_geo_index()/build_bbox_index() remain as documented, validated fallbacks (still used when a file's k-d tree can't be parsed, e.g. eeuz.fea, which doesn't use this table format at all, or if a future subtree-paging case is hit — not observed on any layer tested).

Reader code for the above: research/map_compressed_reader.py's read_kd_geo_index() (parses the full mapHeader + node array), find_tile_for_coord_kdtree() (real point lookup, ~15µs with a pre-parsed tree), and build_kdtree_bbox_index() (structural bbox index, no decompression) — all validated against the real ISO above.

A separate, earlier finding in the same region — scan_native_geo_hints(), still real and still useful

Before the k-d tree structure above was understood, a still-earlier session found that a large fraction of this region's small-integer pairs are literally (tile_id, tile_id + 1) — i.e. some records reference tiles by the SAME numbering the flat directory table uses (a rival "recursive child pointer" hypothesis was tested at scale on 2,275 real records and REFUTED: 0.6% hit rate, noise). This is NOT superseded by the k-d tree finding above — it's a different, real signal in the same byte range, not yet reconciled with the node records (possibly part of the leaf-level payload the lo_leaf/ hi_leaf-flagged entries point at, not yet traced that far).

scan_native_geo_hints() (research/map_compressed_reader.py) scans for [coord][n][n+1] triples on this finding, with NO tile decompression at all. Validated directly against build_bbox_index()'s (below) own already-validated bounds, on 2 independent files: mg4 2,796 candidates, 1,151/2,729 checkable exact-matched a real lon_max/lat_max (42.2%); mg3 8,000 candidates, 3,316/7,881 checkable matched (42.1%) — both files landing within 0.1 points of each other confirms this is real and reproducible. ~20-100x faster than full decompression (0.29s vs 6.6s on mg3's 16,503 tiles). Honestly scoped: recovers ONE bound (lon_max OR lat_max, not distinguishable from the value alone — confirmed this mechanism encodes only MAX-corner coordinates, never min-corner ones: 0/3,034 candidates matched a real min, vs. 652+499 matching a max, consistent with enlargeMBR's own real semantics, a running maximum) for ~42-46% of tiles, not a full bbox and not every tile.

4 more structural hypotheses tested against the real region bytes, all REFUTED: a width/height delta hidden in another record field (0/500 tiles); a tile storing BOTH axes as 2 separate records (0/705 tiles with a confirmed hit had a 2nd one); consecutive tile_ids sharing an STR-bulk-load "strip" axis (median 1.4° spread across 5-consecutive-tile_id windows, nowhere near strip-tight — re- confirms tile_id file order isn't a clean spatial order). Still open, no hypothesis queued: why a tile's record picks lon_max vs. lat_max (no discriminating rule found), and what the ~40% of non- matching candidates actually are (plausibly scan false positives, not confirmed).

Wired into the map viewer's live tile-selection path, then REVERTED the same session — a real, measured performance regression, walked back honestly: testing a hint's single, axis-ambiguous coordinate against a query bbox's lon-OR-lat range independently looked fine on a small hand-picked test box but inflated real pooled- tile counts 4.5-10x at realistic viewport scale (confirmed structural from ~170m to ~21km box spans, not a large-viewport-only issue) — every extra tile costs a real decode, exactly the reported slowdown. See Map Viewer Tool's own "v32 → v37" changelog entry for the full add-then-revert story. scan_native_geo_hints() itself is unaffected and still built (cheap, ~4.35s), just no longer consumed by the viewer's tile selection. The natural next step, if revisited, is a fast first-pass hint layer with a properly 2D-correlated containment test (not 2 independent 1D tests), falling back to full decompression only where that doesn't resolve.

2. Flat offset/length table — ✅ cracked, validated at scale

table_start → first tile: a fixed 8-byte record per tile, in physical tile order:

uint32 LE offset            -- absolute file offset of the tile
uint16 LE decompressed_len
uint16 LE compressed_len    -- == byte gap to the next tile

Validated by exact match against real decompressed data: 6,110/6,110 tiles on mg4, 16,503/16,503 on mg3, 34,209/34,209 on mg2, 80/80 sampled on mp0. Caveat: lengths are uint16, capping any single tile at 65,535 bytes (comfortably true on every file checked so far).

This is enough to resize/relocate/edit a tile that already exists and is already known by ID — rewrite its entry, shift every later tile's offset by the size delta.

eeuz.fea does not use this table format

Its first real tile was found fine, but the literal offset value doesn't appear anywhere in its ~11.8MB pre-tile region — a genuinely different directory encoding. See eeuz.fea.

Tile data

After the directory: hundreds of thousands of independent, standard RFC1950 zlib streams, packed back-to-back with zero padding between them (verified: each tile's consumed byte length exactly equals the offset gap to the next one).

Practical geo-index workaround

Since every tile carries its own absolute anchor coordinate (see Tile Geometry and Topology), build_geo_index(path) sidesteps the uncracked geo-index prefix entirely: decompress every tile, extract its anchor, return {tile_id: (lon, lat, method)}; find_tile_for_coord() does a nearest-anchor lookup over that.

Superseded, as of the k-d tree edge-scale resolution above: this whole section documents the path taken while edge's scale was still unresolved -- build_geo_index()/build_bbox_index() (decompress every tile) plus a persistent on-disk cache (build_geo_index_cached()/build_bbox_index_cached(), same module, keyed by the ISO's own identity + exact byte span) to avoid re-paying that cost on every re-open. Both are still real, still correct, and kept as the documented fallback for files the k-d tree can't parse (eeuz.fea) -- but the viewer's own bbox lookup (MapData._build_layer_bbox_index()) now tries build_kdtree_bbox_index() FIRST: a single structural tree traversal, no decompression at all, reaching 100% of tiles on every layer size tested (vs. this section's content-decode coverage numbers below) in ~2s combined for all 4 FAST_LAYERS and under 1s for the full 2.14GB mp0 (vs. "~55s" and up to "several minutes" below). The cache wrappers remain useful history and a correct fallback, not the primary path anymore. Previously verified (still true, now the 2nd-choice path): cached and freshly-built results are bit-for-bit identical for both indexes; a full load() dropped from 37.76s to 6.89s on a second run against the real reference ISO, and a single layer's geo_index alone dropped from 1.30s to 0.010s (130x) once cached.

Coverage on full-file runs (not samples): 6,106/6,110 (99.93%) on mg4 (~0.6s), 16,495/16,503 (99.95%) on mg3, 34,200/34,209 (99.97%) on mg2, 63,970/63,996 (99.96%) on mg1, and 194,650/194,705 (99.97%) on the full 2.14GB mp0 (~55s).

Known caveat: nearest-anchor is not true bounding-box containment (no tile extent was recovered), so a coordinate near a tile boundary might not resolve to the tile that actually renders it at a given zoom level.

A real, validated, statistical alternative to nearest-anchor: no stored tile width/height FIELD exists (the 0x02-0x17 per-tile header once suspected as "the likely home for this" is now fully accounted for by decode_tile_header() — see Tile Geometry and Topology — none of its 12 words are a width/height; that lead is closed). But real road geometry is clipped exactly at tile boundaries, so a tile's own decoded CONTENT bbox (decode_features()'s point-cloud envelope, nothing new to crack) frequently extends to the tile's true edge on any side with a populated neighbor. Measured at full-layer scale, both mg3 (16,203/16,503 tiles) and mg2 (33,677/34,209 tiles), both lon and lat directions: median gap between geographically-adjacent tiles' own content bboxes is 8-24 meters in every case — roughly half of all measured neighbor pairs touch within ~85m, 61-66% within 170m, 72-78% within 425m. build_bbox_index() + find_tile_for_coord_bbox(bbox_index, geo_index, lon, lat, pad_deg=0.002) prefers a tile whose own (padded) content bbox actually contains the query point, falling back to plain nearest- anchor when no bbox matches — a strict improvement, never worse. Validated: a point drawn from inside a known tile's own content correctly resolves back to that tile; a point in open ocean correctly falls back to nearest-anchor rather than a false containment match; 200 random real interior points across mg3 resolved to their own originating tile 196/200 times (98%) via the "bbox" method, the 4 misses being genuine small-tile-inside-larger-tile overlap ambiguity, not a bug. Wired into the map viewer's own tile_ids_in_bbox() — see Map Viewer Tool's "v32 → v33" changelog entry. Honestly scoped: a statistically validated proxy built from already-decodable data, not a decoded format field — a sparse/rural tile whose content doesn't reach a particular edge will under-report its true extent on that side (why the nearest-anchor fallback exists, rather than ever trusting a bbox miss as proof).

Also noted: decompress_tile()'s consumed byte count is unreliable for ~13% of tiles with default settings (a zlib buffering quirk, not data corruption — declen/content match is unaffected) — don't rely on consumed for anything correctness-sensitive; use the flat table's own complen field instead.

Reader code

research/map_compressed_reader.py — find_tiles(), find_first_tile(), read_directory(), decompress_tile(), pack_table_entry(), build_geo_index(), find_tile_for_coord(), build_bbox_index(), find_tile_for_coord_bbox(), scan_native_geo_hints(), read_kd_geo_index() (the real k-d tree parser), find_tile_for_coord_kdtree() (real point lookup, no decompression), build_kdtree_bbox_index() (structural bbox index, no decompression -- now the primary path, see "Practical geo-index workaround" above), build_geo_index_cached()/build_bbox_index_cached() (persistent disk-cache wrappers around the two decode-every-tile builders, now the documented fallback).