Pack Format Overview

Pack format overview

A pack is how vgl stores baked terrain, imagery and the geoid: a directory of plain binary files laid out so that a reader can fetch any tile with a few HTTP range requests from any static file host. This page is an integrator's summary of the pack format specification (format version 1.0): the files in a pack, the tiling scheme, the manifest, shards, tile encodings, versioning and how reads map to range requests. You do not need to read or write packs yourself: the baker writes them and the runtime reads them. Read this to plan bakes, hosting and caching, or to debug a pack. The specification is the source of truth for every byte; follow its links for detail.

A pack at a glance

A pack holds one layer for some part of the globe:

Layer	layer_type	Contents
Elevation	1	Ellipsoidal heights on a grid of samples
Imagery	2	sRGB images, JPEG or PNG (the format also defines WebP, BC1 and BC7, not yet supported)
Geoid	3	EGM2008 geoid heights, for converting to and from mean sea level

The runtime stacks several packs per layer by priority, so a global base pack and regional high-resolution packs combine without rebaking. A dataset with elevation and imagery ships as two packs. A context takes at most one geoid pack.

On disk (or on a web server) a pack is a directory with three kinds of file:

<pack>/
    pack.vglm                     Manifest: metadata, layer description, shard index
    base-<digest>.vgls            Base shard: all tiles above the shard level
    shards/<y>/<x>-<digest>.vgls  One shard per tile at the shard level, for the subtree below it

The URI you pass to vgl_pack_create is the directory, without a trailing /; the runtime appends /pack.vglm and the shard paths itself. The same layout works for local files and URLs (Hosting packs).

Tiling scheme

Spec: section 2.

• A geographic (equirectangular) quadtree on longitude and geodetic latitude, on the WGS84 ellipsoid.
• Level 0 has two tiles of 180 by 180 degrees: (0, 0, 0) is the western hemisphere, (0, 1, 0) the eastern.
• Level L has 2^(L+1) columns and 2^L rows, each tile 180 / 2^L degrees on a side. Columns count east from the antimeridian, rows count south from the north pole (so tile rows run in the same direction as image rows).
• Children of (L, x, y) are (L+1, 2x+i, 2y+j), numbered 0 north-west, 1 north-east, 2 south-west, 3 south-east.
• Valid levels are 0 to 28; a pack uses any contiguous range.
• Tile keys are (L << 58) | morton(x, y), so siblings and whole subtrees are contiguous in key order.

Elevation tiles are grid-registered: a tile of size N (2^k + 1, 129 by default) has N x N samples on its edges, and neighbours share their border rows and columns, which must decode identically. Pole rows hold one value.

Imagery tiles are cell-registered: a tile of P pixels (a power of two, 256 by default) with border b (0 or 1, 1 by default) is a (P + 2b) square image. The border repeats the neighbours' pixels so textures filter across tile edges without seams.

Spacing for 129-sample elevation tiles at the equator:

Level	Tile span	Sample spacing	Tiles in full level
0	180°	156.5 km	2
4	11.25°	9.8 km	512
6	2.8125°	2.4 km	8,192
8	0.703125°	611.5 m	131,072
10	0.17578125°	152.9 m	2,097,152
11	0.087890625°	76.4 m	8,388,608
12	0.0439453125°	38.2 m	33,554,432
13	0.02197265625°	19.1 m	134,217,728

The runtime's tiling functions are described in Geodesy and tiling.

The manifest

Spec: section 4.

pack.vglm is a small binary file, read once when the pack is added. It is binary rather than JSON so that the runtime needs no JSON parser; vgl-bake inspect prints it as text. It is a 32-byte header (magic VGLM, major and minor version, checksums) and a table of sections, each optionally Zstandard-compressed and covered by a CRC-32C:

Section	Contents
LAYER	Layer type, tiling scheme, tile size, min_level, max_level, shard_level, imagery border, height reference, ellipsoid (WGS84) and informative coverage bounds
HEIGHT_STEPS	Elevation and geoid: the quantization step of each level, a power of two in metres
SHARD_INDEX	Whether the base shard exists, and every level-S shard that exists, with its file size and digest
METADATA	Key and value strings: name, description, attribution (repeatable), license, source, generator, created

The runtime reports the attribution entries of the packs in view through vgl_context_attribution_get; licences typically require you to show them (Attribution and licensing). A runtime rejects a pack whose ellipsoid differs from its context's.

The manifest is the only file in a pack that changes in place when the pack is rebaked, which is why hosts give it a short cache lifetime.

Shards

Spec: section 5.

The manifest's shard_level S splits the pack:

• The base shard holds every tile from min_level to S - 1. It exists if any such tile does.
• Each level-S shard holds the subtree under one tile (S, x, y), down to max_level.
• If S is above max_level, the base shard is the whole pack (as in the sample packs and the geoid pack). If S is at or below min_level, there is no base shard.

The baker defaults to S = max_level - 5, which gave shards of about 10 to 25 MB in the test bake: large enough to keep file counts low, small enough to copy, update and cache by region.

A shard file is a 64-byte header (magic VGLS), then a Zstandard-compressed directory of 24-byte entries (tile key, offset, size, CRC-32C), then the tiles in key order with no gaps. The header and directory of a typical shard fit in its first 64 KiB. Because of Morton order, the four children of a tile are adjacent and can be read with one range request.

Digests in file names. A shard's digest is its header CRC, which covers the directory, which covers every tile's CRC. It is part of the file name (eight lowercase hex digits) and is recorded in the manifest with the file size. So:

• A rebaked shard with different content gets a new name; shard files never change once written.
• Web servers and caches may keep shards forever (immutable), and the runtime's disk cache never needs to revalidate them.
• A reader detects a stale cache or a shard from a different bake by comparing the header with the manifest.
• Updating a pack is atomic for readers: upload new shards, then replace the manifest, then delete the old shards (Hosting packs).

Tile encodings

Spec: section 6.

Every tile starts with an 8-byte header: layer type, encoding, a child mask (which of its four children exist in this pack, so the runtime knows whether it can refine without reading the next shard), flags and the header size.

Elevation and geoid tiles add a height offset, the level's quantization step, and the minimum and maximum height of each child (so the runtime can cull children before loading them). Samples are 16-bit quantized values (offset + scale * q), with -32768 meaning nodata. Encoding 1, the only one, applies a MED (LOCO-I) predictor, zigzag-encodes the residuals, splits them into low and high byte planes and compresses with Zstandard: lossless on the quantized values. Quantization is on each level's global grid of steps, so shared borders decode identically. Heights are always ellipsoidal; the baker converts sources referenced to mean sea level with the geoid pack. Tiles with some nodata samples are partial (typical at the edges of regional packs), and the runtime fills them from lower-priority packs.

Imagery tiles are images in sRGB, with straight alpha where present (alpha 0 is no data):

Encoding	Payload	Written by the baker	Read by the runtime today
1	JPEG	Yes (tiles with data everywhere, by default)	Yes, decoded with stb_image
2	PNG	Yes (tiles with gaps, with alpha)	Yes, decoded with stb_image
3	WebP	No	No
4	BC1 blocks, Zstandard	No	No
5	BC7 blocks, Zstandard	No	No

A pack may mix encodings. The runtime reports VGL_RESULT_NOT_SUPPORTED for a tile in an encoding it cannot read. The specification foresees direct upload of BC1 and BC7 blocks and a pluggable decoder interface for WebP; neither is implemented yet, so bake imagery as JPEG and PNG (the baker's only encodings).

Geoid packs cover the whole globe at every level without nodata. The default geoid pack is EGM2008 at levels 0 to 5 with 129-sample tiles and a step of 2^-8 m, as one base shard of about 22 MB (Baker).

Reading a tile with range requests

Spec: section 1.1.

Every structure can be located from data read earlier, so a reader needs only plain files and byte ranges, no server code:

1. Read pack.vglm (one whole-file read) and keep it in memory.
2. Find the shard: the base shard above the shard level, otherwise the shard of the tile's level-S ancestor. If the shard index does not list it, the tile does not exist.
3. If the shard's directory is not cached, read the header and directory: one range request for the first 64 KiB usually covers both.
4. Binary-search the directory for the tile key.
5. Read the tile's byte range, check its CRC and decode it.

So a host must answer Range: bytes=a-b with 206 Partial Content, must not compress pack files (a range of a compressed response is not a range of the file), and should let browsers on other sites read them with CORS. The defaults module's HTTP reader issues these requests (Defaults module); Hosting packs covers the server side. If a server ignores ranges and sends whole files, the reader still works but downloads a whole shard for every tile, which vgl-bake check reports as an error.

Versioning

Spec: section 7.

The format version (1.0) is independent of the library version. Manifests and shards each carry a major and minor version.

• A major version change may change anything; readers reject unknown major versions with VGL_RESULT_VERSION_MISMATCH.
• A minor version keeps files readable by older readers of the same major version. It may add optional manifest sections, append fields to sections and tile headers (readers skip them by size), define reserved fields and flags (zero keeps the old meaning), and add tile encodings and metadata keys.
• Writers set the lowest minor version that defines what the file uses. Readers accept every minor version up to their own, and for a newer one report VGL_RESULT_VERSION_MISMATCH only on meeting an unknown required section or encoding.

In practice: packs baked by one 1.x baker are readable by any 1.x runtime, except tiles in encodings that runtime does not know. The CHANGELOG notes every change to the format.

Validation

Spec: section 8.

Readers check magic numbers, versions, every CRC, sizes, ranges and sort orders, and never read outside a file or buffer because of its contents. Failures are reported as VGL_RESULT_CORRUPT_DATA (or VGL_RESULT_VERSION_MISMATCH, VGL_RESULT_NOT_SUPPORTED as above), and a pack that cannot be used goes to the failed state (vgl_pack_state_get). On the baking side, vgl-bake validate checks a whole pack, including seams, poles and child masks, before you publish it.

See also

• Baker: writing packs, and choosing tile size, levels and shard level.
• Hosting packs and Reference server: serving them.
• Context API: vgl_pack_create, priorities, opacity and pack states.
• docs/test-bake.md: how the default parameters were chosen.