Hosting Packs

Hosting packs

vgl hosts no data: an application that uses it bakes the elevation and imagery it needs with the baker and publishes the packs where it chooses. Packs are plain files read with HTTP range requests, so hosting them needs no server code: a local directory, an object store, any static web server or the reference server will do. This page covers serving packs from local disk and from web hosts, the HTTP features a host must have, setting up a public endpoint, access control, uploading, updating packs safely, checking a host with vgl-bake check, and connecting the runtime with the defaults module's HTTP reader and disk cache. It expands the hosting guide in the repository.

Contents

• Choosing where packs live
• Packs on local disk
• What a web host must do
• Object stores and static hosts
• Setting up a public endpoint
• Access control
• Uploading packs
• Versioning and updating packs
• Checking a host with vgl-bake check
• Connecting the runtime
• Pitfalls

Choosing where packs live

	Local disk	Object store	Reference server	Other web server or CDN
Examples	Installed with the application, a network share	Amazon S3, Cloudflare R2, Backblaze B2, Azure Blob Storage, Google Cloud Storage	Caddy with tools/server/Caddyfile	nginx, Apache, IIS, a static site host
Runtime reader	File reader	HTTP reader	HTTP reader	HTTP reader
HTTPS	Not needed	Provided	Automatic (Let's Encrypt or ZeroSSL)	Your configuration
Maintenance	None	None	Caddy and operating system updates	Your configuration
Suits	Offline use, fixed regions, kiosks, simulators	Public datasets, most installations	Local networks, private or offline sites, development, full control	Existing infrastructure

You can mix them: the runtime reads each pack through the same I/O interface, and the HTTP reader passes URIs that are not http:// or https:// to a fallback reader, so one context can load a geoid pack from disk and terrain from a server.

Packs on local disk

Copy each pack's directory as it is (pack.vglm and its *.vgls files, keeping the shards/ paths), and pass the directory to vgl_pack_create as its uri, with the defaults module's file reader (vgl_file_reader) or your own I/O interface (Host interfaces, Defaults module). Leave out the baker's .bake directory, which is only for resuming a bake.

On disk there is nothing more to configure. Sharding still helps: an installer or downloader can copy only the shards of the regions a user needs, since the runtime logs an error for a shard the manifest lists but that cannot be read, treats its tiles as absent, and draws coarser levels or lower-priority packs there.

What a web host must do

The runtime reads pack.vglm whole, then reads byte ranges of shards (first a header and directory, usually within the first 64 KiB, then individual tiles). A host must:

Requirement	Why	If missing
Answer Range: bytes=a-b and bytes=a- with 206 Partial Content and the right Content-Range	Every tile read is a range	The HTTP reader keeps working but downloads a whole shard (10 to 25 MB) for every tile: vgl-bake check reports an error
Answer a range past the end with 416	The reader probes with speculative ranges	Reported as a warning
Not compress pack files (no Content-Encoding: gzip, br or zstd)	Compression applies to the whole body, so a range of a compressed response is not a range of the file. Tiles are already compressed	Corrupt reads: an error
Send Access-Control-Allow-Origin and answer CORS preflight requests for Range	Browsers on other sites, including vgl's web build, read packs only with CORS	Browsers on other sites cannot read the packs: a warning
Send Cross-Origin-Resource-Policy: cross-origin (recommended)	vgl's web build runs in cross-origin isolated pages, which load cross-origin resources only through CORS or with this header
Cache shards for long, the manifest briefly	Shard names hold a digest, so shards never change; the manifest changes in place on a rebake	Warnings: caches either hold stale manifests or refetch shards
Send ETag (recommended)	A manifest recheck then costs a 304 Not Modified
Serve pack files as application/octet-stream (recommended)	Keeps CDNs and proxies from compressing or transforming them

The headers that browsers need exposed to scripts are Accept-Ranges, Content-Length, Content-Range and ETag; the request headers a preflight must allow are Range, If-None-Match and If-Range.

Recommended cache headers:

File	Cache-Control
*.vgls (shards)	public, max-age=31536000, immutable
pack.vglm (manifest)	public, max-age=60

Object stores and static hosts

Object stores serve range requests and HTTPS out of the box, and need no maintenance, so they suit most installations. For each, upload the pack with the right content type and cache headers (see Uploading packs) and, if browsers on other sites read it, configure CORS. Then point the runtime at https://<host>/<prefix>/<pack>, and run vgl-bake check on that URL.

S3 and S3-compatible stores (Amazon S3, Cloudflare R2, Backblaze B2, MinIO and others) accept the S3 CORS form, which object stores generally accept:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["*"],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedHeaders": ["Range", "If-None-Match", "If-Range"],
      "ExposeHeaders": ["Accept-Ranges", "Content-Length", "Content-Range", "ETag"],
      "MaxAgeSeconds": 86400
    }
  ]
}

With the AWS CLI (other S3-compatible stores take --endpoint-url):

aws s3api put-bucket-cors --bucket my-packs --cors-configuration file://cors.json

Make the objects publicly readable through the store's public access setting, a bucket policy or a public bucket domain, as your store provides.

Azure Blob Storage. Serve from a container with anonymous read access for blobs (or through a CDN in front of it), and set CORS on the storage account's blob service:

az storage cors add --account-name mypacks --services b \
    --methods GET HEAD OPTIONS --origins '*' \
    --allowed-headers 'Range,If-None-Match,If-Range' \
    --exposed-headers 'Accept-Ranges,Content-Length,Content-Range,ETag' \
    --max-age 86400

Google Cloud Storage. Make the objects publicly readable and set CORS on the bucket from a JSON file:

[
  {
    "origin": ["*"],
    "method": ["GET", "HEAD"],
    "responseHeader": ["Accept-Ranges", "Content-Length", "Content-Range", "ETag", "Range"],
    "maxAgeSeconds": 86400
  }
]

gcloud storage buckets update gs://my-packs --cors-file=cors.json

Do not upload pack files with gzip content encoding (gsutil cp -z or -Z): GCS then serves them with decompressive transcoding, which ignores ranges.

Any other static web server (nginx, Apache, IIS, a static site host) works if it meets the requirements. Most answer ranges for static files by default; check that compression is off for *.vgls and pack.vglm, add the CORS and cache headers, and run vgl-bake check. The reference server's Caddyfile is a complete example of the headers to set.

The CLI commands above are the usual forms for each provider; providers change their tools, so treat vgl-bake check as the final word on whether a host is configured correctly.

Setting up a public endpoint

1. A domain. Give the packs a host name of their own, such as packs.example.org, so you can move between hosts and add a CDN without changing the URIs built into applications.

2. TLS. Serve HTTPS only. Object stores and CDNs provide certificates for their domains and usually for custom domains; the reference server obtains and renews them itself when VGL_SITE is a domain name, and redirects HTTP to HTTPS. Browsers block plain HTTP packs from HTTPS pages.

3. A CDN in front (optional, recommended for public data read by many users). A CDN caches the immutable shards at its edge and absorbs floods of requests, which neither an object store's public endpoint nor Caddy alone limits well. Make sure it:

   • passes Range requests through and caches partial content (most CDNs do for application/octet-stream);
   • does not compress pack files;
   • keeps the origin's Cache-Control (or set the same lifetimes at the CDN: long for *.vgls, a minute for pack.vglm);
   • passes the CORS headers through, or adds them.

   With the reference server behind a CDN that terminates TLS, set VGL_SITE to a port (such as :8080) so Caddy serves plain HTTP to the CDN.

4. Rate limits. Neither the HTTP reader nor the reference server limits clients. Put a CDN in front, or limit connections per address in a firewall, so that one client cannot take all the bandwidth.

5. Attribution. Publish the credit lines each dataset's license requires. They travel inside the manifests, and the runtime reports them with vgl_context_attribution_get so applications can show them (Attribution and licensing).

6. Check it with vgl-bake check from outside your network.

Access control

Packs are usually public data, and the pack format has no access control of its own. The options:

• CORS origin. Naming your own sites instead of * in the CORS rules (or VGL_CORS_ORIGIN for the reference server) keeps other sites from using your packs from browsers. It does not stop native programs or scripts: it is a bandwidth measure, not security.
• Network restrictions. Serve on a private network, a VPN, or behind an IP allow list in the firewall or CDN, for internal or offline sites.
• Authentication. The defaults module's HTTP reader sends no credentials or custom headers (its options are the connection count, timeout, user agent, allocator and fallback). Signed URLs with a query string do not fit either, because the runtime appends file names to the pack's URI. For authenticated access, give the context your own I/O interface that adds the credentials your host needs, for example on your engine's network layer (Host interfaces). The disk cache can wrap such an interface too.

Uploading packs

1. Check the pack where it was baked: vgl-bake validate <pack> reads every tile.
2. Upload the pack's directory as it is, keeping its paths (pack.vglm, base-*.vgls, shards/<y>/<x>-*.vgls), without the .bake directory. Send pack files as application/octet-stream, uncompressed, with long caching for shards and short caching for the manifest. Upload shards first and the manifest last.
3. Configure CORS if browsers on other sites read the packs.
4. Check the result with vgl-bake check.

With rclone, which uploads to every common object store (S3 and compatible stores, Azure Blob, Google Cloud Storage, and more), that is two passes, shards first:

rclone copy packs/glo30 store:bucket/glo30 --include "*.vgls" \
    --header-upload "Content-Type: application/octet-stream" \
    --header-upload "Cache-Control: public, max-age=31536000, immutable"
rclone copy packs/glo30 store:bucket/glo30 --include "pack.vglm" \
    --header-upload "Content-Type: application/octet-stream" \
    --header-upload "Cache-Control: public, max-age=60"

With the AWS CLI:

aws s3 sync packs/glo30 s3://my-packs/glo30 --exclude "*" --include "*.vgls" \
    --content-type application/octet-stream \
    --cache-control "public, max-age=31536000, immutable"
aws s3 cp packs/glo30/pack.vglm s3://my-packs/glo30/pack.vglm \
    --content-type application/octet-stream --cache-control "public, max-age=60"

For the reference server, copy the pack directory under VGL_PACKS (with rsync -a --exclude .bake, for example); its configuration sets the headers itself.

Versioning and updating packs

A rebaked shard gets a new name, and the manifest is the only file that changes in place. To update a pack without readers ever seeing a mix of old and new:

1. Upload the new shards. Readers of the old manifest do not see them. Because bakes are deterministic, shards whose sources did not change keep their names, so a sync uploads only the changed ones.
2. Replace the manifest at once. Upload it under a temporary name and rename it, or rely on the object store's atomic writes of single objects.
3. Delete the old shards that the new manifest no longer names, but only after caches have moved on: a minute for the reference server's manifests (max-age=60), longer behind a CDN (or purge the manifest at the CDN).

A reader that still holds the old manifest and then reads a shard that has been deleted marks that shard as failed and treats its tiles as absent until the pack is created again; the runtime's disk cache keeps working offline with the shards it already has.

For larger changes, version the location instead: publish the new bake under a new path (/glo30-2027/) and switch applications to it when ready, keeping the old path until no one uses it. Use a new path whenever the change is not a drop-in replacement, for example a different tile size, levels, or a new geoid that the elevation was baked against. A pack's format version is recorded inside its files and is independent of the URL (Pack format overview).

Checking a host with vgl-bake check

vgl-bake check <url> checks any host, by the URL of a pack's directory (with or without a trailing /pack.vglm), as vgl and browsers read it:

• the manifest loads and parses, and is not compressed;
• byte ranges get 206 with the right bytes, and a range past the end gets 416;
• browsers on other sites may read it: Access-Control-Allow-Origin, and an answer to a CORS preflight for Range (it sends requests with Origin: https://example.org);
• the manifest is not cached for long, and shards are;
• every shard the manifest names exists, with its size and digest (--sample N checks N of them, spread over the pack);
• one shard's directory and a tile, read with ranges as vgl reads them, match their checksums.

It reports each check as ok, warning or error, and exits with 1 on an error. Errors are what stops vgl reading the pack, or makes every read download a whole shard; warnings are what stops browsers on other sites, or caches, from doing their part.

$ vgl-bake check https://packs.example.org/gebco
ok       manifest: GEBCO 2026 (elevation, levels 0 to 8, 129 shards)
ok       CORS: Access-Control-Allow-Origin: *
ok       byte ranges: 206 Partial Content, Content-Range: bytes 0-31/1843
ok       CORS preflight for Range: 204
ok       shards: 129 of 129 checked exist with the size and digest the manifest gives
ok       a shard read as vgl reads it: header, directory and a tile check out

0 error(s), 0 warning(s)

Option	Default	Meaning
--sample <n>	every shard	Check n shards, spread over the pack
--timeout <s>	30	Seconds a request may take

The command needs a baker built with libcurl (VGL_DEFAULTS_HTTP, which the baker presets turn on; see Baker). Run it after every upload, from a machine outside the host's network, and through the CDN's URL if there is one.

Connecting the runtime

The core library opens no network connections: it reads packs through the I/O interface the host gives the context. The defaults module provides three readers that chain (Defaults module):

Reader	Role
vgl_file_reader	Reads local paths
vgl_http_reader	Reads http:// and https:// URIs with range requests, over a few connections per server (8 by default), on a thread of its own. Other URIs go to its fallback interface, such as the file reader's. Built on libcurl with VGL_DEFAULTS_HTTP; without it, vgl_http_reader_create returns VGL_RESULT_NOT_SUPPORTED. On Emscripten it uses the browser's fetch instead
vgl_disk_cache	Wraps another interface and keeps what it reads from web servers in a directory, within a budget (1 GiB by default), for next time and offline. Shards are never stale; manifests are always fetched again, and served from disk only when the server cannot be reached

Build vgl with -DVGL_DEFAULTS_HTTP=ON (the shared presets do; see Building vgl). libcurl is used if installed, otherwise curl is downloaded and built with HTTP and HTTPS only, on the system's TLS.

In C

#include <vgl/defaults.h>
#include <vgl/vgl.h>

#include <stdio.h>

int main(void)
{
    vgl_thread_pool* pool = NULL;
    vgl_file_reader* files = NULL;
    vgl_http_reader* http = NULL;
    vgl_disk_cache* cache = NULL;
    vgl_context* context = NULL;
    vgl_pack* terrain = NULL;
    vgl_pack* geoid = NULL;
    vgl_task_interface tasks;
    vgl_io_interface file_io;
    vgl_io_interface http_io;
    vgl_io_interface cache_io;
    vgl_result r;

    if (vgl_thread_pool_create(NULL, &pool) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }

    /* Local paths: the file reader, on the pool. */
    vgl_file_reader_create_info file_info = {0};
    file_info.struct_size = sizeof(file_info);
    file_info.pool = pool;
    if (vgl_file_reader_create(&file_info, &files) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }
    vgl_file_reader_io_interface_get(files, &file_io);

    /* http:// and https://: the HTTP reader, with the file reader for everything else. */
    vgl_http_reader_create_info http_info = {0};
    http_info.struct_size = sizeof(http_info);
    http_info.fallback = &file_io;
    r = vgl_http_reader_create(&http_info, &http);
    if (r != VGL_RESULT_SUCCESS)
    {
        /* VGL_RESULT_NOT_SUPPORTED: vgl was built without VGL_DEFAULTS_HTTP. */
        fprintf(stderr, "no HTTP reader (%d)\n", (int)r);
        return 1;
    }
    vgl_http_reader_io_interface_get(http, &http_io);

    /* A disk cache in front of it, for next time and offline. */
    vgl_disk_cache_create_info cache_info = {0};
    cache_info.struct_size = sizeof(cache_info);
    cache_info.directory = "vgl-cache";
    cache_info.source = &http_io;
    cache_info.budget = 2ull << 30; /* 2 GiB; 0 selects 1 GiB. */
    cache_info.pool = pool;
    if (vgl_disk_cache_create(&cache_info, &cache) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }
    vgl_disk_cache_io_interface_get(cache, &cache_io);

    /* The context reads through the cache. */
    vgl_thread_pool_task_interface_get(pool, &tasks);
    vgl_context_create_info context_info = {0};
    context_info.struct_size = sizeof(context_info);
    context_info.tasks = &tasks;
    context_info.io = &cache_io;
    if (vgl_context_create(&context_info, &context) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }

    /* A pack on a web server and one on disk. The URI is the pack's directory, without a
     * trailing slash. */
    vgl_pack_create_info pack_info = {0};
    pack_info.struct_size = sizeof(pack_info);
    pack_info.uri = "https://packs.example.org/gebco";
    if (vgl_pack_create(context, &pack_info, &terrain) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }
    pack_info.uri = "packs/egm2008-geoid";
    if (vgl_pack_create(context, &pack_info, &geoid) != VGL_RESULT_SUCCESS)
    {
        return 1;
    }

    /* ... set the view, call vgl_context_update and draw each frame ... */

    /* Destroy in reverse order: the context, then the cache, the HTTP reader, the file reader
     * and the pool. Destroying the context frees its packs. */
    vgl_context_destroy(context);
    vgl_disk_cache_destroy(cache);
    vgl_http_reader_destroy(http);
    vgl_file_reader_destroy(files);
    vgl_thread_pool_destroy(pool);
    return 0;
}

The structures are zero-initialized and then filled field by field, so that the padding fields some structures have on 32-bit platforms and wasm32 are zero too.

In C++

This is what the samples do (samples/common/pack_reader.hpp):

#include <vgl/defaults.hpp>
#include <vgl/vgl.hpp>

int run()
{
    vgl::expected<vgl::thread_pool> pool = vgl::thread_pool::create();
    if (!pool)
        return 1;
    vgl::expected<vgl::file_reader> files = vgl::file_reader::create(*pool);
    if (!files)
        return 1;
    // Non-HTTP URIs go to the file reader, which must outlive the HTTP reader.
    vgl::expected<vgl::http_reader> http = vgl::http_reader::create(*files);
    if (!http)
        return 1; // vgl::result::not_supported without VGL_DEFAULTS_HTTP
    const vgl_io_interface http_io = http->io_interface_get();
    vgl::expected<vgl::disk_cache> cache =
        vgl::disk_cache::create("vgl-cache", http_io, 0 /* 1 GiB */, &*pool);
    if (!cache)
        return 1;

    const vgl_task_interface tasks = pool->task_interface_get();
    const vgl_io_interface io = cache->io_interface_get();
    auto info = vgl::init<vgl_context_create_info>();
    info.tasks = &tasks;
    info.io = &io;
    vgl::expected<vgl::context> ctx = vgl::context::create(&info);
    if (!ctx)
        return 1;

    vgl::expected<vgl::pack> gebco = vgl::pack::create(*ctx, "https://packs.example.org/gebco", 0);
    vgl::expected<vgl::pack> detail =
        vgl::pack::create(*ctx, "https://packs.example.org/glo30-norway", 1);
    if (!gebco || !detail)
        return 1;

    // ... frame loop ...
    return 0; // Destructors run in reverse order: packs, context, cache, readers, pool.
}

Declare the objects in this order, so that each is destroyed before the one it reads through. Cache statistics (hits, misses, offline hits, entries, bytes, evictions) come from vgl_disk_cache_statistics_get; vgl_disk_cache_clear empties the cache.

In a browser

In the web build, the HTTP reader fetches packs with the browser's fetch. The page must be cross-origin isolated (it runs threads), so packs on another origin load only if their host sends CORS headers (and, ideally, Cross-Origin-Resource-Policy: cross-origin), as the reference server does. Relative URIs are fetched from the page's own address.

Trying it

The samples take --pack <uri> (repeatable) and --cache <directory> with --cache-budget <MiB>; for example, against packs served locally by the reference server:

vgl-headless --pack http://127.0.0.1:8080/gebco --pack http://127.0.0.1:8080/glo30-sognefjord \
             --cache vgl-cache

Pitfalls

• Every tile downloads a whole shard. The server ignores ranges. Fix the server; vgl-bake check reports it as an error.
• Corrupt-data failures over HTTP but not from disk. Something compresses the pack files (the server, a CDN or an upload with content encoding). Turn compression off for *.vgls and pack.vglm.
• Works natively, fails in the browser. CORS headers or the preflight answer are missing, or the pack is on plain HTTP while the page is on HTTPS.
• Readers see an old pack after an update. The manifest is cached for long somewhere (a CDN rule). Keep it at a minute, or purge it after an update.
• Shards missing after an update. Old shards were deleted before caches moved on to the new manifest.
• .bake uploaded. It is harmless but unneeded; upload only pack.vglm and *.vgls.
• Trailing slash in the pack URI. The runtime appends /pack.vglm; pass the directory without a trailing /.

See also Reference server, Pack format overview, Defaults module and Troubleshooting.