Skip to content

Defaults Module

Jump to bottom
Jeremiah van Oosten edited this page Oct 18, 2026 · 1 revision

Defaults module

The defaults module (vgl::defaults) is an optional, separately linked library with ready-made implementations of vgl's host interfaces: a thread pool for the task interface, a file reader and an HTTP reader for the I/O interface, and a disk cache that sits in front of the HTTP reader. With it, a host needs no callbacks of its own to load packs from disk or a web server. This page covers every function and create-info field, how the pieces chain together, the order in which to create and destroy them, and when to write your own implementation instead.

The API is in include/vgl/defaults.h, with the C++ wrapper in include/vgl/defaults.hpp; the sources are in src/defaults. The design is in sections 7.5 and 8.6 of the design document.

Why it is a separate library

The core library follows strict rules: it creates no threads, opens no files and connects to no servers. The defaults module does all three, so it lives in its own target, vgl_defaults (alias and exported name vgl::defaults, file name vgl-defaults), which uses vgl only through the public C API. A host that brings its own job system and I/O can leave it out entirely. The samples use it.

CMake option Default Effect
VGL_BUILD_DEFAULTS ON Builds vgl::defaults.
VGL_DEFAULTS_HTTP OFF Gives it an HTTP reader on libcurl (on the browser's fetch under Emscripten). Without it, vgl_http_reader_create returns VGL_RESULT_NOT_SUPPORTED, so the same host code builds either way.
VGL_FETCH_DEPENDENCIES ON With VGL_DEFAULTS_HTTP, downloads and builds curl 8.11.1 (HTTP and HTTPS only, on Schannel, Secure Transport or OpenSSL) when no installed libcurl is found. 
find_package(vgl REQUIRED)
target_link_libraries(my_app PRIVATE vgl::vgl vgl::defaults)

A pkg-config file, vgl-defaults.pc, is installed as well. See Building vgl and Installing vgl for the rest of the options, and Web builds for Emscripten.

When vgl is a shared library, so is the defaults module; its functions are marked VGL_DEFAULTS_API.

Quick start

The common case: decode on a thread pool, read files on the same pool.

#include <vgl/defaults.h>
#include <vgl/vgl.h>

vgl_thread_pool* pool = NULL;
vgl_file_reader* reader = NULL;
vgl_file_reader_create_info reader_info = {.struct_size = sizeof(reader_info)};
vgl_task_interface tasks;
vgl_io_interface io;
vgl_context_create_info info = {.struct_size = sizeof(info)};
vgl_context* context = NULL;

if (vgl_thread_pool_create(NULL, &pool) != VGL_RESULT_SUCCESS)
{
    return 1;
}
reader_info.pool = pool;
if (vgl_file_reader_create(&reader_info, &reader) != VGL_RESULT_SUCCESS)
{
    vgl_thread_pool_destroy(pool);
    return 1;
}
vgl_thread_pool_task_interface_get(pool, &tasks);
vgl_file_reader_io_interface_get(reader, &io);

info.tasks = &tasks;
info.io = &io;
if (vgl_context_create(&info, &context) != VGL_RESULT_SUCCESS)
{
    vgl_file_reader_destroy(reader);
    vgl_thread_pool_destroy(pool);
    return 1;
}

/* ... add packs, update and draw ... */

vgl_context_destroy(context);   /* first */
vgl_file_reader_destroy(reader);
vgl_thread_pool_destroy(pool);  /* last */

The same in C++, where the RAII classes destroy themselves (declare them so that they are destroyed in the right order, or reset them explicitly):

#include <vgl/defaults.hpp>
#include <vgl/vgl.hpp>

vgl::expected<vgl::thread_pool> pool = vgl::thread_pool::create();
if (!pool)
{
    return 1;
}
vgl::expected<vgl::file_reader> files = vgl::file_reader::create(*pool);
if (!files)
{
    return 1;
}
const vgl_task_interface tasks = pool->task_interface_get();
const vgl_io_interface io = files->io_interface_get();

auto info = vgl::init<vgl_context_create_info>();
info.tasks = &tasks;
info.io = &io;
vgl::expected<vgl::context> ctx = vgl::context::create(&info);
if (!ctx)
{
    return 1;
}

// ... packs, updates, frames ...

ctx->reset();   // the context first,
files->reset(); // then the reader,
pool->reset();  // then the pool

Thread pool

A fixed set of worker threads that run tasks from one queue in the order they were submitted. It implements the task interface.

typedef struct vgl_thread_pool_create_info
{
    uint32_t struct_size;
    uint32_t thread_count;
    const vgl_allocator_interface* allocator;
    VGL_POINTER_PADDING(allocator_padding)
} vgl_thread_pool_create_info;
Field Meaning
struct_size sizeof(vgl_thread_pool_create_info).
thread_count Number of threads. 0 means one less than the hardware threads, and at least one.
allocator Optional. Allocator for the pool and its queue; must be thread-safe. Null uses the C heap.
Function What it does
vgl_thread_pool_create(info, &pool) Creates the pool and starts its threads. info may be null for all defaults. Returns VGL_RESULT_OUT_OF_MEMORY also when a thread cannot be started.
vgl_thread_pool_task_interface_get(pool, &tasks) Writes a vgl_task_interface that submits to the pool, for vgl_context_create_info.tasks. It has no destroy callback, so the pool must outlive every context that uses it.
vgl_thread_pool_destroy(pool) Runs the tasks still queued, then stops the threads and frees the pool. Every accepted task runs exactly once. Externally synchronized, and must not be called from one of the pool's own tasks. Null is allowed.

C++: vgl::thread_pool::create(const vgl_thread_pool_create_info* = nullptr), task_interface_get(), get(), release(), reset().

The threads are created with the platform's own functions (CreateThread, pthread_create) rather than std::thread, so a failure to start one is reported as a result code instead of an exception. One pool can serve several contexts, the file reader and the disk cache at once. If you already have a job system, adapt it instead of adding a second set of threads (see Host interfaces).

File reader

Reads pack files from the local filesystem for the I/O interface.

typedef struct vgl_file_reader_create_info
{
    uint32_t struct_size;
    uint32_t reserved;
    vgl_thread_pool* pool;
    VGL_POINTER_PADDING(pool_padding)
    const vgl_allocator_interface* allocator;
    VGL_POINTER_PADDING(allocator_padding)
} vgl_file_reader_create_info;
Field Meaning
struct_size sizeof(vgl_file_reader_create_info).
pool Optional. Reads on the pool's threads. Null reads synchronously inside vgl_io_interface.read, on the thread that calls into the context (inside vgl_context_update, for example).
allocator Optional. Allocator for the reader and its reads; must be thread-safe with a pool.
Function What it does
vgl_file_reader_create(info, &reader) Creates the reader. info may be null: reading without a pool.
vgl_file_reader_io_interface_get(reader, &io) Writes a vgl_io_interface for vgl_context_create_info.io, with a cancel callback and no destroy callback: the reader must outlive every context that uses it.
vgl_file_reader_destroy(reader) Waits for the reads in progress, then frees the reader. Its pool must still exist. Externally synchronized; not from one of the pool's tasks.

C++: vgl::file_reader::create(const vgl_file_reader_create_info* = nullptr), vgl::file_reader::create(vgl::thread_pool&), io_interface_get().

URIs are UTF-8 paths, absolute or relative to the working directory, with an optional file:// prefix (on Windows file:///C:/data/gebco works too). They are not percent-decoded. On Windows, paths are converted to UTF-16 and opened with the wide API, so non-ASCII paths work.

Behaviour:

  • Each read opens the file, reads the range and closes it again: the reader keeps no file handles open between reads.
  • A missing file completes with VGL_RESULT_NOT_FOUND; any other failure with VGL_RESULT_IO_ERROR.
  • A range past the end of the file completes successfully with the bytes up to the end, or none.
  • With a pool, a read cancelled before it starts completes with VGL_RESULT_CANCELLED; one already running finishes normally. If the pool declines a read (for lack of memory), the reader reads synchronously instead.

HTTP reader

Reads packs from a web server with HTTP range requests, so that any server, CDN or object store that serves range requests can hold packs (see Hosting packs). It needs VGL_DEFAULTS_HTTP. URIs that start with http:// or https:// (case-insensitively) are read over HTTP; all others go to a fallback interface, typically a file reader, so that one interface reads both.

typedef struct vgl_http_reader_create_info
{
    uint32_t struct_size;
    uint32_t max_connection_count;
    const vgl_allocator_interface* allocator;
    VGL_POINTER_PADDING(allocator_padding)
    const vgl_io_interface* fallback;
    VGL_POINTER_PADDING(fallback_padding)
    double timeout;
    const char* user_agent;
    VGL_POINTER_PADDING(user_agent_padding)
} vgl_http_reader_create_info;
Field Default Meaning
struct_size sizeof(vgl_http_reader_create_info).
max_connection_count 0 selects 8 Connections to each server at most. Ignored in a browser.
allocator C heap Optional. Allocator for the reader and its reads; must be thread-safe. libcurl's own allocations do not go through it.
fallback none Optional. Reads the URIs that are not HTTP, such as a file reader's interface or a host's own. The table is copied; the object behind its user_data must outlive the reader. Without it, those URIs complete with VGL_RESULT_NOT_FOUND, or in a browser are fetched relative to the page.
timeout 0 selects 30 Seconds a read may take before it fails with VGL_RESULT_IO_ERROR. Must be finite and not negative.
user_agent "vgl/<version>" Optional. The User-Agent header, copied. Ignored in a browser.

There is no option for extra request headers (such as an authorization token) or for HTTP/2. If you need either, implement the I/O interface on your own network layer, or wrap the reader (see Chaining I/O interfaces).

Function What it does
vgl_http_reader_create(info, &reader) Creates the reader and starts its thread. info may be null for all defaults. Returns VGL_RESULT_NOT_SUPPORTED when vgl was built without VGL_DEFAULTS_HTTP, VGL_RESULT_INVALID_ARGUMENT for a bad timeout or a fallback without read or with an unknown version, and VGL_RESULT_IO_ERROR when libcurl cannot be set up.
vgl_http_reader_io_interface_get(reader, &io) Writes a vgl_io_interface with a cancel callback and no destroy callback: the reader must outlive every context that uses it.
vgl_http_reader_destroy(reader) Cancels the reads in progress (they complete with VGL_RESULT_CANCELLED), stops the reader's thread and frees it. Externally synchronized.

C++: vgl::http_reader::create(const vgl_http_reader_create_info* = nullptr), vgl::http_reader::create(const vgl_io_interface& fallback), vgl::http_reader::create(const vgl::file_reader&), io_interface_get().

Native behaviour (libcurl):

  • The reader runs one thread of its own on libcurl's multi interface, which overlaps reads over up to max_connection_count connections per server. Reads may arrive from any thread; they complete from the reader's thread.
  • Redirects are followed (up to 8).
  • Status 200 or 206 delivers the data. 404 or 410 completes with VGL_RESULT_NOT_FOUND. 416 (a range past the end of the file) completes successfully with no bytes, as the file reader does. Any other status, a connection failure or a timeout completes with VGL_RESULT_IO_ERROR.
  • A server that ignores range requests and sends the whole file still works: the reader keeps only the range. It is slow for large shards, so configure the server properly (see Hosting packs and Reference server).

In a browser (Emscripten): with VGL_DEFAULTS_HTTP in an Emscripten build, the reader is built on the browser's fetch instead, with no library and no thread of its own. Fetches start and finish on the main thread, between the page's frames, and reads that arrive on other threads are passed to it. Reads that fail without an HTTP answer are tried again a few times (up to 4 attempts, with a growing wait), which covers browsers whose cache refuses a range of a file it is still storing. Without a fallback, URIs that are not HTTP are fetched too, relative to the page's address. The browser chooses the connections and the User-Agent header. Hosts link Emscripten's fetch support (-sFETCH=1), which the target passes on. See Web builds.

Files and HTTP together

vgl_file_reader* files = NULL;            /* on the pool, as in the quick start */
vgl_io_interface file_io;
vgl_http_reader_create_info http_info = {.struct_size = sizeof(http_info)};
vgl_http_reader* http = NULL;
vgl_io_interface io;
vgl_result r;

vgl_file_reader_io_interface_get(files, &file_io);
http_info.fallback = &file_io;          /* local paths go to the file reader */
http_info.timeout = 20.0;
http_info.user_agent = "my-app/2.1 (vgl)";
r = vgl_http_reader_create(&http_info, &http);
if (r == VGL_RESULT_NOT_SUPPORTED)
{
    /* Built without VGL_DEFAULTS_HTTP: read local files only. */
    io = file_io;
}
else if (r == VGL_RESULT_SUCCESS)
{
    vgl_http_reader_io_interface_get(http, &io);
}
else
{
    /* handle the error */
}
info.io = &io;
vgl::expected<vgl::http_reader> http = vgl::http_reader::create(*files); // files: vgl::file_reader
const vgl_io_interface io = http ? http->io_interface_get() : files->io_interface_get();

Packs can then be named either way:

pack_info.uri = "https://data.example.org/packs/gebco";  /* over HTTP */
pack_info.uri = "packs/glo30-norway";                    /* from disk, through the fallback */

Disk cache

Keeps what another I/O interface reads from web servers in a local directory, so that areas seen once load again without the network, and offline. It wraps an interface: put it between a context and an HTTP reader, or your own network layer. Only http:// and https:// URIs are cached; others go straight to the wrapped interface.

typedef struct vgl_disk_cache_create_info
{
    uint32_t struct_size;
    uint32_t reserved;
    const char* directory;
    VGL_POINTER_PADDING(directory_padding)
    const vgl_io_interface* source;
    VGL_POINTER_PADDING(source_padding)
    uint64_t budget;
    vgl_thread_pool* pool;
    VGL_POINTER_PADDING(pool_padding)
    const vgl_allocator_interface* allocator;
    VGL_POINTER_PADDING(allocator_padding)
} vgl_disk_cache_create_info;
Field Default Meaning
struct_size sizeof(vgl_disk_cache_create_info).
directory required The cache directory, UTF-8; created if missing.
source required The interface whose reads are cached, usually the HTTP reader's. The table is copied; the object behind its user_data must stay valid until the cache is destroyed and its reads have completed. With a pool, the cache may call its read from the pool's threads (when an entry on the disk cannot be read), so it must accept reads from any thread; the HTTP reader does.
budget 0 selects 1 GiB Bytes the entries may take on disk.
pool none Optional. Reads and writes the disk on the pool's threads. Null does it inside vgl_io_interface.read and on the thread that completes the source's reads.
allocator C heap Optional. Allocator for the cache and its reads; must be thread-safe.
Function What it does
vgl_disk_cache_create(info, &cache) Creates the cache and indexes what the directory already holds. Returns VGL_RESULT_IO_ERROR when the directory cannot be created or read, VGL_RESULT_INVALID_ARGUMENT without a directory or source (or a source without read), VGL_RESULT_VERSION_MISMATCH for a source or allocator of unknown version.
vgl_disk_cache_io_interface_get(cache, &io) Writes a vgl_io_interface for vgl_context_create_info.io, with no destroy callback: the cache must outlive every context that uses it.
vgl_disk_cache_statistics_get(cache, &statistics) Writes a vgl_disk_cache_statistics; set its struct_size (at least 8) first. Free-threaded.
vgl_disk_cache_clear(cache) Removes every entry. Reads in progress complete and may store what they read. Returns VGL_RESULT_IO_ERROR if some entries could not be removed (the others were). Free-threaded.
vgl_disk_cache_destroy(cache) Destroys the cache after the disk work in progress, so that what was read is stored. Reads of the source still in progress complete later without storing anything; the source and the pool must outlive them. Externally synchronized.

C++: vgl::disk_cache::create(const vgl_disk_cache_create_info&), vgl::disk_cache::create(const char* directory, const vgl_io_interface& source, std::uint64_t budget = 0, vgl::thread_pool* pool = nullptr), io_interface_get(), statistics_get(), clear().

How it decides what to serve

What Policy
Shards (*.vgls) Their names contain a digest of their content (pack format, section 5.2), so a cached shard is never stale. A hit is read from the disk; a miss goes to the source and is stored.
Manifests (pack.vglm) A manifest changes in place when a pack is rebaked, so it is always read from the source first, and stored. Only when the source fails (other than with VGL_RESULT_NOT_FOUND) is the stored copy served instead: that is the offline case. A manifest the server says is missing is missing, not offline.
Non-HTTP URIs Passed straight to the source, never cached.

So a host that has seen an area once keeps working there offline: the manifest comes from the disk when the server cannot be reached, and every shard range read before is on disk.

Entries, eviction and robustness

  • Each read is one entry, keyed by its URI and byte range, stored as a file <directory>/<hh>/<hash>.vglc. The file holds its key, so a hash collision reads as a miss, and a checksum, so a damaged entry reads as a miss too and is fetched again.
  • Entries are written under a temporary name and renamed, so a cache left by a crash holds only whole entries; files left half-written are removed at the next start.
  • The index of entries is built from the directory when the cache is created and kept in memory. File modification times carry the last use across runs.
  • When the entries exceed the budget, the least recently used go, until the rest fit in nine tenths of the budget. Files are removed outside the cache's lock, so the context's thread never waits for the disk.
  • Cancelling a read completes it with VGL_RESULT_CANCELLED at once, but a download in progress goes on and is stored. During a fast flight, tiles that were requested and then left behind are cached anyway, for the way back.
  • One process at a time may use a directory. Give each running instance its own directory.

Statistics

typedef struct vgl_disk_cache_statistics
{
    uint32_t struct_size;
    uint32_t reserved;
    uint64_t hits;          /* reads served from the disk */
    uint64_t misses;        /* reads of shards that went to the source */
    uint64_t offline_hits;  /* manifests served from the disk because the source failed */
    uint64_t entries;       /* entries on disk */
    uint64_t bytes;         /* bytes the entries take on disk */
    uint64_t budget;        /* bytes they may take */
    uint64_t evictions;     /* entries removed to keep to the budget */
} vgl_disk_cache_statistics;

A rising offline_hits count means the server is unreachable; the OpenGL sample shows these numbers in its panel.

Chaining a cache in front of HTTP

vgl_thread_pool* pool = NULL;
vgl_file_reader* files = NULL;
vgl_http_reader* http = NULL;
vgl_disk_cache* cache = NULL;
vgl_file_reader_create_info file_info = {.struct_size = sizeof(file_info)};
vgl_http_reader_create_info http_info = {.struct_size = sizeof(http_info)};
vgl_disk_cache_create_info cache_info = {.struct_size = sizeof(cache_info)};
vgl_io_interface file_io, http_io, io;
vgl_task_interface tasks;

vgl_thread_pool_create(NULL, &pool);                 /* check every result in real code */
file_info.pool = pool;
vgl_file_reader_create(&file_info, &files);
vgl_file_reader_io_interface_get(files, &file_io);

http_info.fallback = &file_io;
vgl_http_reader_create(&http_info, &http);
vgl_http_reader_io_interface_get(http, &http_io);

cache_info.directory = "cache/vgl";                  /* an application data directory */
cache_info.source = &http_io;
cache_info.budget = 4ull * 1024 * 1024 * 1024;       /* 4 GiB */
cache_info.pool = pool;                              /* disk work off the main thread */
vgl_disk_cache_create(&cache_info, &cache);
vgl_disk_cache_io_interface_get(cache, &io);

vgl_thread_pool_task_interface_get(pool, &tasks);
info.io = &io;          /* context -> cache -> HTTP reader -> file reader */
info.tasks = &tasks;

The C++ samples do exactly this in samples/common/pack_reader.hpp, enabled with --cache <directory> and --cache-budget <MiB>:

vgl::expected<vgl::file_reader> files = vgl::file_reader::create(pool);
vgl::expected<vgl::http_reader> http = vgl::http_reader::create(*files);
vgl::expected<vgl::disk_cache> cache =
    vgl::disk_cache::create("cache/vgl", http->io_interface_get(), 0 /* 1 GiB */, &pool);
const vgl_io_interface io = cache->io_interface_get();

In a browser, the browser's own HTTP cache plays the part of the disk cache.

Teardown order

Each object must outlive everything that reads through it. Create from the bottom up and destroy from the top down:

  1. Packs (optional: the context destroys any that are left, with a warning) and the context.
  2. The disk cache.
  3. The HTTP reader.
  4. The file reader (its pool must still exist).
  5. The thread pool, last: the file reader and the disk cache run work on it.

None of the defaults' interfaces has a destroy callback, so the context never tells them it is gone: destroying them while a context still uses them is a use-after-free. In C++, declare the RAII objects in creation order (pool, file reader, HTTP reader, cache, context) so that their destructors run in reverse, or call reset() in the order above, as the samples do.

Allocators and threads

  • Each defaults object takes its own optional allocator (allocator in its create info), separate from the context's. They are not shared automatically: pass the same table to each if you want all memory to reach one allocator. With a pool or the HTTP reader, the allocator must be thread-safe.
  • The thread pool's threads, the HTTP reader's thread and the work they run are the only threads involved. Completions from the file reader (with a pool), the HTTP reader and the disk cache arrive on those threads, so the context's allocator and log callback must be thread-safe too, which they are if you leave them at their defaults.

When to use the defaults, and when to write your own

Situation Recommendation
Tools, tests, samples, a first integration Use the defaults: thread pool plus file reader.
Packs on a web server, desktop host The HTTP reader with a file reader as fallback, and the disk cache in front if the host should work offline or save bandwidth.
Browser (Emscripten) The HTTP reader, which uses fetch there; no disk cache needed.
An engine with its own job system Write a task adapter instead of the thread pool, so that vgl's decoding shares your workers and priorities.
Data in an archive, an Android APK or an iOS bundle Write an I/O reader for it (the file reader reads only the filesystem).
Authenticated or signed URLs, custom headers, your own HTTP stack Implement the I/O interface on your network layer. The disk cache can still sit in front of it, since it wraps any interface.
Retries, logging, decryption Wrap any of these interfaces with a chained interface.

Mixing is fine: for example your own task adapter with the defaults' HTTP reader and disk cache, or the defaults' thread pool with your own archive reader.

See also

vgl

Getting vgl

Using the library

Rendering

Data

Reference

Specifications

Clone this wiki locally