v2.0.0

openglacier Core v2.0.0 — antaa palaa, continued

Glacier 6 goes public, Places become searchable, and the Core learns to run a whole organisation.

openglacier Core v2.0.0 carries on the antaa palaa line opened by v1.0: same code name, same promise. The
architecture stays the one you can build on; what changes is how much of it is now in your hands.

The major version moves for one reason: the storage format changes. Every public 1.0.x build stored its
data with Glacier 5. v2.0.0 publishes Glacier 6, the column-segment store the Core has been developed and
tested on, and reads format 6 only. Upgrading is one backup and one restore (see Upgrading from 1.0.x).

---

Highlights

• Glacier 6, a column-segment storage engine, now in the public tree.
• Backups of their own: OGB2, columnar and compressed, with diffs, chains and full history.
• Place search: place.search over values, files and Apps, including the text of PDF and office documents.
• Native data import: CSV, text and Excel without Python, streamed batch by batch.
• OGML without |: stages separated by line breaks or ;, and expressions directly in group and pivot.
• A scheduler instead of rejections: light and heavy pools, a waiting queue, admission by learned memory peak.
• Policies and pools: enrollment by policy token, usage accounting, enforced limits, App offers.
• Durable machine access, mail, public Places on demand.
• Base Apps read from a file, so a distribution can ship its own set.
• New release archives: og-core-full and og-cli-tui, with build provenance attestations.

---

Glacier 6

Glacier is a column-segment store, on-disk format version 6. A store is a directory:

<store>/
  MANIFEST            live segments, deleted rows, field facts, at one generation
  0000000000000005.g6 immutable column segments, one collection each
  wal-00000007.log    commits after the manifest generation
  LOCK                held by the one process that has the store open

• Rows are sorted by identifier and split into row groups of up to 65,536 rows, with one chunk per top-level
  field. Scalar leaves of object fields are stored as their own columns.
• Each stream picks its smallest encoding: frame of reference and bit packing, sorted dictionaries, scaled
  decimals that stay bit-exact, string dictionaries.
• Per-chunk statistics (kinds present, smallest and largest value) feed the planner, which uses them to prove
  expressions total and to skip row groups that leading filters drop.
• Typed blocks decode straight into the query engine's batches: filters, groups, distinct, pivot and Top-N run on
  typed columns and dictionary codes instead of documents.

It is also stricter about what it accepts:

• data without a manifest is refused;
• the store is locked by one process;
• corruption in the middle of the log is refused instead of being silently truncated;
• segment chunks are verified against their checksum on first read;
• a store of another format, or a manifest written by a newer ogd, is reported as such, not as corrupt data:

... is a Glacier store of format 5 (ogd 1.x): this ogd reads format 6 only

The format is documented in src/storage/backend/GLACIER.md.

---

Backups: OGB2, diffs and history

Backups now have a format of their own, independent of the storage files.

• OGB2 writes each collection in blocks of 65,536 documents laid out by field (identifiers and integers as
  deltas, strings through a block dictionary, exact types kept), compressed with zstd. On a 6.2 M-document store,
  a backup went from 1.52 GB to 73.6 MB, and its restore from 54.8 s to 21.7 s. A store restored then backed up
  again gives the same bytes.
• Diffs: backup.create with a parent writes only what changed since the parent. Restoring a diff
  restores its whole chain, which is found and checked before the store is touched. Diffs need the history
  (OGD_HISTORY=on) or the lighter backup journal (OGD_HISTORY=journal), which keeps the commits since the last
  full backup.
• Full backups with history: backup.create {"history": true} writes the state, then the history of every
  collection and the timeline of commit times and authors. ogd restore-backup <backup.ogb> <new-store-directory>
  rebuilds that history offline, with its original generations, times and authors, so as of and history
  answer as they did on the source.
• OGB1 backups written by 1.0.x are still read.

---

Place search

place.search finds what a Place holds by its words (feature search, in the default and full builds):

• the distinct text values of its collections, with their row count;
• its files and folders, by path;
• its App instances and tables.

Accents, case, plurals and acronyms are folded, prefixes match, and a typo is accepted only when no entry holds
the word as typed; scores weigh rare words higher. Scope keys narrow a search: app:, collection:, field:,
folder:. A search reads only what the caller may read (visible Apps, row filters).

With search-documents (default and full), the text of files is searched too: plain text, PDF, spreadsheets
(xlsx, xls, ods) and office documents (docx, pptx, odt, odp), cut into passages of about 1,000 characters. A file
that cannot be read is still found by its name; files over 32 MB are not read.

The index is made of parts, one per source, each read again only when its collection or file changes: after a
first search of about a second on a large Place, the next ones answer in milliseconds.

---

Native data import

data.import no longer needs Python.

• ogd data-worker (feature native-import, included in full) reads CSV, text and Excel files in a
  process of its own and answers the same protocol as the Python worker. When it is compiled, it is the worker;
  OG_DATA_WORKER=python asks for the Python one instead.
• It reproduces the Python worker's fingerprints, separators, headers, column profiles and date rules, and goes
  further: UTF-16, workbooks recognised by their content, named Excel tables, several tables per sheet, merged
  cells and group headers. A doubtful date becomes null instead of stopping the whole import.
• Imports are streamed batch by batch (5,000 rows or about 1 MB each): a file of any size goes through,
  and no step holds it whole. One million rows through a Gateway: 113 s and 1.39 GiB, where it took 9.85 GiB
  before.
• The last step of an import, moving the staged rows into their collection, now stays within the query's memory.

---

OGML

Without |

A query that holds no top-level | may separate its stages with line breaks and ;, and reads as the
same stages:

on sales
where amount > 100
group
  by region
  sum amount as total
end
sort total desc

on sales; where amount > 100; count

| stays exactly as it was: a query holding one reads as before. A line break ends a stage when brackets are
balanced, the line does not end on a continuation, and the next line starts with a stage word or a directive of
its block. One style per query; errors give the line.

Also new

• Expressions with an alias as compact group items, group keys and pivot rows and columns:
  on sales | group region, revenue - cost as margin sums the margin per region.
• Functions: median, percentile, stddev, month_end, days_in, and half_even rounding.
• Values of different kinds sort by kind instead of failing.
• Materialized views follow the views they read, at any depth, and are computed again when one of them is
  redefined.

---

Running under load

• Light and heavy pools with a waiting queue: queries wait for their turn instead of being rejected. Heavy
  queries are admitted by their learned memory peak rather than a fixed count; operators that cannot spill
  wait for released memory, and pivot cells now spill and merge back. On a 1 GB node, every level of the
  concurrent load campaign completes, with twice the throughput and no errors down to 256 MB.
• Result cache of Place reads (OGD_RESULT_CACHE, on by default): rows are kept under the prepared text,
  the Place, the caller's access filters and the change tokens of everything the read touches, so they are never
  stale. Identical reads arriving together wait for one execution. Five simultaneous logins on a demo stack:
  32.2 s → 1.3 s.
• OGD_QUERY_TRACE writes one JSON line per query.execute.
• Node channels relay as soon as they can: an exchange through a Gateway channel takes 74 µs instead of
  27 ms, and 100 MiB move at 1,480 MiB/s instead of 266.

---

Policies and pools

A Core can now serve many organisations under explicit rules (OGD_ENROLLMENT_MODE=policy):

• a pool is a client organisation; a policy belongs to a pool and carries optional limits, per-user
  limits, rights and an App selection;
• identities enroll with a policy token (ogp_…, shown once, stored as a SHA-256);
• usage is computed per Place, owner, policy and pool (usage.get, every OGD_USAGE_INTERVAL seconds);
• limits are enforced: Places, storage (writes that add documents, imports and file writes are refused past
  the limit, while set and delete still pass), devices, public and external sharing;
• Apps: a policy offers some Apps as included and others as add-ons; an instance whose App is no longer
  included turns read-only;
• identities can be deactivated (identity.state.set), which frees their seat.

New operations: pool.*, policy.*, policy.token.*, identity.policy.set, identity.state.set,
usage.get. Without OGD_ENROLLMENT_MODE=policy, nothing changes.

---

Access

• Durable machine access: a program that deposits files gets a token (ogm_…) that outlives any session,
  bound to one App instance of a Place (machine.access.create, .list, .revoke, auth.machine).
• Mail: the mailsend feature (in full) adds mail.send, submitted through a remote SMTP server over
  SMTPS or mandatory STARTTLS.
• Classic accounts: email identifiers, sharing by identifier, and a forgotten-password flow by mail
  (auth.classic.reset.request, auth.classic.reset.complete).
• Public Places on demand: OGD_ALLOW_PUBLIC=false refuses public access at creation and later.
• Place profiles: the Place owner can restrict what some members see (Apps and rows) on top of their role.

---

Apps

• Base Apps are read from a file (OGD_APPS_PATH, by default /usr/share/openglacier/apps/apps.json). When
  the file is missing or invalid, the Core says so in its log and starts with the public Apps built in.
• An App that a new set no longer carries is kept for compatibility: its definition and its instances stay
  and keep working, and it becomes an add-on that an App manager or a policy can install again.
• Apps created with the Builder are private to their creator and maintainers; the built-in Apps form a public
  catalog.
• place.schema answers the declared schema of a Place: its App instances, tables, fields and their meaning.

---

Local AI

• llm.embed returns one unit vector per text.
• llm.generate accepts a caller's GBNF grammar, greedy sampling and a per-request reasoning switch, and can
  keep a Place's prompt prefix from one request to the next.
• agent.app (experimental) turns a request into an App definition in one call, streaming each step.

---

ogcli

• import loads CSV and TSV files from the client, streamed (2 M rows in 24.7 s).
• --table / format table draws results as tables.
• Tab completion of commands, operations, stages, collections, fields and paths, in the REPL and the TUI.
• In the TUI: Ctrl-O or :edit puts the focused job's query in the input, Alt-Enter runs it in place,
  Ctrl-R or :rerun runs it again, ↑/↓ browse the history.

---

Release archives

Next to the 19 default archives, each release now publishes:

Archive	Content	Platforms
og-core-full-<version>-<platform>	ogd and ogcli with every server capability (native import, search, files, events, LLM, Agent, mail, Fabric)	linux-x86_64-gnu, linux-x86_64-v3-gnu, linux-aarch64-gnu
og-cli-tui-<version>-<platform>	ogcli with its terminal interface	linux-x86_64-gnu, linux-aarch64-gnu

The full builds run on Ubuntu 22.04 runners and need glibc 2.35 or newer (Debian 12, Ubuntu 22.04 and later).
Every archive has its .sha256 and a GitHub build provenance attestation:

gh attestation verify og-core-full-2.0.0-linux-x86_64-gnu.tar.gz --repo openglacier/core

The openglacier-core crate on crates.io ships its sources and the public base Apps only.

---

Security

Dependencies updated for RUSTSEC-2026-0367, -0368 and -0369 (zip 9), RUSTSEC-2026-0187 (lopdf 0.42 through
pdf-extract 0.12), RUSTSEC-2026-0382 (rustls-webpki 0.103.16) and RUSTSEC-2026-0381 (webpki-roots 1.0.11).

---

Upgrading from 1.0.x

v2.0.0 does not open a 1.0.x store: it refuses it with the message above and leaves it untouched. Move your
data through a backup:

# with ogd 1.0.x, running on the old store
ogcli '.backup.create {"name": "before-2.0"}'

# with ogd 2.0.0, on a new store directory and the same OGD_BACKUP_PATH
ogcli '.backup.restore {"name": "before-2.0", "replace": true}'

This path was tested with the published 1.0.5 binary: 49,715 documents (generated, updated and deleted rows,
unicode, floats, integers up to 2^53 − 1, nested objects, system collections) came back character for character
and stayed across restarts.

Also worth knowing:

• When native-import is compiled, the native data worker is used; set OG_DATA_WORKER=python to keep the
  Python one.
• Point OGD_APPS_PATH at your Apps file, or let the Core start with the public Apps built in.

---

Full Changelog: v1.0.5...v2.0.0

Small core, long life.