v0.15.4

Faster bundle imports. A backfill spent nearly all its time sleeping: it paces at half
the lower of the namespace and principal rates (25 items/s at default limits), 40 s per
1,000-item batch, against about half a second of work.

• Under an allowance (§6.6) the importer now paces at the allowance's full rate, since its
  bucket holds up no other writer, and splits batches by its itemsPerBatch/batchSize. It
  finds its own allowance as the core does (the grant's root sub and kid, or the author
  with authentication disabled), counts every draw (dry runs, uploads, copies) so it doesn't
  run into 429s, and from a minute before the allowance's until goes back to the
  namespace's limits. With an allowance of 1,000 items/s, 2,000 documents wait 3.5 s instead
  of 80 s; 145,000 would take minutes instead of over an hour. Without one, pacing is as
  before, less the time each batch took. Progress lines say what the import is paced by.
• Planning reads target heads from /heads pages instead of one GET per document
  (twice in snapshot mode): 0.005–0.014 requests per document instead of 1–2. A small import
  into a large namespace keeps per-document lookups.
• BenchmarkImport (internal/bundle) measures an import's work, requests per document and
  the time it would sleep.

The client's requests use a connection pool of their own, a clone of
http.DefaultTransport, so code that closes the default one's idle connections doesn't
break them.

Full Changelog: v0.15.3...v0.15.4

v0.15.3

Index queries cost what their matches cost (B6). Every index query read the namespace's
documents in order and tested each against its facets, ranges and ?ref=, so a facet
matching 100 of 48.7k documents read all of them, and with sort= sorted them all. A query
is now driven by its most selective filter when that pays: its matches are read from the
filter's index and looked up, and the other filters tested per document. Which filter drives
is decided by a bounded count on the index, and the plan doesn't depend on SQLite's
statistics. On 48.7k documents, one query / 50 at once:

query	before	after
facet[/accountId]=…	109 ms / 6.2 s	9.4 ms / 0.57 s
… &sort=/accountId	237 ms / 17.2 s	9.7 ms / 0.64 s
… &counts=…	602 ms / 29.8 s	10 ms / 1.15 s
… &q=…	5.4 s / 193 s	137 ms / 10.8 s
ref=…	3.9 s / 421 s	10 ms / 0.76 s
a facet with 5,000 values	7.2 s	39 ms
ge[/score]=500&lt[/score]=501	289 ms	3.7 ms

Queries with no selective filter (a plain listing, a facet most documents match) scan as
before. The index replaces refs_q with refs_t, which leads with the namespace; it is
built when the index opens (about a second per 500k references).

Development: CI runs the race detector in jobs of its own; tests run in parallel
(t.Parallel()); and Postgres tests reuse migrated databases from a pool, emptied between
tests, instead of creating one each (internal/pgtest). CI takes about 2 minutes instead of
5.

Full Changelog: v0.15.2...v0.15.3

v0.15.2

Performance, follow-ups from the v0.15.1 review:

• Branch /heads pages looked up the branch's base namespace once per name that reads
  through. A read transaction now reads each namespace row once: a branch page takes 27–33 ms
  instead of 161–164 ms on Postgres (a branch of a branch, 34–38 ms instead of 288–297), and
  20–29 ms instead of 36–38 ms on SQLite.
• The index's -fetch-concurrency now limits fetches across all followed namespaces
  and branches together, not per namespace: -ns a,b,c sent up to 24 GETs at once to the
  core (whose Postgres pool has 16 connections) and now sends at most 8. It must be at least
  1.
• The index reuses its connections to the core: its client keeps an idle connection per
  fetch (new client.WithIdleConns), where http.DefaultTransport kept 2 and redialled the
  rest after every page. Catching up 3 × 2,000 documents opened 8–12 connections instead of
  1,500–2,100, at the same speed.

Full Changelog: v0.15.1...v0.15.2

v0.15.1

Performance (found by profiling with the new OpenTelemetry spans). End to end, against
v0.15.0 on a shared 4-CPU box, mixed reads (head pointers, revisions, logs, /heads) went
from 557 to 1,552 requests/s on SQLite and from 549 to 1,015 on Postgres, with p99 down from
281 to 66 ms and from 248 to 68 ms:

• Namespace log pages resolved each entry's prev with its own query (one per entry, a
  round trip each on Postgres). They now come from the page's own rows: a page takes 18 ms
  instead of 160 ms on Postgres, and 13 ms instead of 48 ms on SQLite.
• /heads on SQLite now batches its lookups as Postgres already did: 48 ms instead of
  218 ms per page.
• The index fetches a page's documents concurrently (-fetch-concurrency, default 8)
  instead of one at a time, applying them in log order as before. On Postgres under steady
  writes its lag fell from about 950 ms (max 2.6 s) to 24 ms, and ?min reads no longer
  time out with 503. The trade-off: it now keeps up, so its reads compete with writers, and
  write throughput with the index and tree following fell by about 23% in that test; lower
  -fetch-concurrency trades freshness back.
• Authenticated reads now use the read cache once the request is authorised (it served
  only public namespaces before), and verified grants are cached by token, keeping only
  grants that verified. Warm authenticated reads take 0.8 ms instead of 1.3 ms, and the
  authenticated workload ran 29% faster. Authorisation still runs on every request, and
  cache-control is unchanged.

Build:

• Smaller release binaries: releases, the Docker image and make build build with
  grpc-go's grpcnotrace tag. It drops gRPC's own request tracing (golang.org/x/net/trace,
  off unless grpc.EnableTracing is set, which nothing here does), and with it
  html/template and text/template, whose reflection disabled the linker's dead-code
  elimination. OTLP export over gRPC and HTTP is unchanged. linux/amd64 is 28.4 MB (32.2 MB
  in v0.15.0), linux/arm64 26.7 MB (30.3 MB), darwin/arm64 27.5 MB (31.2 MB),
  windows/amd64 29.0 MB (32.8 MB). A plain go build/go install works the same, about
  3.8 MB larger.

Full Changelog: v0.15.0...v0.15.1

v0.15.0

Implements spec v0.46, which adopts most of Doors' feedback, fixes the server bugs
Doors reported, and adds OpenTelemetry.

Changes to check before upgrading:

• Keys follow the base both ways: a branch now accepts keys added to its base after
  it was created (recursively through local bases); a kid both have is the base's entry
  only (§C.4).
• Operator grants authorise only creating namespaces (remote branches included) and
  forced purges. A request under one to a namespace that doesn't exist is now 401, not
  404 (a forced purge still gets 404).
• The gestures listing answers any reader, filtered to resources its grant may read
  (§7.4); a ?after naming a resource it can't read is 404.
• Unfreezing a branch whose base already has branchesPerNamespace live branches is
  422 limit; frozen branches no longer count toward the limit (Doors B3).
• CORS: -cors-origin '*' with credentials no longer echoes an arbitrary origin;
  credentialed origins are listed with -cors-credentials-origin.
• Tree database gains an items.title column, and index database a sort.raw
  column; both are added or rebuilt on start.
• GET / answers "spec": "0.46".

New (v0.46):

• schemaReads: { for } in a namespace document: its schema revisions resolve for writes
  in the listed namespaces that pin them, and can be read by revision path by readers of
  documents pinning them, without a grant on the namespace (§6.1).
• POST /edge-grants issues edge grants as Secure, HttpOnly cookies per prefix, which
  the origin verifies itself when no edge is in front (-edge-grant-key, else derived from
  -edge-secret); cookies authorise only GET/HEAD under their prefix (§C.5).
• Catalog: ?min on POST /grants and /read-grants; a title pointer copied into
  listings; inheritPowers in $access (§B.5, §B.11).
• Index: schema documents indexed under their dialect, with their $refs as references;
  self: true on self-references; hits carry sort values and fields= (§A).
• Client: undo stacks count a merged gesture for its source author, and Undo with
  UndoAuthor finds it (§11.2). Bundles look up keys through a branch's bases.

OpenTelemetry (off unless OTEL_* exporters are configured): traces and metrics over
OTLP (gRPC or HTTP), console output, W3C propagation; route-named HTTP server spans for
serve and every service, client spans on outbound calls, engine write spans
(core.WriteResource, core.Batch, …), transaction spans, and metrics for writes,
group-commit sizes, Postgres lock waits and consumer lag. See the README's Observability
section; compose.yaml has an optional Jaeger profile (--profile otel). No measurable
cost when off; the binary grows by about 11 MB (gRPC).

Fixes (reported by Doors):

• The index's ?min= redirect could bounce between two checkpoints during an update;
  redirects now follow the committed checkpoint and only move forward (B1).
• schema_unavailable named the dialect URL instead of the unresolved pin; it now names
  the revision path. A schema document could $ref a revision the writer can't read once
  the validator had cached it; read permission is now checked on the whole closure (B2).
• The standalone tree service left heads and URLs out of listings for grants reading only
  some resources of a content namespace (B4).
• The batch retry lookup read every batch the author had written; it now looks up one
  resource's history (5,000 earlier batches: 24.8 → 0.5 ms per batch on SQLite) (B5).

Full Changelog: v0.14.1...v0.15.0

v0.14.1

Implements spec v0.45, which settles the reference's remaining notes from v0.42 and
v0.44. Nothing changes in behaviour: the text now says what the reference already does.
GET / answers "spec": "0.45".

Full Changelog: v0.14.0...v0.14.1

v0.14.0

Implements spec v0.44: reverse-reference queries in the indexing service (Addendum A).

New:

• The index collects every x-ref reference of typed documents by §6.5's walk, with the
  schema revision each document pins (no x-index needed), into a new refs table.
• ?ref=/r/{ns}/{name} finds the documents that reference a resource in any form;
  …/rev/{id} only those pinned to that revision; …%23{entry} only those naming that
  entry. It combines with the other filters, and hits carry refs: [{ path, ref }].
  Malformed or repeated ref is 400.
• An index database from before v0.44 is rebuilt once at startup, so its references are
  collected.
• GET / answers "spec": "0.44".

Full Changelog: v0.13.0...v0.14.0

v0.13.0

Implements spec v0.43, small fixes from the reference's v0.42 notes.

Changes to check before upgrading:

• An operator key authorises only within its period: grants it signed are refused
  before its from as well as after its until (§C.4). A key configured with
  -operator-key and no history entry is unaffected.
• GET / answers "spec": "0.43".

The other v0.43 changes state what the reference already did: grants in sealed namespaces
are sealed under the epoch of the first entry recording them, repeated query parameters are
400, and a bundle line whose writing namespace is unknown carries neither written nor
grant.

Full Changelog: v0.12.0...v0.13.0

v0.12.0

Implements spec v0.42, which settles the reference's v0.41 notes on signatures.

Changes to check before upgrading:

• Unknown query parameters are 400 bad_input on every core endpoint, before
  authentication, as are repeated parameters and flag values other than 1
  (dry-run=true used to be ignored and the write made).
• The gestures listing pages with ?after=, as §7.4 says; it took ?since=. The Go
  client and the playground follow.
• Author signatures are checked at §6.2 step 2.3, after rate limits, the retry lookup
  and settling the verb: an idempotent retry is answered as first recorded whatever
  signature it carries, even after required was turned on; 403 and 429 come before
  422; a dry run reports 422 signature per item; a write without a usable precondition
  gets 428/400. A batch whose config change is stale fails with that change's 412.
• GET /ns/{ns}/grants/{gid} is offered in sealed namespaces (sealed, pl: { ns, grant },
  stored once) and end-to-end ones (clear, Cache-Control: private), and is 410 once the
  namespace is purged; not_offered is gone. client.Grant opens sealed answers.
• An operator key past its until authorises nothing; it used to be accepted after it.
• GET / answers "spec": "0.42".

Bundles and archives:

• A grant line's ns is the namespace whose entry first recorded the grant; key is
  optional, looked up in the namespace document at that entry, then the operator key history
  at the first naming revision's created, and left out otherwise.
• History lines for revisions written elsewhere (a branch's read-through, a remote branch's
  base) carry written, used in the signing input, so branch bundles verify.
• Sealed and end-to-end exports carry grant lines (ExportOptions.Bearer/Identity for
  sealed grants); a grant the exporter can't fetch is left off the lines that would name it.
• Pruning archives (§8.6) carry authors and grant lines; restore skips them.

Performance:

• Heads pages on Postgres take a few statements, not three per resource. A page now
  reads its resources, their heads at at (one LATERAL join on head_history) and those
  revisions in one statement each per namespace level, and resolves from memory.

Full Changelog: v0.11.1...v0.12.0

v0.11.1

Fixes:

• GET /ns/{ns}/rev/{at}/heads pages in time proportional to the page, not the
  namespace. Each page used to list every name and resolve every head at at, and only
  then page, so listing a namespace cost O(N²): about 14 s for 20 000 documents on SQLite
  and 14 minutes for 50 000 on Postgres. A page now takes its names from the (ns, name)
  index after the cursor, in byte order (on Postgres, a new name COLLATE "C" index),
  and resolves only those. Names with no head at at are skipped by fetching further
  chunks. Reported by an implementer.

Full Changelog: v0.11.0...v0.11.1