evidencestore v0.4.0

evidencestore is an independent witness for logs. A collector next to the log sends every line to a store on another machine, and the store can later prove any of those lines to an auditor, offline and without being trusted: the exact bytes, their place in a hash chain, a signed chain head and RFC 3161 time-stamps from before and after the record. Retention works by destroying a month's key: the bodies become unreadable in every copy, while the chain and the proofs stay valid.

This release changes what the store's artifacts show and how a chain head is made. It reads nothing written by v0.3.1 or earlier: see Compatibility before you upgrade.

New in this release

• A single-record bundle no longer shows other records. To check the collector's promise, a single-record bundle carries every leaf from the head the earlier time-stamp certifies to the record's link. Until now those leaves were whole, so the bundle named the sources, record numbers and receive times of everything the store took in meanwhile. Now only time-stamps are whole and every other leaf is just its hash; each leaf enters the Merkle tree of its link with a flag that says whether it is a time-stamp, so a time-stamp shown by its hash breaks its link's root and cannot be hidden. What still shows is the pace of the whole store: how many leaves each link holds and when it was made (Bundles). The bundle is smaller too: up to about 165 KB with one record a second and a time-stamp every 10 minutes.
• The remote copy is encrypted. Chunks of the evidence log and the chain go to replica.url encrypted with copy.key, a random key that init creates and that must reach keys.backup_dir before anything is copied. Whoever holds the copy sees chunk sizes, when they arrive and the signed heads — how fast the store grows — but no source names, records or receive times. A chunk moved under another name, offset or generation does not open. restore takes the key from keys.backup_dir (or -copy-key FILE) (Copy, restore and the witness).
• A chain head is a checkpoint over a Merkle tree of the links. A head is now the number of links and the root of an RFC 6962 tree over all of them, signed in the C2SP checkpoint format (the signature is also a valid C2SP signed note). TSA time-stamps certify the same number and root. Every bundle carries the frontier of that tree before its first link, at most one hash per bit of the link number, and esverify checks the signed head, the time-stamps and a witness's heads against the roots it computes. Whether one head extends another is shown by a few hashes, not by every link between them. The witness, the copy and evidencestore statement carry checkpoints. The store keeps the tree's hashes in data_dir/tree.hashes, a file derived from the chain and checked against it on every start.
• Collectors are witnesses of the chain. The store's answers to a collector carry its signed head and a proof of a few hashes that this head extends the one the collector accepted last. The collector writes the last head it accepted to checkpoints.jsonl, with the time by its own clock, at most every checkpoint_every (10 minutes by default) and when it stops, and a head that does not extend the last one goes, together with that one, to checkpoints-conflict.jsonl, with an error in its log; records keep being delivered. esverify -heads DIR reads these files as they are. When the store and the sources are run by different people, a rewritten chain shows at the collectors without any third party (Collectors as witnesses).
• Restore checks the head against its name in the copy: a signed head put under another number no longer stops restore with a crash or takes it to an older head.

Compatibility

• This release reads nothing written by v0.3.1 or earlier: the Merkle trees of the links and the heads are built differently. The store refuses an older data_dir and says why, and esverify refuses older bundles by version (single-record bundles are now version 4, window and one-source bundles version 3) and names the release whose esverify reads them. Set the store up as a new installation: a new data_dir, new key directories and a new location for the copy and for the witness, since the new chain starts again from link 1. Keep the old installation's esverify for its bundles.
• replica.url now needs keys.backup_dir: a copy whose key is lost with data_dir cannot be restored. keys.backup_dir receives copy.key next to keyring.dat and sign.key, and init refuses to run where it already holds one of them.
• Collectors start over too, since the new store has a new key and knows none of their numbers: let each deliver what it holds to the old store, then give it an empty spool_dir, the new store's public key as store_key and this release's escollect, which keeps the store's heads as a witness. Without from_start it reads from the end of the log; with it, the whole log goes to the new store again.
• New dependency: golang.org/x/mod (the tree and its proofs, from the Go checksum database).

Known issue

• esverify -heads takes the first head at or after the record's link by number and does not read when the witness received it. Whoever holds both sign.key and the witness credentials can sign a head later and add it to a witness bucket under an unused number, and esverify takes that head instead. Until it reads receive times, check the head it names yourself, as Witness of the chain heads describes; the same holds for v0.2.0 to v0.3.1. A collector's head journal is not open to this: the store cannot put a head there.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

With Go 1.25.4 or newer: go install github.com/Nikolay1994Kaz/evidencestore/cmd/esverify@v0.4.0, likewise cmd/evidencestore and cmd/escollect.

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

• The test gate with the race detector, in CI on Go 1.25.4.
• Mutation testing: 650 injected bugs on this release's code, 1–2 November, each caught by the tests (four only because the tests run out of time). In the first run two survived and one did not compile: one test saw the check it guards only through the tree over the links, another had lost its premise, and one mutation no longer fitted the code; the tests are rewritten and the mutation fixed (P-38 and P-39 in docs/dev/FINDINGS.md, in Russian). The last fix of the collector (P-42, P-43) came after the full run: the five mutations it added or rewrote are caught, and 39 of the collector's other 106 mutations, run again on the final code, are caught; the rest were not run again.
• Simulations on this release's code, no failures: a dishonest store owner, 2,000 scenarios (each of the 632 window bundles that left a record out was exposed by that record's promise); the receive-time clock, 1,000 scenarios; the store with crashes, 2,000 scenarios of 300 steps (37,294 crashes, 6,735 restores from the copy, 1,396 disks that lied about fsync); the collector, 2,000 scenarios in each of four worlds and 150 per world against the real store, in which no collector found a head that did not extend the last one it had accepted (after the last fix of the collector, 500 per world and 150 against the real store again).
• The demo passes. A test runs the real store with a collector and verifies a record's bundle against the collector's head journal, read the way esverify -heads reads it.
• The code of this release went through a review round by AI agents (verification of bundles, the store, the collector and the documentation) and two more passes over the fixes, the last of which found nothing critical; the findings and their fixes are in docs/dev/FINDINGS.md, in Russian.

Details: README, How it was tested.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.

evidencestore v0.3.1

v0.3.0 and one command for trying the store. evidencestore dev-setup -dir DIR prepares a demo store in a new or empty directory: a demo CA with certificates for the store and for the source, the store's config with time-stamps from the demo TSA built into the store, init, and the collector's config with the store's public key pinned in it. It starts nothing: serve, escollect, stamp, proof and esverify stay the commands you type (README, Run it yourself). The demo CA and the built-in TSA are for a try, not for real logs.

Also in this release:

• The copy's S3 credentials come only from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The old, never documented names KVSTORE_S3_ACCESS_KEY and KVSTORE_S3_SECRET_KEY are no longer read: if you set them, set the AWS_ ones instead.
• The README says plainly that the two approvals a key destruction needs are names the chain records, not people the store authenticates (Known limitations).

Nothing else changed: intake, the chain and the proofs, the collector, esverify, the configuration and every on-disk and bundle format are those of v0.3.0, and data and bundles of v0.3.0 work as they are.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

With Go 1.25.4 or newer: go install github.com/Nikolay1994Kaz/evidencestore/cmd/evidencestore@v0.3.1, likewise cmd/escollect and cmd/esverify.

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

• The test gate with the race detector, in CI on Go 1.25.4.
• dev-setup has its own tests: both configs it writes pass the loaders of the store and of the collector, the key pinned in the collector's config is the store's own, a second run changes nothing, and a non-empty directory is left alone. Seven bugs injected into the command by hand were each caught by these tests. A test checks that the old KVSTORE_S3_* variables give no credentials.
• The section Run it yourself was run as written, through dev-setup and through the steps it replaces: both end with VERIFIED.
• Apart from dev-setup and the copy's credentials, the code is that of v0.3.0 and was tested for it: v0.3.0.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.

evidencestore v0.3.0

evidencestore is an independent witness for logs. A collector next to the log sends every line to a store on another machine, and the store can later prove any of those lines to an auditor, offline and without being trusted: the exact bytes, their place in a hash chain, a signed chain head and RFC 3161 time-stamps from before and after the record. Retention works by destroying a month's key: the bodies become unreadable in every copy, while the chain and the proofs stay valid.

New in this release

• Bundles over a window of time. A single-record bundle proves one record and says nothing about the records the auditor was not given. evidencestore proof -window -from FROM -to TO gives every record the store put into its chain between two TSA time-stamps around the window, and esverify checks that none was left out, added or moved; the report lists every source with its record numbers, recorded gaps, silences and lost collectors. evidencestore proof -source S -from FROM -to TO gives the same answer for one source, for an auditor from outside: of the other sources it shows only how many there are and the pace of the whole store. Every record now refers to the one before it from its source, and the source tails the store writes right before each time-stamp fix both ends.
• The promise is checked against time-stamps, not against the store's word. The collector holds a signed promise for every record it delivered. esverify -promise now checks it against the TSA time-stamps alone: no time-stamp taken more than a minute after the record's receive time may lie before the record in the chain. A record the store held back past a window fails this check, and an honest store passes it whatever its clocks did: a time-stamp token enters the chain only after the records received before the TSA answered. For that, a single-record bundle carries every leaf from the head that the earlier time-stamp certifies to the record's link.
• The collector checks the store's promises: a receive time more than a minute later than the store's answer by the collector's own clock, or a deadline more than a minute after the receive time, is logged as an error, and the promise is kept as evidence against the store (promises-ahead.jsonl, promises-deadline.jsonl).
• Collector heartbeat. With nothing to send, the collector reports every heartbeat_every (30 seconds by default) with the last number it issued. A collector that stays quiet longer than the source's expect_contact (2 minutes by default) shows as lost, in the chain and in /metrics (evidencestore_source_contact_lost), and as back when it returns.
• A failed write of the evidence log stops time-stamps until the store restarts, and the restart writes the records not yet in the chain to disk again before they enter it.
• Safer start: the key backup, the roots mirror and a file:// copy or witness directory are no longer created by init or the store (a missing one looks like an unmounted disk; init refuses to run without the first two), and a new sign.key is made only for an empty chain and an empty key backup.

Compatibility

• Single-record bundles are now version 3. They grow with the records taken in since the earlier time-stamp, about 210 bytes per leaf: up to about 250 KB with one record a second and a time-stamp every 10 minutes. esverify v0.2.0 does not read them. This esverify still verifies bundles of versions 1 and 2, but -promise refuses them: ask the store for the record's bundle again, it builds version 3 for any record, older ones included.
• chain.promise_mmd must be at least 55 seconds plus two chain ticks and at most a minute (the demo scripts of earlier releases had 10 seconds), and chain.tick at most 2.5 seconds; a configuration outside these limits is refused at start.
• Upgrade the store first and the collectors soon after: collectors of v0.2.0 send no heartbeat, so with this store their quiet minutes read as a lost collector, and collectors of this release flag every promise whose deadline is more than a minute after its receive time, which a v0.2.0 store with chain.promise_mmd above a minute gives.
• Keep the collectors' clocks within a minute of real time (NTP): a collector whose clock runs more than a minute behind logs an error for every record and fills promises-ahead.jsonl.
• Data written by v0.2.0 stays readable: upgrade by replacing the binaries. There is no way back: once this release has put a record into the chain, v0.2.0 does not start on that data_dir, nor on one restored from a copy this release wrote. Records from v0.2.0 carry no reference to the previous record, so for a window of them the store refuses a one-source bundle and points to -window.
• Records from v0.2.0 are held to a minute after their receive time, not to the deadline in their promise. v0.2.0 did not hold a time-stamp back for the records in flight, so its record that took more than about five seconds to reach the chain across a TSA answer, on a host whose clock ran close to 55 seconds behind the TSA, can fail -promise although the store kept its promise. This esverify also checks the time of the link that holds each time-stamp token it is shown: a chain written by v0.2.0 on a host whose clock ran more than a minute behind the TSA when it took a time-stamp fails that check in the bundles that show that token.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

With Go 1.25.4 or newer: go install github.com/Nikolay1994Kaz/evidencestore/cmd/esverify@v0.3.0, likewise cmd/evidencestore and cmd/escollect.

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

• The test gate with the race detector, in CI on Go 1.25.4.
• Mutation testing: 578 injected bugs on this release's code, 28–29 October, each caught by a test. Six survived the first run, because tests saw what they broke only through the old promise check; tests added the same night catch them (Q-84 in docs/FINDINGS.md).
• Simulations on this release's code, no failures: a dishonest store owner, 2,000 scenarios (each of the 632 window bundles that left a record out was exposed by that record's promise); the receive-time clock, 1,000 scenarios; the store with crashes, 2,000 scenarios of 300 steps (37,307 crashes, 6,727 restores from the copy, 1,395 disks that lied about fsync); the collector, 2,000 scenarios in each of four worlds and 150 per world against the real store.
• Live runs on the release candidate, with test hooks: an upgrade from v0.2.0 with kill -9 during intake (all 632 promises verify), and an evidence log whose fsync fails (no time-stamp until restart, then all 244 promises verify). The demos pass.
• The code of this release went through review rounds by AI agents (two independent reviewers, a tester and a judge) until a round found nothing critical; the findings and their fixes are in docs/FINDINGS.md.

Details: README, How it was tested.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.

evidencestore v0.2.0

evidencestore is an independent witness for logs. A collector next to the log sends every line to a store on another machine, and the store can later prove any of those lines to an auditor, offline and without being trusted: the exact bytes, their place in a hash chain, a signed chain head and RFC 3161 time-stamps from before and after the record. Retention works by destroying a month's key: the bodies become unreadable in every copy, while the chain and the proofs stay valid.

New in this release

• Witness of the chain heads. Whoever holds the store's signing key can sign a second version of the chain. With witness.url (S3 or a directory) the store now sends its signed chain heads, and nothing else, to a witness run by someone that administrator does not control: the auditor, another organization, a cloud bucket with S3 Object Lock. esverify -heads DIR checks a bundle against the heads the auditor took from the witness: a chain rewritten after the witness saw a head, or two versions of the chain signed with the store key, no longer verify. evidencestore proof -to-head N extends a bundle to a witnessed head. The witness has its own S3 credentials (ES_WITNESS_*), and writes to S3 carry Content-MD5, which buckets with Object Lock require.
• Proof bundles no longer grow with the chain. A bundle used to carry every link up to the current head, about 173 bytes for each link written after the record: megabytes after a day of steady intake, gigabytes after a year. It now ends at the head under the time-stamp that covers the record, and the links are read as a stream instead of loading the whole chain file.
• esverify -body FILE writes the record body byte for byte once the bundle verifies, so the original log can be rebuilt from the store.
• Output in English from esverify, the chain errors and the demos.
• go install works: the module path is github.com/Nikolay1994Kaz/evidencestore.
• A new demo, scripts/demo-intruder.sh: an intruder cleans auth.log, and every deleted line comes back from the store, verified offline.
• What stands against the store's own administrator is now stated in the README as it is: the time-stamps, and a witness that administrator does not control.

Compatibility

• Proof bundles keep format version 1. Bundles from v0.1.1 verify with this esverify, and bundles from this store verify with the v0.1.1 esverify (which has no -heads).
• The on-disk formats (evidence log, chain, keyring) did not change. Upgrade by replacing the binaries: the store first, then the collectors.
• The witness section of the config is optional.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

With Go 1.25 or newer: go install github.com/Nikolay1994Kaz/evidencestore/cmd/esverify@v0.2.0, likewise cmd/evidencestore and cmd/escollect.

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

The test gate with the race detector, in CI on Go 1.25.4. Mutation testing: 316 injected bugs ran in full on 12 October (commit 957345c), each caught by a test; every change since came with its own mutations, 22 in all, run against the code they guard and all caught. The witness and the copy were tried end to end on MinIO with Object Lock. Details: README, How it was tested.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.

evidencestore v0.1.1

Same code as v0.1.0, rebuilt. In v0.1.0 eight of the nine binaries reported their version as v0.1.0+dirty: they were built from the tagged commit (their provenance attestation says so), but the build directory sat inside the source tree and made it look modified. The build directory is now ignored by git, and the release script refuses to publish a binary marked as built from a modified tree. Use this release.

evidencestore keeps audit records — actions of AI agents, logs of core banking systems, a copy of the SIEM stream — so that an auditor can check any record offline, without trusting the store: the exact bytes, their place in a hash chain, a signed chain head and RFC 3161 time-stamps from before and after the record. Retention works by destroying a month's key (two approvers, a waiting period): the bodies become unreadable in every copy, while the chain and the proofs stay valid.

In this release

• evidencestore — the store: intake over mutual TLS, an fsync-before-"accepted" evidence log, a hash chain anchored by an external TSA, per-month keys with crypto-erasure, a copy to S3 or a directory with restore, Pipelock flight-recorder verification, Prometheus metrics.
• escollect — the collector: follows a log file or a glob of files through rotations, spools lines on disk, keeps the store's signed promises.
• esverify — the auditor's tool: verifies a proof bundle offline and says what is and is not proven.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

Test gate with the race detector; 316 injected bugs, each caught by a test; simulations — the store with crashes (2,000 scenarios of 300 steps), the receive-time clock (3,000), the collector (over 10,000 scenarios, part of them against the real store); the demo with a live Pipelock v3.6.0; a night on real data, where 104,738 records from the host's own logs and a live Pipelock matched their sources line by line and the store restored from its copy gave the same result. Details: README, How it was tested.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.

evidencestore v0.1.0

Use v0.1.1. Same code; here eight of the nine binaries report their version as v0.1.0+dirty (built from the tagged commit, as the attestation shows, but the build directory made the tree look modified).

First public version of evidencestore, for pilots.

evidencestore keeps audit records — actions of AI agents, logs of core banking systems, a copy of the SIEM stream — so that an auditor can check any record offline, without trusting the store: the exact bytes, their place in a hash chain, a signed chain head and RFC 3161 time-stamps from before and after the record. Retention works by destroying a month's key (two approvers, a waiting period): the bodies become unreadable in every copy, while the chain and the proofs stay valid.

In this release

• evidencestore — the store: intake over mutual TLS, an fsync-before-"accepted" evidence log, a hash chain anchored by an external TSA, per-month keys with crypto-erasure, a copy to S3 or a directory with restore, Pipelock flight-recorder verification, Prometheus metrics.
• escollect — the collector: follows a log file or a glob of files through rotations, spools lines on disk, keeps the store's signed promises.
• esverify — the auditor's tool: verifies a proof bundle offline and says what is and is not proven.

Install

Binaries: evidencestore and escollect for Linux amd64 and arm64, esverify also for macOS (amd64, arm64) and Windows (amd64). Check them before use:

sha256sum -c SHA256SUMS --ignore-missing
gh attestation verify esverify-linux-amd64 --repo Nikolay1994Kaz/evidencestore

Setting up a store and collectors, with systemd unit files: README, Install.

How it was tested

Test gate with the race detector; 316 injected bugs, each caught by a test; simulations — the store with crashes (2,000 scenarios of 300 steps), the receive-time clock (3,000), the collector (over 10,000 scenarios, part of them against the real store); the demo with a live Pipelock v3.6.0; a night on real data, where 104,738 records from the host's own logs and a live Pipelock matched their sources line by line and the store restored from its copy gave the same result. Details: README, How it was tested.

Before you use it

• Until 1.0 the on-disk formats (evidence log, chain, keyring) may change; each release will describe the way over.
• The pilot cryptography is not certified: AES-256-GCM from the Go standard library, root keys in files. The root store is a plug-in point for an HSM.
• Read Known limitations.
• There is no container image yet.