Repository navigation
ChronoLog 3.4.0
Highlights
Durable writes (#730). A keeper used to answer log_event with CL_SUCCESS as soon as the event was in its in-memory queue, 60 to 90 s before it reached the archive, so a keeper crash in that interval lost an event the writer had been told was stored. Now a write made with LogEventOptions{Durability::Durable} returns CL_SUCCESS only after ChronoKeeper has written the event to its write-ahead log (WAL) and fdatasync returned. A restarted keeper reads its WAL back, ships the recovered events to the archive over the usual path, and deletes each record once ChronoGrapher confirms it.
- The client API.
log_event(record, options, event_time)in C++ andlog_event(record, options)in Python, withtimeout_ms(10000) andattempts(3) per durable write. A retry resends the same event, so a keeper that already stored it does not store a second copy. Three new codes say whether anything was stored:CL_ERR_DURABILITY_UNAVAILABLE(-19) andCL_ERR_DURABILITY_FULL(-20) mean nothing was, andCL_ERR_DURABILITY_UNKNOWN(-21) means it may have been, withevent_timeset so the writer can check with a replay. - One sync for many writers. The keeper syncs every durable event queued while the previous sync ran, in one group; with 8 writers a 1 KB durable write took about 480 µs at the median on a local NVMe drive (Debug build), about 15,000 events/s.
- A synced archive. ChronoGrapher syncs each window file, its directories and the archive manifest before it reports the window written, so a power loss on the grapher node no longer loses events the keepers have already freed.
- A restarted keeper takes its stories back. A keeper that registers again gets a pipeline for every story clients still hold, and the deploy scripts can restart one keeper with
--restart-keeper N. - Docs and tests. A new Keeper WAL page, the durable API and error codes for C++ and Python, and the WAL settings. The unit suite grows from 480 to 758 tests, seven new integration scripts cover the WAL, and a nightly workflow runs every integration script and the distributed keeper restart.
Upgrading from 3.3
- The WAL is off by default. Turn it on with the
WALblock inchrono_keeper(enabled,dir), or with--wal/--wal-dir DIR[:DIR...]in the deploy scripts, which pick a local non-OS drive on each keeper host. Put the WAL on a local drive, not NFS or tmpfs; the keeper warns at startup. See the server configuration and the multi-node guide. - Without durable writes nothing changes. Accepted writes, reads and the archive format are unchanged, and
log_event(record)keeps its path and cost. The protocol version stays 3, so 3.3.0 clients connect. A durable write to a 3.3.0 keeper, or to one with the WAL off, gets -19 at once and stores nothing. - Configuration. Only the new
WALblock; no existing key changes. - Stopping and cleaning.
--stopand--cleanreport durable events still pending in a WAL, and--cleanrefuses to remove such a WAL directory without--force.
Behavior changes
- Sealed chunks leave the keeper as soon as they seal. The extraction engine had no progress thread, so with many stories they took up to an hour to reach the grapher.
- The grapher's added syncs raise the mean window write from 0.27 ms to 1.85 ms in a burst of 1,000 stories.
- A grapher or player whose answer to a keeper fails logs it and goes on; before, the error ended the process.
- A client's event index starts at 0; it was an uninitialized value.
deploy_local.shruns the binaries given with-v/-g/-p/-ainstead of the installed ones.
Known issues
- Only the durable write and the player's hot fetch have RPC deadlines. Over
ofi+tcp, an accepted write to a dead keeper blocks about 20 s, and an acquire waits about 17 s on the visor's call to it; the shipped configuration usesofi+sockets. - After a keeper restart, tail reads (
playback) do not return the recovered events; a replay does. - A bad record in the last group of the newest WAL segment is treated as a torn write and zeroed, with the intact records after it; recovery keeps a copy and logs an error.
- A durable retry that reaches the keeper after the grapher confirmed the event is stored again; reads drop the copy.
- No client failover to another keeper while one is down.
- Other open items: #707, #719, #725 (clients), #708, #709 (failed archive writes), #711, #712, #715 (resource use), #716, #717, #718 (deployment).
Full Changelog: v3.3.0...v3.4.0