ChronoLog 3.2.0

Highlights

Durable chunk retention. Until now a ChronoKeeper freed a sealed chunk as soon as it had shipped it, so a chunk ChronoGrapher had received but not yet written to the HDF5 archive existed only in the grapher's memory. In 3.2.0 a keeper keeps every sealed chunk until the grapher confirms it written, and a replay reads the events that are not archived yet straight from the keepers. The design, the review guide and the verification are in #703.

• Watermark feedback. ChronoGrapher tracks, per story, how far its archive is written without gaps, gives each delivered keeper chunk a receipt, and reports both to the keepers every second. A keeper frees a chunk only once the archive covers it, its receipt is settled, and players can see the file.
• Replay across archive and keepers. ChronoPlayer splits a replay at the highest point any keeper reports as archived: the archive serves everything before it, the keepers everything after, fetched from all keepers at once. Each event is returned once, and an incomplete answer is reported as CL_ERR_PARTIAL_RESULT instead of looking complete.
• Safer archive files on shared file systems. A window is written under a temporary name and hard-linked into place, so readers on other nodes never open a half-written file, and a window written more than once gets a file of its own instead of replacing the first.
• Clean shutdown and destroy. A keeper stopped with SIGTERM delivers what it holds and waits for the grapher to confirm it. Destroying a story releases its chunks on the keepers and deletes its archive files only after writes in progress finish.
• Docs. A new Durable Chunk Retention page, and a table of the timing rules that keeper, grapher and player settings must keep between them in the server configuration reference.
• Tests. 209 new unit test cases, four integration scripts that deploy real daemons, and two new stages in the distributed pipeline.

Upgrading from 3.1

Clients must be rebuilt. The client protocol version is now 3: the replay response carries a flag for a partial result. A 3.1 client is refused at Connect with CL_ERR_PROTOCOL_VERSION_MISMATCH. Rebuild C++ clients and reinstall the Python wheel from this release.

Configuration. Start from the 3.2.0 template, or apply these changes:

• tail_retention_secs (keeper) is gone. Chunks are freed once the grapher confirms them written, not by age.
• New keeper settings: watermark_resend_timeout_secs (300), archive_visibility_delay_secs (10), shutdown_confirm_timeout_secs (150), retention_cap_mb (4096).
• New grapher setting: watermark_report_interval_secs (1). The grapher's default window is now 30 s with a 60 s acceptance window (was 60 s / 180 s).
• New player settings: archive_scan_interval_secs (5), archive_window_secs (30, must equal the grapher's story_chunk_duration_secs).
• A grapher's extraction chain must include hdf5_extractor; a chain with only csv_extractor or logging_extractor no longer starts.
• The keeper's default extractor is single_endpoint_rdma_extractor to the grapher. dual_endpoint_rdma_extractor still works, but its player copy is no longer read.
• Some of these settings have to agree across components; see "Timing rules across components" in the server configuration reference.

Archive on NFS. Mount the archive directory with lookupcache=positive and a short acdirmin/acdirmax on the player nodes; see "Archive on a Shared File System" in the multi-node deployment guide. deploy_cluster.sh warns when the mount can hide new files.

Behavior changes

• ReplayStory can return CL_ERR_PARTIAL_RESULT (-18) together with the events it did gather, when a keeper did not answer in time or an archive file could not be read. Retry the range to pick up the rest.
• A replay returns events that are not archived yet; it no longer has to wait for the archive.
• A keeper stopped with SIGTERM waits up to shutdown_confirm_timeout_secs for the grapher to confirm its chunks, so a keeper can take that long to exit.
• A window written more than once is archived as numbered files (<chronicle>.<story>.<start>.vlen.<n>.h5) next to the first.

Known issues

• Replay can miss events in rare timing windows: #706 (a story acquired again soon after release can move to another recording group), #713 (a window written late is found only by the player's next directory scan).
• A failed archive write is not fully recovered: #708 (its receipts stay pending), #709 (a keeper stopped shortly afterwards frees the chunk unconfirmed).
• Archive file names: #710 (stories whose names join to the same <chronicle>.<story> share files; a name containing / cannot be archived).
• Resource use and robustness: #707, #711, #712, #714, #715, #719.
• Deployment checks and cleanup: #705, #716, #717, #718.

Full Changelog: v3.1.1...v3.2.0