EventLogReader: add readNew() — incremental (byte-offset) streaming read for growing cluster event logs

[HanSur94]

Problem / motivation

EventLogReader (libs/Concurrency/EventLogReader.m) — the reader for the cluster-mode, append-only, per-tag NDJSON event logs — exposes only whole-file or count-based reads: readAll(), tail(n), readAllWithStats(). Its private read_ (lines 142–187) gates on the file's mtime: on any change it re-parses the entire file (AtomicWriter.readWithRetry → ndjsonDecode over the full text, lines 174–186) and caches the whole array. There is no incremental path — no way to read only the events appended since the last read.

These logs are written append-only by MonitorTag.emitEvent_ → EventLog.append (MonitorTag.m:523), using fopen(..., 'a') (in-place growth, no rename — EventLog.m:160). So the file only grows over a recording.

A live consumer — e.g. the Phase 1033 Companion UI that the reader's own header comment anticipates (EventLogReader.m:29-30, 111) — polling this log for new alarms must today re-parse the whole growing NDJSON file on every tick. For a long recording that is O(whole-file) work per poll, which runs directly against the project constraints:

• "Performance: Detached live-mirrored widgets must not degrade dashboard refresh rate."
• long-recording sensor-analysis workflows.

Proposed feature

Add an incremental, byte-offset streaming read:

r = EventLogReader(logPath);
first = r.readAll();     % or readNew() — establishes the offset at EOF
% ... writers append more events ...
newOnly = r.readNew();   % returns ONLY events appended since the last readNew()/readAll()
n = r.LastNewEventCount; % count from the most recent readNew()

readNew() reads from the reader's persisted byte offset to EOF and returns just the newly-appended events — O(new-bytes) per poll instead of O(whole-file). This is a genuine capability (a caller cannot cheaply reproduce it — it requires the reader's persistent offset state), and the natural live-tail primitive for a growing multi-writer log.

Rough sketch

• Lib/class: libs/Concurrency/EventLogReader.m (single file).
• New state: private lastOffset_ (double, bytes; default 0); public read-only LastNewEventCount.
• New method: events = readNew(obj):
  1. If the file is missing → events = [].
  2. If current file size < lastOffset_ (rotation / replacement) → reset lastOffset_ = 0 and re-emit from the start.
  3. fseek to lastOffset_, read to EOF, parse complete lines only via the existing corrupt-line-tolerant ndjsonDecode; leave any trailing partial line unconsumed (do not advance past it) so it is picked up once the writer finishes it.
  4. Advance lastOffset_ to the byte just after the last complete newline; set LastNewEventCount.
• Header: the V1 magic header (#FASTSENSE_EVENTLOG_V1) only appears in the first segment (offset 0) and is already tolerated by ndjsonDecode (as readAll proves today).
• readAll/tail/readAllWithStats and the existing mtime/full-file cache are untouched; readNew maintains its own independent offset state.

Value

• Live monitoring UIs (Companion, detached mirrors) can tail a cluster event log without re-parsing the whole file each tick — the read side of the same append-only path, at O(new) cost.
• Directly serves the long-recording + refresh-rate constraints.
• Fills a clear read-surface asymmetry: today the reader has whole-file (readAll) and count-tail (tail(n)) but no since-last-read tail.

Constraints check

• Toolbox-free: yes — fopen/fseek/fread/ftell + the existing ndjsonDecode. No toolbox.
• Backward-compatible: yes — new method + new read-only property; readAll/tail/readAllWithStats and their cache are byte-for-byte unchanged. Nothing serialized.
• Pure MATLAB / Octave-safe: yes — all primitives exist in both; EventLog uses in-place append (no rename), so byte offsets stay valid.
• Contracts: no change to Tag / DashboardWidget / DataSource. Confined to the Concurrency lib.

Effort estimate

M — one file (EventLogReader.m): lastOffset_ + LastNewEventCount, the readNew() method (seek / read-to-EOF / parse-complete-lines / advance-offset / reset-on-truncation), plus a test (append 3 → readNew=3; append 2 more → 2; no change → []; trailing partial line held until completed; file replaced/shrunk → offset resets and re-emits; empty/missing file → []).

Product micro-decisions to flag

• Offset-reset-on-truncation semantics: re-emit from 0 vs surface a "rotation detected" flag.
• Trailing partial line: hold-and-retry (proposed) vs skip.
• State independence: readNew keeps its own offset, independent of the existing full-file eventsCache_ (proposed — the two do not interact).

---

AI-proposed via /feature-scout — needs a human product decision before implementation.