Skip to content

v3.0.0 - Lazy decoding, API cleanup, and hardened parsing

Latest

Choose a tag to compare

@pryamcem pryamcem released this 03 Oct 20:34
a7d6560
  • ~5x faster, ~9x less memory reading a log without decoding every field (measured on the public test log, v2 vs v3, DATAFLASH_BENCH_FILE=testdata/testlog.bin go test -bench . -benchmem, averaged over several runs on the same machine) — ~7.6x faster, ~10x less memory on a real-world ~184MB log; the gap grows with log size and with reading fewer fields (see Super fast)
  • Lazy field decoding: messages keep their raw body and decode fields on demand instead of building a full map up front — much faster and far fewer allocations when you only need a few fields (see Reading Fields)
    • Message.Fields is now a method, Message.Fields() (decodes all fields once and caches them)
    • New Message.Get(field) to decode a single field without building a map — prefer Fields() instead of calling Get for every column, which is slower
    • GetScaled / GetScaledFields now go through Get
  • New Parser.ReadInto(msg) to reuse a message and its buffer across reads, avoiding per-message allocation (see Reusing Messages)
  • New Parser.Messages() range-over-func iterator: for msg, err := range parser.Messages() { ... } — ends at io.EOF, no manual loop needed
  • New Parser.Stats() reports how much of the log was skipped or resynced (SkippedBytes, Resyncs, InvalidHeaders, UnknownTypes, Truncated), so you can tell whether a log is damaged
  • New Message.HasTimeUS distinguishes a message with no TimeUS field from one whose TimeUS is genuinely 0; Message.TimeUS is meaningful only when HasTimeUS is true
  • ReadMessage and ReadInto now always end with io.EOF, including when the log is cut off mid-message (previously io.ErrUnexpectedEOF); check Stats().Truncated to tell a truncated log from a clean end
  • SetFilter is now atomic: an invalid name leaves the previous filter and read position unchanged instead of leaving the parser half-updated; a message name shared by more than one type now selects all of them
  • Fixed a panic when a corrupt FMT record declares a message length shorter than the 3-byte header, or a message body shorter than its own schema needs
  • Fixed the parser looping forever on a persistent (non-EOF) read error from the source, in NewParser, ReadMessage and ReadInto
  • Fixed schema discovery losing FMT records after junk bytes: it now resyncs to the next valid header instead of stepping through the data 3 bytes at a time
  • Parser.Close removed — it did nothing since v2; the caller owns and closes the source
  • DecodeMessageBody and the header constants (HEAD1, HEAD2, FMTType, FMTLength, HeaderSize) are no longer exported
  • Minimum Go version lowered to 1.24 (go.mod previously pinned the exact 1.25.5 toolchain)
  • Module path updated to /v3