Repository navigation
First release of the 2.x series, relative to 1.0.4. It breaks source and binary
compatibility: rebuild every binary that links data_tamer, and read docs/MIGRATION.md,
which gives the replacement for each removed API and lists the behaviour changes that
compile without a warning. Readers need the 2.0 parser for files recorded by 2.0; it still
reads 1.x files.
Breaking changes
- The namespace is
DataTamer2(wasDataTamer), the headers are in
data_tamer_v2/(weredata_tamer/) and the standalone parser is
data_tamer_v2/data_tamer_parser.hpp(was
data_tamer_parser/data_tamer_parser.hpp).SerializeMeandDataTamerParser
keep their names, inside an inline namespacev2. DataSinkBaseis replaced byDataSink, whose protected callbacks are
onSchema(),onSnapshot(const SnapshotRef&)and the optionalonStop()and
onStart()(throw to report a failure), owned by aSinkWorkerthat runs the
delivery thread.addChannel(),pushSnapshot(),storeSnapshot(),
stopThread(),stopAcceptingSnapshots(),processQueuedSnapshots()and
startAcceptingSnapshots()are gone: useSinkWorker::stop(),drain()and
start(). To write data_tamer MCAP files from your own data, use
sinks/mcap_encoding.hpp.- Sinks are created with
MCAPSink::create(),ROS2PublisherSink::create(),
DummySink::create()orSinkWorker::create<T>(), which return a
std::shared_ptr<SinkWorkerOf<T>>: aSinkWorkerwhosesink()is theT
(static cast, nodynamic_cast), converting to thestd::shared_ptr<SinkWorker>
thataddDataSink()andaddDefaultSink()take.worker->as<T>()reaches the
sink through a plainSinkWorker(std::bad_castfor the wrong type). Passing
std::make_shared<MCAPSink>(path), as in 1.x, toaddDataSink(),
removeDataSink()oraddDefaultSink()fails with astatic_assertthat names
MCAPSink::create()andSinkWorker::create<T>().
LogChannel::addDataSink()takes the worker and returnsbool(falseif it was
attached already).MCAPSink::finishQueueAndStop()is removed (worker->stop()),
restartRecording()no longer resumes a stopped worker, andDummySinkhas
accessors (latestSnapshot(),schema(hash), ...) instead of public members. Snapshothas nochannel_name(useSchema::channel_name, matched by
schema_hash), andschema_hashisuint64_t.DataSnapshotis removed.takeSnapshot()returns aSnapshotResultinstead ofbool(accepted(result)
replaces thebool).
getActiveFlags()is removed:isEnabled(id)in the application,
Snapshot::active_maskin a sink.LogChannel::writeMutex()and theMutexalias are removed: another thread writes
registered variables insidescopedWrite().LoggedValue: scalars are atomic and useset()andget()only. Structs and
vectors keepgetMutablePtr()andgetConstPtr(), whose guards are not movable and
have nooperator booland nomutex().getLockedPtr()and theauto_enable
parameter ofset()are removed. ALoggedValuecan't be moved or
default-constructed.RegistrationIDis opaque: nofirst_index,fields_countoroperator+=.CustomSerializerT(std::string)isexplicit,CustomTypeName<T>::get()is not
constexpr,Schema::custom_typesandcustom_schemasarestd::map(were
std::unordered_map), andlong doubleis a compile error. Removed:
getPointerType(),SerializeMe::enable_if_same_tand the library's
AddFieldToHash(), replaced bySchemaTextHash()andComputeSchemaHash()(the
parser keeps its own, for version 4 texts).- Parser:
ParseSnapshot()loses its unusedcallback_customparameter and is
[[nodiscard]];ParseSnapshotRecursivemoves toDataTamerParser::detailand
NullCustomCallbackis removed;SnapshotView::schema_hashisuint64_t;
BuilSchemaFromText()is a deprecated alias ofBuildSchemaFromText(). The header
needs a little-endian host and no longer includes<iostream>. sinks/ros2_publisher_sink.hppno longer includes thedata_tamer_msgsheaders.
Behaviour changes
MCAPSinkcontinues in a new numbered file at every reset (600 s by default) instead
of truncating its file, so disk use is unbounded;setCreateNewFileOnReset(false)
truncates as before. Rollover files are namedrun_1.mcap(1.x:run.mcap_1) and
skip names that exist; the path passed tocreate()is still overwritten.SinkWorker::stop()delivers what is queued, then finishes the sink:MCAPSink
closes its file, andstart()continues in the next numbered file. Destroying a
running worker stops it.- A channel's snapshot pool bounds the memory of its sinks: by default as many slots as
fit in 8 MiB, from 64 to 1024, per channel. A sink that falls behind loses snapshots
(pool_exhausted) instead of growing a queue. ROS2PublisherSink: the data topic is reliableKeepLast(100)(wasKeepAll;
ROS2PublisherOptions::data_qos), and theschemastopic is transient-local
KeepLast(1), published bystartLogging().LoggedValue::set()never re-enables a disabled value.set()andget()of a
struct or vector lock the channel's write mutex, so they wait for a snapshot in
progress.- An id kept across
unregister()and a new registration of the same name is stale:
setEnabled()andunregister()throwstd::invalid_argument(1.x acted on the
new registration). - A snapshot taken by a thread that holds the write mutex (
scopedWrite(), a
LoggedValueguard) returnsblocked; in 1.x it deadlocked. - From a sink callback, a control call that would wait for that callback's own worker
throwsstd::logic_errorand changes nothing instead of deadlocking (seeDataSink
indata_sink.hpp). - Value names, custom type names and field names must be non-empty and free of whitespace
and control characters (1.x refused only spaces); a channel name can't be empty or hold
a control character. Two C++ types with one custom type name in a channel make
registerValue()throw. - A channel holds at most 8 sinks, and a registry at most 8 default sinks.
addDefaultSink()also attaches the sink to the channels that exist already. - MCAP messages carry a sequence number, 1, 2, 3, ... per channel and file (was always 1).
The MCAP sinks don't write a snapshot with a negative timestamp; they count it in
errors(). - Wire format: the schema text is version 5 and its hash is FNV-1a 64 of the text, the
same on every platform. Astd::stringis written aschar[], auint32count
and the characters. 1.x readers reject 2.0 files: use PlotJuggler 4 or a 2.0 parser. - The parsers reject a schema text without a
### version:line (releases before
0.7.0).ParseSnapshot()returns false when bytes are left after the last field, and
throws on malformed data.
New
LogChannel::startLogging()freezes the schema, allocates the pool and announces the
schema; the firsttakeSnapshot()ortryTakeSnapshot()with a sink attached
calls it.tryTakeSnapshot(), for real-time threads, never blocks or allocates once
logging started, also when data_tamer is loaded withdlopen(); the call that starts
logging is not real-time (RealtimeSanitizer reports it), so callstartLogging()
before the loop. Without a sink it returnsno_sinks, in real time; if starting
throws,start_failed, and the next call retries. Several threads may take snapshots
of one channel.accepted(SnapshotResult),constexprand real-time safe: true forokand
partial, whatstats().acceptedcounts.tryTakeSnapshot()isnoexcept: a serializer that throws ends it with
SnapshotResult::serialize_failed, counted byserializeFailures();
takeSnapshot()lets the exception through, as in 1.x. The library catches it, so
serializers compiled into binaries built against earlier headers are covered too.LogChannel::scopedWrite()groups writes from another thread into one snapshot.
registerValue()accepts aconst std::atomic<T>*.setEnabled()is lock-free; newtrySetEnabled()(noexcept) and
isEnabled(id).setPoolCapacity()(a count, or a stall tolerance and a period),
setPayloadCapacity()andChannelsRegistry::setChannelDefaults().LogChannel::stats()andSinkWorker::stats(): snapshot attempts and accepted
snapshots, pool, oversize, serialization-failure and lock counters, refusals per sink,
deliveries, errors and queue high water.MCAPRingSink, a flight recorder: it keeps the lastwindowof every channel in
RAM and writes an MCAP file onrequestDump(), which is real-time safe.ROS2PublisherSinktakesROS2PublisherOptions: QoS, aggregation into
data_tamer_msgs/SnapshotBatch(new message) and the schema format.- A YAML rendering of the schema (wire format section 2.1, also version 5):
ToYaml(),
RenderSchema(),SchemaFormat::Yaml. DataSink::onStop()andonStart(),SinkWorker::Delivery::Manual(no thread:
the application callsdrain()).ChannelsRegistry::stopAll()andshutdown()(stopAll()thenclear(), for
the end ofmain(): every sink is finished and no channel is left borrowing your
variables).TypeDefinitionTrait<T>describes a type without reopening its namespace (Eigen,
...).- Real-time checks with Clang 20 or later: the real-time API is declared
[[clang::nonblocking]](DATA_TAMER_NONBLOCKING), so in a function marked
[[clang::nonblocking]]-Wfunction-effectsrejects a call totakeSnapshot(),
a registration or a non-scalarLoggedValue::set(), and RealtimeSanitizer
(-fsanitize=realtime) reports every allocation, lock or system call at run time,
in data_tamer and in custom serializers. On Linux (ELF) data_tamer declares to RTSan
the futex calls the real-time contract allows. A newrtsanpreset and CI job. - New headers:
names.hpp(JoinNames()andIsCanonicalName()build and check
hierarchical names),sinks/mcap_encoding.hpp(the MCAP encoding ofMCAPSink, for
code that writes data_tamer MCAP files itself) andfwd.hpp(forward declarations). - Decoding in C++ in a few lines:
data_tamer_v2/mcap_reader.hppreads an MCAP file
(ForEachValue()decodes each value,ForEachSnapshot()hands over each snapshot),
in log time order (a crashed recording up to its last complete chunk, in file order),
optionally restricted to some topics, with every error a
std::runtime_errornaming the file and the topic, like Python'siter_mcap();
data_tamer_v2/decode.hpphasToSnapshotView(), the parser's view of a
Snapshotin a custom sink. T07 and T08 use them. - Parser:
SchemaRegistry,DataTamerParser::ToSnapshotView()of a ROS 2 message and
ForEachSnapshotInBatch()decode the ROS 2 messages without ROS;
SplitMcapMessage()splits an MCAP message. - Parser:
ParseSnapshot()decodes several times faster and allocates nothing per
value (benchmarks/parser_benchmark.cppmeasures it); its depth limit no longer
depends on the order of the fields. - Text:
std::string, any otherstd::basic_string<char>(std::pmr::string) and
std::string_vieware recorded as the new basic schema typestring(auint32
byte count and the bytes): registered, as members of custom types, in a
std::vectororstd::array, and asLoggedValue<std::string>. Their
registration now also compiles with Clang 18 and older.ParseSnapshot()and
ForEachValue()take an optional string callback that gets astd::string_view
into the payload (without one, text is skipped); the Python decoder returns astr
and the TypeScript decoder and the Foxglove extension a string, decoded as UTF-8 with
invalid bytes replaced by U+FFFD.std::vector<char>andstd::array<char, N>
stay one value per character.BasicType::STRINGis appended to the enum. A custom
type named after a basic type is refused.docs/wire_format/vectors/has a strings
vector. - Python decoder, standard library only:
pip install "data-tamer-parser[mcap]".
parse_snapshot()compiles each schema and active mask intostructformats,
5 to 15 times faster than a value-by-value walk;parse_snapshot_values()returns
names and values as tuples, for column-oriented code. - Foxglove extension in
data_tamer_foxglove/extension/for ROS 2 topics and
rosbag2 recordings: decoded snapshots with header timestamps, schema catalogs,
batch inspection, and packaged.foxerelease artifacts. - TypeScript snapshot decoder in
data_tamer_foxglove/, with schema versions
4 and 5 (line format and YAML), bounded compiled layouts, nested objects and ROS 2 message helpers. - Tutorials
T01toT08, run by ctest;docs/USER_MANUAL.md,
docs/llm_user_manual.md,docs/MIGRATION.mdand the wire format specification
docs/wire_format.md, with golden vectors.
Fixed
long long,unsigned long long,wchar_t,char16_t,char32_tand enums
over them were recorded asother, which no reader decodes.- A
std::stringwas written with a 16-bit length that no reader decoded. - Two C++ types with one custom type name in a channel: the second was serialized with the
first one's serializer. - A registration that threw could leave a custom type in the schema; it now leaves the
channel unchanged. - The schema text and its hash depended on the global C++ locale.
- Concurrent
takeSnapshot()calls on one channel raced on its snapshot buffer. - The parser read past the buffer before checking its size. Both decoders now check bounds
before every read and reject masks shorter than the schema, oversized counts and cyclic
custom types.
Packaging and ABI
- ROS 2: the package is
data_tamer_v2and installs besidedata_tamer_cpp1.x (the
library islibdata_tamer_v2.so.2). Linkdata_tamer_v2::data_tamer, or use
find_package(data_tamer 2)anddata_tamer::data_tamer.mcap_vendoris linked
publicly;-DBUILD_TESTING=OFFbuilds no test.data_tamer_msgskeeps its name. - Without ROS: package
data_tamer, targetdata_tamer::data_tamer, SONAME
libdata_tamer.so.2(1.x had none). Building needs CMake 3.22 and C++20; consumers
need C++17.-Werroris opt-in (DATA_TAMER_WERROR). As a subproject
(add_subdirectory(), FetchContent) data_tamer installs nothing and builds no test,
example or ROS 2 sink. - Outside ROS 2, MCAP is 2.1.3 (1.x bundled 1.3.0). CMake downloads it, pinned by hash
(FETCHCONTENT_SOURCE_DIR_DATA_TAMER_MCAPfor offline builds), unless
DATA_TAMER_USE_SYSTEM_MCAPis on (conan). The library contains the MCAP
implementation and installs its headers: don't defineMCAP_IMPLEMENTATION. The
mcap_libtarget is gone; zstd and lz4 are linked privately. The vendored moodycamel
queue is removed. - ABI: within 2.x a newer library runs binaries built against older 2.x headers.
tests/abi_tests.cpppins the layouts and theabiCI job (libabigail) compares
each change with the last 2.x release. TheDataSinkandCustomSerializervtables
and the MCAP version are frozen; thestats()functions are inline and built from
exported getters, so their structs can grow.ROS2PublisherSinkalso follows the
rclcpp ABI. Header defaults apply to rebuilt consumers only, Pimpl defaults to old
binaries too. SinkWorkerOf<T>has the layout ofSinkWorker(pinned in
tests/abi_tests.cpp), and the factories that return it are inline.- One version number for the library, the ROS packages, conan, the Python and TypeScript
decoders and the Foxglove extension, checked bytools/check_versions.py. Pushing a
tagX.Y.Zcreates the GitHub release, whose notes are this section
(tools/release_notes.py).