Skip to content

2.0.0

Latest

Choose a tag to compare

@github-actions github-actions released this 24 Oct 15:20

First release of the 2.x series, relative to 1.0.4. It breaks source and binary
compatibility: rebuild every binary that links data_tamer, and read docs/MIGRATION.md,
which gives the replacement for each removed API and lists the behaviour changes that
compile without a warning. Readers need the 2.0 parser for files recorded by 2.0; it still
reads 1.x files.

Breaking changes

  • The namespace is DataTamer2 (was DataTamer), the headers are in
    data_tamer_v2/ (were data_tamer/) and the standalone parser is
    data_tamer_v2/data_tamer_parser.hpp (was
    data_tamer_parser/data_tamer_parser.hpp). SerializeMe and DataTamerParser
    keep their names, inside an inline namespace v2.
  • DataSinkBase is replaced by DataSink, whose protected callbacks are
    onSchema(), onSnapshot(const SnapshotRef&) and the optional onStop() and
    onStart() (throw to report a failure), owned by a SinkWorker that runs the
    delivery thread. addChannel(), pushSnapshot(), storeSnapshot(),
    stopThread(), stopAcceptingSnapshots(), processQueuedSnapshots() and
    startAcceptingSnapshots() are gone: use SinkWorker::stop(), drain() and
    start(). To write data_tamer MCAP files from your own data, use
    sinks/mcap_encoding.hpp.
  • Sinks are created with MCAPSink::create(), ROS2PublisherSink::create(),
    DummySink::create() or SinkWorker::create<T>(), which return a
    std::shared_ptr<SinkWorkerOf<T>>: a SinkWorker whose sink() is the T
    (static cast, no dynamic_cast), converting to the std::shared_ptr<SinkWorker>
    that addDataSink() and addDefaultSink() take. worker->as<T>() reaches the
    sink through a plain SinkWorker (std::bad_cast for the wrong type). Passing
    std::make_shared<MCAPSink>(path), as in 1.x, to addDataSink(),
    removeDataSink() or addDefaultSink() fails with a static_assert that names
    MCAPSink::create() and SinkWorker::create<T>().
    LogChannel::addDataSink() takes the worker and returns bool (false if it was
    attached already). MCAPSink::finishQueueAndStop() is removed (worker->stop()),
    restartRecording() no longer resumes a stopped worker, and DummySink has
    accessors (latestSnapshot(), schema(hash), ...) instead of public members.
  • Snapshot has no channel_name (use Schema::channel_name, matched by
    schema_hash), and schema_hash is uint64_t. DataSnapshot is removed.
  • takeSnapshot() returns a SnapshotResult instead of bool (accepted(result)
    replaces the bool).
    getActiveFlags() is removed: isEnabled(id) in the application,
    Snapshot::active_mask in a sink.
  • LogChannel::writeMutex() and the Mutex alias are removed: another thread writes
    registered variables inside scopedWrite().
  • LoggedValue: scalars are atomic and use set() and get() only. Structs and
    vectors keep getMutablePtr() and getConstPtr(), whose guards are not movable and
    have no operator bool and no mutex(). getLockedPtr() and the auto_enable
    parameter of set() are removed. A LoggedValue can't be moved or
    default-constructed.
  • RegistrationID is opaque: no first_index, fields_count or operator+=.
  • CustomSerializerT(std::string) is explicit, CustomTypeName<T>::get() is not
    constexpr, Schema::custom_types and custom_schemas are std::map (were
    std::unordered_map), and long double is a compile error. Removed:
    getPointerType(), SerializeMe::enable_if_same_t and the library's
    AddFieldToHash(), replaced by SchemaTextHash() and ComputeSchemaHash() (the
    parser keeps its own, for version 4 texts).
  • Parser: ParseSnapshot() loses its unused callback_custom parameter and is
    [[nodiscard]]; ParseSnapshotRecursive moves to DataTamerParser::detail and
    NullCustomCallback is removed; SnapshotView::schema_hash is uint64_t;
    BuilSchemaFromText() is a deprecated alias of BuildSchemaFromText(). The header
    needs a little-endian host and no longer includes <iostream>.
  • sinks/ros2_publisher_sink.hpp no longer includes the data_tamer_msgs headers.

Behaviour changes

  • MCAPSink continues in a new numbered file at every reset (600 s by default) instead
    of truncating its file, so disk use is unbounded; setCreateNewFileOnReset(false)
    truncates as before. Rollover files are named run_1.mcap (1.x: run.mcap_1) and
    skip names that exist; the path passed to create() is still overwritten.
  • SinkWorker::stop() delivers what is queued, then finishes the sink: MCAPSink
    closes its file, and start() continues in the next numbered file. Destroying a
    running worker stops it.
  • A channel's snapshot pool bounds the memory of its sinks: by default as many slots as
    fit in 8 MiB, from 64 to 1024, per channel. A sink that falls behind loses snapshots
    (pool_exhausted) instead of growing a queue.
  • ROS2PublisherSink: the data topic is reliable KeepLast(100) (was KeepAll;
    ROS2PublisherOptions::data_qos), and the schemas topic is transient-local
    KeepLast(1), published by startLogging().
  • LoggedValue::set() never re-enables a disabled value. set() and get() of a
    struct or vector lock the channel's write mutex, so they wait for a snapshot in
    progress.
  • An id kept across unregister() and a new registration of the same name is stale:
    setEnabled() and unregister() throw std::invalid_argument (1.x acted on the
    new registration).
  • A snapshot taken by a thread that holds the write mutex (scopedWrite(), a
    LoggedValue guard) returns blocked; in 1.x it deadlocked.
  • From a sink callback, a control call that would wait for that callback's own worker
    throws std::logic_error and changes nothing instead of deadlocking (see DataSink
    in data_sink.hpp).
  • Value names, custom type names and field names must be non-empty and free of whitespace
    and control characters (1.x refused only spaces); a channel name can't be empty or hold
    a control character. Two C++ types with one custom type name in a channel make
    registerValue() throw.
  • A channel holds at most 8 sinks, and a registry at most 8 default sinks.
    addDefaultSink() also attaches the sink to the channels that exist already.
  • MCAP messages carry a sequence number, 1, 2, 3, ... per channel and file (was always 1).
    The MCAP sinks don't write a snapshot with a negative timestamp; they count it in
    errors().
  • Wire format: the schema text is version 5 and its hash is FNV-1a 64 of the text, the
    same on every platform. A std::string is written as char[], a uint32 count
    and the characters. 1.x readers reject 2.0 files: use PlotJuggler 4 or a 2.0 parser.
  • The parsers reject a schema text without a ### version: line (releases before
    0.7.0). ParseSnapshot() returns false when bytes are left after the last field, and
    throws on malformed data.

New

  • LogChannel::startLogging() freezes the schema, allocates the pool and announces the
    schema; the first takeSnapshot() or tryTakeSnapshot() with a sink attached
    calls it. tryTakeSnapshot(), for real-time threads, never blocks or allocates once
    logging started, also when data_tamer is loaded with dlopen(); the call that starts
    logging is not real-time (RealtimeSanitizer reports it), so call startLogging()
    before the loop. Without a sink it returns no_sinks, in real time; if starting
    throws, start_failed, and the next call retries. Several threads may take snapshots
    of one channel.
  • accepted(SnapshotResult), constexpr and real-time safe: true for ok and
    partial, what stats().accepted counts.
  • tryTakeSnapshot() is noexcept: a serializer that throws ends it with
    SnapshotResult::serialize_failed, counted by serializeFailures();
    takeSnapshot() lets the exception through, as in 1.x. The library catches it, so
    serializers compiled into binaries built against earlier headers are covered too.
  • LogChannel::scopedWrite() groups writes from another thread into one snapshot.
    registerValue() accepts a const std::atomic<T>*.
  • setEnabled() is lock-free; new trySetEnabled() (noexcept) and
    isEnabled(id).
  • setPoolCapacity() (a count, or a stall tolerance and a period),
    setPayloadCapacity() and ChannelsRegistry::setChannelDefaults().
  • LogChannel::stats() and SinkWorker::stats(): snapshot attempts and accepted
    snapshots, pool, oversize, serialization-failure and lock counters, refusals per sink,
    deliveries, errors and queue high water.
  • MCAPRingSink, a flight recorder: it keeps the last window of every channel in
    RAM and writes an MCAP file on requestDump(), which is real-time safe.
  • ROS2PublisherSink takes ROS2PublisherOptions: QoS, aggregation into
    data_tamer_msgs/SnapshotBatch (new message) and the schema format.
  • A YAML rendering of the schema (wire format section 2.1, also version 5): ToYaml(),
    RenderSchema(), SchemaFormat::Yaml.
  • DataSink::onStop() and onStart(), SinkWorker::Delivery::Manual (no thread:
    the application calls drain()).
  • ChannelsRegistry::stopAll() and shutdown() (stopAll() then clear(), for
    the end of main(): every sink is finished and no channel is left borrowing your
    variables).
  • TypeDefinitionTrait<T> describes a type without reopening its namespace (Eigen,
    ...).
  • Real-time checks with Clang 20 or later: the real-time API is declared
    [[clang::nonblocking]] (DATA_TAMER_NONBLOCKING), so in a function marked
    [[clang::nonblocking]] -Wfunction-effects rejects a call to takeSnapshot(),
    a registration or a non-scalar LoggedValue::set(), and RealtimeSanitizer
    (-fsanitize=realtime) reports every allocation, lock or system call at run time,
    in data_tamer and in custom serializers. On Linux (ELF) data_tamer declares to RTSan
    the futex calls the real-time contract allows. A new rtsan preset and CI job.
  • New headers: names.hpp (JoinNames() and IsCanonicalName() build and check
    hierarchical names), sinks/mcap_encoding.hpp (the MCAP encoding of MCAPSink, for
    code that writes data_tamer MCAP files itself) and fwd.hpp (forward declarations).
  • Decoding in C++ in a few lines: data_tamer_v2/mcap_reader.hpp reads an MCAP file
    (ForEachValue() decodes each value, ForEachSnapshot() hands over each snapshot),
    in log time order (a crashed recording up to its last complete chunk, in file order),
    optionally restricted to some topics, with every error a
    std::runtime_error naming the file and the topic, like Python's iter_mcap();
    data_tamer_v2/decode.hpp has ToSnapshotView(), the parser's view of a
    Snapshot in a custom sink. T07 and T08 use them.
  • Parser: SchemaRegistry, DataTamerParser::ToSnapshotView() of a ROS 2 message and
    ForEachSnapshotInBatch() decode the ROS 2 messages without ROS;
    SplitMcapMessage() splits an MCAP message.
  • Parser: ParseSnapshot() decodes several times faster and allocates nothing per
    value (benchmarks/parser_benchmark.cpp measures it); its depth limit no longer
    depends on the order of the fields.
  • Text: std::string, any other std::basic_string<char> (std::pmr::string) and
    std::string_view are recorded as the new basic schema type string (a uint32
    byte count and the bytes): registered, as members of custom types, in a
    std::vector or std::array, and as LoggedValue<std::string>. Their
    registration now also compiles with Clang 18 and older. ParseSnapshot() and
    ForEachValue() take an optional string callback that gets a std::string_view
    into the payload (without one, text is skipped); the Python decoder returns a str
    and the TypeScript decoder and the Foxglove extension a string, decoded as UTF-8 with
    invalid bytes replaced by U+FFFD. std::vector<char> and std::array<char, N>
    stay one value per character. BasicType::STRING is appended to the enum. A custom
    type named after a basic type is refused. docs/wire_format/vectors/ has a strings
    vector.
  • Python decoder, standard library only: pip install "data-tamer-parser[mcap]".
    parse_snapshot() compiles each schema and active mask into struct formats,
    5 to 15 times faster than a value-by-value walk; parse_snapshot_values() returns
    names and values as tuples, for column-oriented code.
  • Foxglove extension in data_tamer_foxglove/extension/ for ROS 2 topics and
    rosbag2 recordings: decoded snapshots with header timestamps, schema catalogs,
    batch inspection, and packaged .foxe release artifacts.
  • TypeScript snapshot decoder in data_tamer_foxglove/, with schema versions
    4 and 5 (line format and YAML), bounded compiled layouts, nested objects and ROS 2 message helpers.
  • Tutorials T01 to T08, run by ctest; docs/USER_MANUAL.md,
    docs/llm_user_manual.md, docs/MIGRATION.md and the wire format specification
    docs/wire_format.md, with golden vectors.

Fixed

  • long long, unsigned long long, wchar_t, char16_t, char32_t and enums
    over them were recorded as other, which no reader decodes.
  • A std::string was written with a 16-bit length that no reader decoded.
  • Two C++ types with one custom type name in a channel: the second was serialized with the
    first one's serializer.
  • A registration that threw could leave a custom type in the schema; it now leaves the
    channel unchanged.
  • The schema text and its hash depended on the global C++ locale.
  • Concurrent takeSnapshot() calls on one channel raced on its snapshot buffer.
  • The parser read past the buffer before checking its size. Both decoders now check bounds
    before every read and reject masks shorter than the schema, oversized counts and cyclic
    custom types.

Packaging and ABI

  • ROS 2: the package is data_tamer_v2 and installs beside data_tamer_cpp 1.x (the
    library is libdata_tamer_v2.so.2). Link data_tamer_v2::data_tamer, or use
    find_package(data_tamer 2) and data_tamer::data_tamer. mcap_vendor is linked
    publicly; -DBUILD_TESTING=OFF builds no test. data_tamer_msgs keeps its name.
  • Without ROS: package data_tamer, target data_tamer::data_tamer, SONAME
    libdata_tamer.so.2 (1.x had none). Building needs CMake 3.22 and C++20; consumers
    need C++17. -Werror is opt-in (DATA_TAMER_WERROR). As a subproject
    (add_subdirectory(), FetchContent) data_tamer installs nothing and builds no test,
    example or ROS 2 sink.
  • Outside ROS 2, MCAP is 2.1.3 (1.x bundled 1.3.0). CMake downloads it, pinned by hash
    (FETCHCONTENT_SOURCE_DIR_DATA_TAMER_MCAP for offline builds), unless
    DATA_TAMER_USE_SYSTEM_MCAP is on (conan). The library contains the MCAP
    implementation and installs its headers: don't define MCAP_IMPLEMENTATION. The
    mcap_lib target is gone; zstd and lz4 are linked privately. The vendored moodycamel
    queue is removed.
  • ABI: within 2.x a newer library runs binaries built against older 2.x headers.
    tests/abi_tests.cpp pins the layouts and the abi CI job (libabigail) compares
    each change with the last 2.x release. The DataSink and CustomSerializer vtables
    and the MCAP version are frozen; the stats() functions are inline and built from
    exported getters, so their structs can grow. ROS2PublisherSink also follows the
    rclcpp ABI. Header defaults apply to rebuilt consumers only, Pimpl defaults to old
    binaries too.
  • SinkWorkerOf<T> has the layout of SinkWorker (pinned in
    tests/abi_tests.cpp), and the factories that return it are inline.
  • One version number for the library, the ROS packages, conan, the Python and TypeScript
    decoders and the Foxglove extension, checked by tools/check_versions.py. Pushing a
    tag X.Y.Z creates the GitHub release, whose notes are this section
    (tools/release_notes.py).