DataLogger

DataLogger

The DataLogger is a SysMod that records device data to on-board flash storage (LittleFS) in either CSV or JSONL format. It is designed for long-duration autonomous logging — sessions can be time-limited or run until stopped. It is part of the Axiom firmware (components/DataLogger in RoboticalAxiom1).

Overview

• Records decoded (CSV) or raw (JSONL) poll data from bus devices
• Per-device logging rate control (from the device's own rate down to 1/day)
• Per-device attribute selection (CSV - log a subset of fields); an array attribute has a column per element
• Configurable session duration with automatic stop
• A start that would produce more data than can sensibly be written is refused, with an explanation - and the rate can be estimated before starting
• Starting and stopping never hold up the main loop: start and stop reply when they are done (a deferred reply)
• Double-buffered writes, so the bus polling task never waits for the file system
• Log files stored in /local/logs/ on the device file system

REST API

All actions are on datalog:

Action	URL	Description
Start	datalog?action=start&config=<json>&label=<label>	Start a logging session (see below)
Estimate	datalog?action=estimate&config=<json>	The data rate a start with this config would give, and the limit, without starting
Stop	datalog?action=stop	Stop the current session; the reply (with the session's statistics) comes when the file is closed
Status	datalog?action=status	Session state and statistics
Clear	datalog?action=clear&confirm=true	Delete the last session's log file (not while a session is running)
Simulate	datalog?action=simulate&rateMs=<ms>&numDevices=<n>&recordSize=<bytes>&label=<label>	Start a session of generated JSONL records, for testing write throughput (defaults 1000 ms, 1 device, 80 bytes, label sim-test)

Start parameters:

Parameter	Description
config	Session configuration JSON (below). If omitted, the SysTypes config is used; if there's none, every identified device is logged as JSONL
label	Session label, used in the file name (default log)
UTC	Set the device clock first, e.g. 2026-10-21T13:02:21Z (so the log is timestamped without NTP)
epoch	Set the device clock first, from seconds since 1970 (if UTC isn't given)
tz	POSIX timezone string for the local start time in the CSV header (overrides SysTypes)

Configured devices still being identified (shortly after boot) are waited for, for up to 5 s. Where the connection supports deferred responses (HTTP, serial console, BLE and WebSocket via RICREST) the reply comes once the session has started (or failed); otherwise the reply is "state":"starting" and status shows how it went.

Start Configuration JSON

{
  "format": "csv",
  "csvHeader": true,
  "durationMs": 600000,
  "devices": [
    {
      "bus": "I2CA",
      "addr": "76a",
      "rateMs": 100,
      "attrs": ["gx", "gy", "gz"]
    },
    {
      "bus": "I2CA",
      "addr": "736",
      "rateMs": 0
    }
  ]
}

Field	Type	Description
format	string	"csv" or "jsonl" (default jsonl)
csvHeader	boolean	Include the metadata comment block at the top of a CSV file (default: true)
durationMs	number	Auto-stop after this many milliseconds (0 = unlimited)
devices	array	Per-device configuration (see below)

Per-device fields:

Field	Type	Description
bus	string	Bus name (e.g. "I2CA") or number (e.g. "1"); omitted = any bus
addr	string	Device address as lowercase hex, no 0x prefix (e.g. "76a" for slot 7, I2C addr 0x6a)
rateMs	number	Minimum interval between logged records in milliseconds. 0 = log every sample
attrs	string[]	(CSV only) Attribute names to log. Omit or empty to log all of them. An array attribute is selected as a whole

Data Rate Limit and Estimate

A start whose estimated data rate is over the limit is refused: the file system couldn't write it all without losing records, and it would hold up the main loop. The estimate is made from the devices identified at the start (and again once any still being identified have been found):

{"req":"datalog?action=start&config=...","reason":"the estimated data rate of 52000 bytes/s is over the limit of 40000 bytes/s (more can't be written to the file system without losing records) - log fewer devices or attributes, or set a longer rateMs",
 "bytesPerSec":52000,"maxBytesPerSec":40000,"withinLimit":false,
 "devices":[{"bus":"I2CA","addr":"76a","type":"LSM6DS3","recsPerSec":416.0,"recordBytes":125,"bytesPerSec":52000}],
 "fsFreeBytes":13918208,"fillSecs":267,"rslt":"fail"}

datalog?action=estimate gives the same fields (with rslt ok), so a client can warn before starting - the raftjs example dashboard shows the estimated rate and how long until the storage is full, and warns if it is over the limit.

Field	Description
bytesPerSec	Estimated data rate of the session
maxBytesPerSec	The limit (SysTypes maxBytesPerSec; 0 = no limit)
withinLimit	Whether a start would be accepted
devices	Per device: records per second, bytes per record and bytes per second
fsFreeBytes, fillSecs	Free space on the file system, and how long until it is full at this rate (when known)

How it is estimated, per device:

• Records per second: CSV has a row per sample. For a device with a sample rate setting (a _conf.rate action, e.g. the LSM6DS3's ODR) that is the rate the device is set to (devman/devconfig?sampleRateHz=); otherwise a sample per poll. JSONL has a record per poll. Either way at most one every rateMs.
• Bytes per record: CSV - the time, a comma for every column of the session, and typical widths of the device's values. JSONL - the record for the most a poll reads, so for a FIFO device it is an upper bound.

The estimate is on the high side (by about a quarter for an LSM6DS3 in CSV). The default limit, 40000 bytes/s, was measured on Axiom018: over 75 KB/s was written without losing records, but each 1 KB write takes the main loop 2.3 ms on average, and the default keeps that to about a tenth of the main loop's time.

Status Response

{"req":"datalog?action=status","active":true,"state":"active","label":"log","fileName":"/logs/261021-130221-log.csv",
 "simulate":false,"durationMs":16126,"totalBytesWritten":95821,"samples":1789,"flushCount":115,
 "flushLatency":{"minUs":40,"maxUs":22973,"avgUs":3811},"bufferOverflows":0,"maxBufferFillBytes":4254,
 "bytesPerSec":5942.0,"numDevices":6,"config":{"devices":[...]},"rslt":"ok"}

Field	Description
active	Records are being logged
state	idle, starting (waiting for devices still being identified), active or stopping (writing out the buffers and closing the file)
startError	Why the last start failed - the rate limit (the same text as the refusal), stopped (stopped while starting), or the log file couldn't be opened - until the next start. Omitted if it didn't fail
label, fileName, simulate	The session (also after it has stopped, until the next one)
durationMs	Time since the session started (while active)
totalBytesWritten, samples	Bytes written to the file, records logged
flushCount, flushLatency	File writes (1 KB each at most) and their time
bufferOverflows	Records dropped because the write buffer was full (the file system couldn't keep up)
maxBufferFillBytes	Most data waiting to be written
bytesPerSec	Data rate so far (while active)
numDevices, config	Devices registered, and the configuration with each device's registered state

CSV Format

CSV files begin with an optional metadata comment block (lines prefixed with #), followed by a column header row and data rows.

Metadata Header

# Axiom Data Log
# Start (UTC): 2026-10-21 13:09:52
# Start (local): 2026-10-21 13:09:52
# Label: arr
# Bus I2CA, Addr 736: MAX17048A - regs[3], vc[2], status
# Bus I2CA, Addr 76a: LSM6DS3 - gx, gy, gz, ax, ay, az

One line per device: its bus, address, type and the attributes logged (an array attribute as <name>[<count>]). The start times are given only if the clock was set (by NTP, or UTC/epoch on start).

Column Layout

The first column is always time — elapsed seconds from session start, with millisecond precision (e.g. 10.050).

The other columns are the logged attributes, in device order. A column is named by the attribute (gx), or <busNum>_<addr>_<attrName> (e.g. 1_76a_gx) if more than one device has an attribute of that name. An array attribute (a type like <h[64], e.g. a thermal camera's pixels) has a column per element, <name>[i]: temperature[0] ... temperature[63].

Each data row contains values for one device only. Columns belonging to other devices are left empty (sparse rows). This avoids timestamp alignment issues when devices produce data at different rates.

time,regs[0],regs[1],regs[2],vc[0],vc[1],status,gx,gy,gz,ax,ay,az
0.000,,,,,,,-0.14,0.00,-0.07,-0.03,0.12,-0.96
0.129,53456,25163,0,208.8,98.3,64,,,,,,
0.502,,,,,,,-0.07,0.00,-0.07,-0.03,0.12,-0.97

Field values are decoded using the device type's decoder and have divisors and addends applied: a float, or a scaled value, is written with the attribute's f format (e.g. .2f) or as %g; an integer attribute with no divisor or addend is written exactly as an integer.

JSONL Format

Each line is a self-contained JSON object. The file begins with a header record, followed by device-info records for each configured device, then data records.

Header Record

{"_t":"header","fmt":"jsonl","label":"log","startTime":"2026-04-06T17:03:48Z","startMs":128019,"config":{"devices":[...]}}

fmt is poll-jsonl for a simulated session (which has "simulate":true instead of config).

Device Info Record

{"_t":"devinfo","a":"76a","b":1,"ti":11,"info":{...}}

Contains the full device type info JSON, enabling offline decoding without access to the device. A device identified after the session started gets its record then.

Data Record

{"_t":"d","t":26,"a":"76a","b":1,"ti":11,"x":"f42c24000000feff..."}

Field	Description
_t	Record type: "d" for data
t	Timestamp — milliseconds since session start
a	Device address (lowercase hex)
b	Bus number
ti	Device type index
x	Raw poll data as hex string

The x field contains the raw poll response bytes in hex. This can be decoded offline using the attribute definitions or custom decode function from the corresponding devinfo record.

Implementation Details

Data Flow

1. At session start, registerBusCallbacks() registers (through DeviceManager) a data callback for each device matching the session configuration; devices still being identified are registered when they are found (waiting for up to 5 s; in a JSONL session also later, checked every 5 s while logging - a CSV file's columns are fixed when it starts).
2. The bus polling task calls deviceDataCallback() when new data is available.
3. The callback decodes the data (CSV) or hex-encodes it (JSONL), formats the record, and appends it to the active write buffer (in PSRAM).
4. loop() swaps the buffers (when 4 KB is waiting, or every 5 s) and writes the one swapped out to the file 1 KB per call, so the main loop isn't held up for long. Stopping writes out what is left the same way, then closes the file.

The value formatting (CSV cells, JSON values, array attributes) and the data rate estimate are in DataLogFormat.h, which is free of Raft dependencies and tested on the host (tests/datalogger in RoboticalAxiom1).

Rate Limiting

Each device callback context tracks lastOutputTimeMs. If rateMs > 0, a sample less than rateMs after the last one logged is dropped, so for a FIFO device producing several samples per poll the first sample of each rate window is logged.

Sign Extension

When using custom decode functions (e.g. LSM6DS3 FIFO), the pseudocode assembles values as unsigned integers. The code generator automatically inserts sign-extension code for attributes with signed underlying types ("t":"<h") stored as float ("o":"float"). See Device Type Record Format — Automatic Sign Extension.

Timestamp Handling

The decode state is pre-seeded with the current timestamp wrap offset when callbacks are registered. This ensures that 16-bit timestamps from FIFO data are correctly unwrapped even if wraps occurred before the logging session started.

In CSV, a device whose poll gives one record uses that record's poll timestamp. A FIFO device, whose decode gives several samples per poll (its custom decode code says next inside a loop - e.g. the LSM6DS3 and MAX30101), is timed by its sample cadence instead: an estimate of the sample interval (an exponential moving average over polls), with the first sample at time 0. (The choice is made from the decode, not from pollInfo "s", which is how many poll results are kept.)

SysType Configuration

The DataLogger is added as a SysMod in the SysType JSON:

{
  "DataLogger": {
    "enable": true,
    "logDir": "/logs",
    "maxBytesPerSec": 40000
  }
}

Field	Type	Default	Description
enable	boolean	true	Enable the DataLogger SysMod
logDir	string	"/logs"	Directory on the file system for log files
maxBytesPerSec	number	40000	Data rate limit for a start (0 = no limit) - see Data Rate Limit and Estimate
config	object	none	Session configuration used when a start doesn't give one (the same as the start's config)

Log File Management

Log files are stored in /local/logs/ and named <yymmdd>-<HHMMSS>-<label>-log.<format> from the UTC start time (e.g. 261021-130952-arr-log.csv; 261021-130221-log.csv for the default label). If the clock isn't set, the time since boot in milliseconds is used instead of the date and time.

Files can be listed, downloaded and deleted via the standard Raft file API (filelist/local/logs, fileread, filedelete, with ~ in place of / in a folder path - see File System), or downloaded from the web server at /files/local/logs/<name>.

See Also

• Device Type Record Format — device definitions used for decoding
• Device Data Publishing — real-time data publishing (complementary to logging)
• Device Manager — bus and device management