DeviceTypeSources

Selecting Device Types (DeviceTypeSources.json)

Device type records are kept one file per record: RaftCore's are in devtypes/records/, as <Key>.json, for example MCP9808.json. Each file holds the record exactly as it appears under "devTypes" in a catalogue file (see Device Type Record Format). The file name is the device type key, and a deviceType field, if present, must equal it.

A project chooses the records compiled into its firmware with a DeviceTypeSources.json file in its SysType folders.

Existing projects

Nothing changes for a project without a DeviceTypeSources.json:

• DEV_TYPE_JSON_FILES in features.cmake and systypes/Common/DevTypes.json / systypes/<SysType>/DevTypes.json work as before;
• /devtypes/DeviceTypeRecords.json still means "all RaftCore's records": with a RaftCore whose records are one file per record, it reads the devtypes/records folder;
• A DEV_TYPE_JSON_FILES entry can also be a folder of record files.

DeviceTypeSources.json

Put it in systypes/Common/ and/or systypes/<SysType>/. The SysType's file adds to Common's.

{
    "sources": {
        "raftcore": { "bundled": true },
        "local": { "path": "devtypes" }
    },
    "builtin": {
        "include": ["raftcore:VL53L0X", "raftcore:MCP9808", "raftcore:TrillRing", "local:*"],
        "exclude": [],
        "policy": { "TrillRing": "disabled" }
    }
}

sources

Each source is named, and the name is used in include:

Source	Means
{ "bundled": true }	RaftCore's records, at the RaftCore version the build uses ("builtin": true, its earlier name, still works)
{ "path": "devtypes" }	A folder of record files, a single record file (devtypes/MySensor.json), or a catalogue file ({"devTypes": {...}}). The path is relative to the DeviceTypeSources.json that names it
{ "git": "<url>", "ref": "<ref>", "path": "records" }	Records in a git repo - see Git sources

builtin

The records compiled into the firmware:

• include: "*" (everything from every source), "<source>:*", "<source>:<key>", or a bare "<key>" (only when one source has it). The default is everything.
• exclude: the same forms, removed from what include selected.
• policy: "enabled", "disabled" or "disabledClaim" for a selected record (as typePolicy).

Choosing between sources

• A path source replaces the same key from another source, with a build message, so a project changes a RaftCore record by putting its own copy in a local folder. Of two path sources, the later one wins.
• The same key from two other sources is an error, unless include names the source to use (e.g. "raftcore:LSM6DS3").
• Naming a source in include ("raftcore:LSM6DS3") takes the record from that source even if a local folder has one too.

Board choices

A record describes the device, and every board uses the same one. Choices for the board or product - the poll bus speed and the slots it applies to, the poll interval, samples kept, sample rate and scan priority - are set for the type in the SysType's DevMan typeSettings, not by changing a copy of the record.

Order

Records are compiled in device type key order, whatever order they are listed in. At an address, records are tried in descending identPriority, then descending detection strength, then key order. So where the order between two records matters, give one of them an identPriority.

DEV_TYPE_JSON_FILES and DevTypes.json records still come after the selected records, in their files' order, and replace selected records with the same key, as before. With a DeviceTypeSources.json, select RaftCore's records there rather than with /devtypes/... in DEV_TYPE_JSON_FILES (the build stops if both are used).

Git sources

"sources": {
    "acme":   { "git": "https://github.com/acme/raft-devtypes.git", "ref": "v1.2.0" },
    "widget": { "git": "git@github.com:bob/widget-fw.git", "ref": "3f2a9c1e7b0d...", "path": "devtypes/Widget.json" }
}

• ref: a tag, a branch, or a commit given as its full 40-character hash.
• path: a folder of record files or one record file in the repo. The default is records.
• Fetching: the source is fetched shallowly with only path checked out, into build/<SysType>/raft/devtypes/. A commit already fetched isn't fetched again, so builds work offline once the cache is filled.
• The lock file, systypes/<SysType>/DeviceTypeSources.lock, records the commit each source's ref resolved to, and the SHA-256 of each record used. Commit it with the project.
  • While a source's git and ref are unchanged, the build uses the locked commit, so a moved tag or a new commit on a branch doesn't change a build.
  • A new or changed source is resolved and the lock updated, with a build message.
  • To take new commits, run raft devtypes update (or python3 RaftCore/scripts/AssembleDeviceTypes.py ... --update).
  • A cached record that doesn't match the lock is an error.
• Local copy: a folder (or link) raftdevtypes/<source> at the project root replaces that git source, for editing records against a real build before committing them. A broken link is an error. Adding or removing one re-runs the selection.
• Source manifest: a repo may have raftdevtypes.json at its root, {"name": "acme", "formatVersion": 1, "minRaftCore": "1.60.0"}. A format this RaftCore doesn't know, or a minRaftCore newer than the build's, is an error with a clear message.
• Decode code: a record's decode code (resp.c) is compiled into the firmware. A record from a git source whose code uses anything outside the decode language (function calls, other names, other characters) is rejected.
• Same key in two sources: a key that a git source and RaftCore (or two git sources) both have must be named in include, e.g. "acme:LSM6DS3".
• Credentials: builds use the git on the build machine, with its credentials. A git@github.com: URL needs an SSH key that the build can use. In the Docker build container that means mounting the key, so a public https:// URL is simplest.

Build

The selection is resolved when the build is configured:

• An error, such as a key no source has, stops the build with a message naming it.
• Changing a selection file, or adding or changing a record file in a selected folder, re-runs it.

The build writes:

File	Contents
build/<SysType>/raft/DeviceTypeRecords.json	The selected records, as one catalogue file (what the generator reads)
build/<SysType>/raft/DeviceTypeSources.manifest.json	For each record, its source, file, SHA-256, and for a git source the commit
systypes/<SysType>/DeviceTypeSources.lock	The commits git sources are pinned to (commit it)
build/<SysType>/raft/devtypes/	Fetched git sources, by commit

To check a selection without a full build:

python3 RaftCore/scripts/AssembleDeviceTypes.py --raftcore RaftCore \
    --select systypes/Common/DeviceTypeSources.json,systypes/Main/DeviceTypeSources.json \
    --out /tmp/DeviceTypeRecords.json --manifest /tmp/manifest.json

--all writes every RaftCore record as one catalogue file (also attached to each RaftCore release as DeviceTypeRecords.json). --split <catalogue.json> --out-dir <folder> splits a catalogue file into one file per record, keeping each record as written.

Runtime type files

The typeFile section of DeviceTypeSources.json selects records for a device type file - records loaded at startup from the device's file system, so they can be added or changed without rebuilding the firmware:

"typeFile": {
    "include": ["acme:Gadget", "local:MPU6050"],
    "policy": { "TrillRing": "disabled" }
}

It takes the same include, exclude and policy as builtin, but includes nothing unless it says so. A policy for a type that isn't in the file applies to the compiled record (disabled or disabledClaim - a type file can't enable a compiled type). To build the file:

python3 RaftCore/scripts/AssembleDeviceTypes.py --raftcore RaftCore --select systypes/Main/DeviceTypeSources.json \
    --typefile --out devlib.json [--compiled build/Main/raft/DeviceTypeRecords.json] [--report report.json]

• The file is minified, without _notes, within the 64 KB limit (--max-bytes).
• Each record is reported as new, an override of a compiled record, or unchanged - identical to the compiled record, so left out of the file. Policies are reported as policy.
• Records with data but no compiled decoder are listed: new records, and overrides whose response layout differs from the compiled record's. Firmware with runtime decoding (RAFT_RUNTIME_DECODE, the default) decodes them on the device from their attributes; a record with custom decode code (resp.c) is decoded on the device only by firmware with the code interpreter (RAFT_RUNTIME_DECODE_CODE); otherwise only raftjs and the WebUI decode it. The report's entries give noDecoder: true and decoder: "runtime" or "none" (custom code). A record with its own initCode is reported with initCode: "runtime" (only firmware with the interpreter runs it) or "runtime or compiled" (firmware without it uses the compiled record's).
• The compiled records are those of the builtin selection of the same files, unless --compiled gives a catalogue file. For a project that adds records with DEV_TYPE_JSON_FILES or DevTypes.json, use --compiled build/<SysType>/raft/DeviceTypeRecords.compiled.json: the build writes every record it compiles there.

In the file system image

With "fsImage": true in typeFile, the build also puts the type file in the SysType's file system image, so flashing the board (raft flash, without --no-fs) installs it and the device applies it at its first boot:

"typeFile": {
    "include": ["local:Gadget"],
    "fsImage": true
}

• It is built against every record the build compiles (the selection, DEV_TYPE_JSON_FILES and DevTypes.json).
• It goes where the device loads it from: DevMan typeFile in SysTypes.json (default /local/devlib.json), which must be on /local/. DevMan typeFileLimits.maxFileBytes is the size limit.
• A file at the same path in the SysType's FSImage folder is replaced, with a build warning. The SysType needs a file system image (FS_TYPE and FS_IMAGE_PATH in features.cmake).
• Only a full flash writes it: an OTA update, or raft flash --no-fs, leaves the device's file as it is. And flashing the file system image replaces a type file pushed or saved on the device since.

Then upload it as /local/devlib.new.json, check it on the device (devman/typefile/check?file=/local/devlib.new.json - the device's own rules decide) and apply it (devman/typefile/apply?file=/local/devlib.new.json). raft devtypes push does these steps.

raft CLI

raft devtypes (alias raft dt) works on a project's selection files (-s <SysType>), or on one DeviceTypeSources.json with --config <file>. It runs scripts/DeviceTypesTool.py from the project's RaftCore (raftdevlibs/RaftCore, the build's copy, or --raftcore), so the rules are those of the firmware being built. Once the project has been built, type files are built against the records that build compiled (build/<SysType>/raft/DeviceTypeRecords.compiled.json - the firmware the device runs); before that, against the builtin selection.

Command	Does
list	The records for builtin and typeFile, with source, commit, policy, and for the type file new / override / unchanged / policy and records with no on-device decoder
init	Create the SysType's DeviceTypeSources.json (Common's with --common) selecting raftcore:*
add <pattern>...	Add to builtin.include (--typefile: typeFile.include). A pattern that doesn't resolve isn't added
remove <pattern>...	Remove from include; a record selected by a wildcard is added to exclude
new <key> [--source <name>] [--address 0xHH]	A new record file in a path source, with the fields to fill in
check	The generator's checks on the compiled selection, and grammar and decode code checks on type file records
update	Resolve git sources' refs again and update the lock file
diff [<key>]	Local copies against the records they replace, and type file overrides against the compiled record
typefile	Build the type file (build/<SysType>/raft/devlib.json, or --out), reporting size, CRC and kinds
push <ip>	Build the type file, upload it as devlib.new.json, show the device's check, ask, apply, wait for the restart and confirm the device loaded it. --dry-run stops after the check; --yes doesn't ask; warns if the device's file was changed since the last push from the project
pull <ip>	Download the device's type file and list the records that differ from the project's
status <ip>	What the device loaded at startup (per entry), its limits, and whether it is the project's file

Device commands use HTTP (-p <port>). The same steps are available from raftjs (checkTypeFile, applyTypeFile, getTypeFileStatus).

Plan

See devdocs/device-type-record-management-plan.md for the whole design, including:

• builds with no compiled records.