Skip to content

documentation improvement plan

Jump to bottom
Rob Dobson edited this page May 3, 2026 · 1 revision

Raft Documentation Improvement Plan

Status (May 2026): Phases A–F all complete — the structural overhaul, the communications-stack documentation, the missing built-in SysMod pages, the architecture/tutorial pages, and the helper-class index have all landed. Remaining work is now confined to (a) optional polish items called out below and (b) any future content as the framework evolves. See § Remaining work at the bottom of this document.

This plan was created to coordinate a structural overhaul of the Raft wiki and to track the discrete content gaps and quality fixes identified during a full review of RaftCore.wiki/ against the source in RaftCore, RaftSysMods, RaftI2C and RaftWebServer.

Progress snapshot

Phase Scope State
A Quick wins (factual fixes, stale toolchain pins, empty index page) ✅ Done
B Restructure navigation (new Home.md, _Sidebar.md, H1/intro clarifications) ✅ Done
C Communications stack docs (overview, channels, ProtocolExchange, RICREST) ✅ Done
D Built-in SysMod pages (WebServer, MQTT, FileManager, LogManager, SerialConsole, CommandSerial/Socket/File, ESPOTAUpdate) ✅ Done
E Tutorials and architecture (Architecture-at-a-glance, Writing Your First SysMod, Adding a Comms Channel) ✅ Done
F Helper-class index, link audit, plan recording maintainer decisions ✅ Done

Pages added during the project (17): CommunicationsStackOverview, CommsChannels, ProtocolExchange, RICRESTProtocol, RaftWebServerSysMod, RaftMQTTManagerSysMod, RaftFileManagerSysMod, RaftLogManagerSysMod, RaftSerialConsoleSysMod, RaftCommandSerialSysMod, RaftCommandSocketSysMod, RaftCommandFileSysMod, RaftESPOTAUpdateSysMod, RaftArchitecture, WritingYourFirstSysMod, AddingACommsChannel, RaftHelperClasses.

Pages restructured / fixed: Home, _Sidebar, PlatformIOArduinoQuickStart, RaftSysModSettings, RaftBLEManagerSysMod (stray fence), RaftNetworkManagerSysMod (xref), RaftCoreAPI, RaftSysModAPI, RaftFullAPIList, RaftRestAPIEndpoints, QuickStart (broken link).

Goals

  1. Restructure the wiki top-down so new users have an obvious learning path and advanced users can find reference material quickly.
  2. Close major content gaps — particularly around the message/transport stack (ProtocolExchange, CommsChannels, RICREST) and several SysMods that have no documentation at all.
  3. Rationalise naming, navigation and cross-links so every page is reachable from Home.md via a clear hierarchy.
  4. Promote stable content out of devdocs/ in each repo into user-facing wiki pages where appropriate.
  5. Avoid breaking external links — wiki pages are linked from READMEs and possibly external sources. Renames should be done with redirects/stubs where feasible.

Proposed top-down structure for Home.md

The new Home.md would replace the current flat link list with a layered, audience-oriented table of contents. Suggested sections (names indicative, not final):

1. Introduction

  • What Raft is, who it's for, supported chips
  • Architecture at a glance (one diagram showing SysManager + SysMods + Comms stack + Web/BLE/Serial transports + I2C/Device stack)
  • Repository map (RaftCore, RaftSysMods, RaftI2C, RaftWebServer, optional libraries) — what lives where and why

2. Getting Started

  • Quick Start with RaftCLI (existing QuickStart, expanded with a working example)
  • "Writing your first SysMod" walkthrough (new) — toggle an LED on a timer, expose a REST endpoint, publish state
  • PlatformIO/Arduino Quick Start (refreshed; currently stale)

3. Scaffolding and Building

4. Raft Architecture

5. Built-in SysMods (reference)

Grouped by capability, each with a settings page:

6. Communications and Protocols (new section — biggest current gap)

  • Communications stack overview (new) — channels, codecs, ProtocolExchange, REST endpoint manager
  • CommsChannelManager and CommsCoreIF (new)
  • ProtocolExchange (new)
  • RICREST / RICSerial / RICFrame / RICJSON wire formats (new)
  • Bridges between channels (new)
  • "Anatomy of a request" walkthrough (new) — trace one request through the stack
  • Adding REST API endpoints from a SysMod (existing; rename suggested)
  • File Download Protocol (OKTO)
  • File Upload Protocol (new)
  • OTA update flow end-to-end (new)

7. Devices and Buses

8. Helper Classes and Utilities

  • LEDPixels and LEDPixels JSON Configuration
  • RaftJson family (extend JSON or split): RaftJson, RaftJsonPrefixed, RaftJsonNVS, chained/cascading lookup
  • One-liner index for the rest: RingBuffer, ThreadSafeQueue, ExpressionEval, NumericalFilters, DebounceButton, MiniHDLC, NamedValueProvider, SupervisorStats, ConfigPinMap, StatusIndicator, DNSResolver, ArPreferences, libb64

9. REST API Reference

  • Built-in REST endpoints (rename suggested — see §10)
  • Subscription and publish topics (currently in RaftSysModAPI, almost empty)
  • Per-SysMod endpoint sections cross-linked from §5

10. Operations and Tooling

  • Remote Logging
  • Debug over IP (raft debug)
  • OTA in production
  • Performance and supervisor tuning

11. Reference / Index

  • All settings pages (replaces the empty RaftSysModSettings)
  • Full alphabetical page index (sidebar)

Naming and structural cleanup

Current page Issue Proposed action
RaftCoreAPI.md Name suggests a C++ API; content is actually a built-in REST endpoint table Rename to BuiltInRestEndpoints.md (or fold into a unified REST reference)
RaftRestAPIEndpoints.md Name suggests a reference list; content is a how-to-add guide Rename to AddingRestEndpoints.md
RaftSysModAPI.md Almost empty; only contains subscription Either complete (subscription + publishing endpoints) or merge into the REST reference
RaftFullAPIList.md Just two links; not a list Either flatten into a single canonical REST reference or rename to RestAPIIndex.md
RaftSysModSettings.md Literally [] Replace with a per-SysMod settings index
Inconsistent page-naming Mix of DeviceManager.md, I2CBus.md (bare) vs RaftBLEManagerSysMod.md (prefixed+suffixed) Pick one convention. Recommend <Name>SysMod.md for SysMod pages, bare names for concept pages
Home.md sidebar Many pages unreachable from Home New layered ToC (above) plus a _Sidebar.md

When renaming, leave a one-line stub at the old path with a link to the new page so external links don't 404.

Content gaps — pages to create

Priority 1 (blocks understanding of the framework):

  1. ✅ ProtocolExchange — role, RICREST element codes, file/stream session lifecycle, FW updater hookup, activity hook
  2. ✅ CommsChannels — CommsChannelManager, CommsCoreIF, registration, inbound/outbound flow, CommsChannelMsg, settings, bridges
  3. ✅ RICRESTProtocol — wire formats, element-code byte layouts, framing-per-transport guidance
  4. ✅ CommunicationsStackOverview — layered diagram and "Anatomy of a Request" walkthrough
  5. ✅ WritingYourFirstSysMod — end-to-end BlinkSysMod tutorial

Priority 2 (built-in SysMods with no docs):

  1. ✅ RaftWebServerSysMod — config, websockets, static files, certs endpoint (TLS internals and Netconn-vs-Sockets implementation note still optional follow-up)
  2. ✅ RaftMQTTManagerSysMod
  3. ✅ RaftFileManagerSysMod — includes a pointer to the upload endpoint
  4. ✅ RaftLogManagerSysMod — destinations (RaftRemote, Papertrail, Loki), levels
  5. ✅ RaftSerialConsoleSysMod
  6. ✅ RaftCommandSerialSysMod
  7. ✅ RaftCommandSocketSysMod
  8. ✅ RaftCommandFileSysMod
  9. ✅ RaftESPOTAUpdateSysMod
  10. ⏳ OTAUpdateFlow.md — dedicated end-to-end page across HTTP and RICREST. Material is currently spread across RaftESPOTAUpdateSysMod, FileDownloadProtocol and ProtocolExchange. Optional consolidation page.

Priority 3 (tutorials and architecture):

  1. ✅ AddingACommsChannel
  2. ✅ AddingAnI2CDeviceType — step-by-step tutorial using the HDC1080 as the worked example.
  3. ✅ Repository map — included as a table at the bottom of Home and as the source map at the bottom of RaftArchitecture. Standalone page not needed.
  4. ✅ RaftJsonAdvanced — RaftJsonPrefixed, RaftJsonNVS, change callbacks, cascading lookup.

Priority 4 (helpers and operations):

  1. ✅ RaftHelperClasses — one-line index of utility classes in RaftCore/components/core/*
  2. ⏳ WebUI build pipeline — not yet written. Touched on briefly in SysTypes; a dedicated page covering the parcel.js / npm flow, asset bundling and serving via WebServer is still missing.
  3. ⏳ Partitions and flash layout — not yet written. Partition tables, FS image, OTA partition selection.

Quality fixes for existing pages

  • ✅ PlatformIOArduinoQuickStart.md — refreshed; stale toolchain pins removed, redirects users to RaftCLI as primary path.
  • ✅ QuickStart.md — a fuller worked SysMod example is provided in WritingYourFirstSysMod, and a one-paragraph teaser at the end of QuickStart now points readers at the tutorial.
  • ✅ Configuration.md — expanded to cover the three layers (compiled defaults / SysType / NV-mutable), the chained RaftJson* wiring in RaftCoreApp, prefixed per-SysMod views, runtime updates and change callbacks. Cross-links JSON, SysTypes, RaftJsonAdvanced.
  • ✅ JSON.md — expanded to cover full constructor surface, all simple/array/key/type accessors, generation helpers, escaping rules, size/performance/NUL/thread-safety notes, with onward links to RaftJsonAdvanced.
  • ✅ SysMods.md — "interface overview" bullets now link each method to a focused page or section (RaftSysModAPI, JSON, RaftJsonAdvanced, RegisteringAsADataSource, ProtocolExchange, FileDownloadProtocol).
  • ✅ RegisteringAsADataSource.md — fully expanded with API, callback contracts, trigger / rate / minMs interaction, threading, and JSON + devbin worked examples.
  • ✅ RaftBLEManagerSysMod.md — fully rewritten with peripheral/central roles, GATT layout, named values, REST endpoints, and a Throughput-and-tuning section.
  • ✅ RaftNetworkManagerSysMod.md — incorrect "API for DeviceManager" cross-reference fixed.
  • ✅ DeviceFactoryAndClasses.md — reviewed; example was not actually truncated (initial misdiagnosis). No action needed.
  • ✅ RaftStatePublisherSysMod.md / RaftStatePublisherSettings.md — verified against the "no pre-configured pubList" behaviour; stale pubList references in subscription notes corrected.
  • ✅ RaftFullAPIList.md / RaftCoreAPI.md / RaftSysModAPI.md / RaftRestAPIEndpoints.md — H1 titles and intros clarified (full renames skipped to preserve external links).
  • ✅ Home.md — fully restructured into a 9-section layered ToC with repository map.
  • ✅ _Sidebar.md — created; reflects the new structure on every wiki page.

Promotion candidates from devdocs/ folders

Stable content currently buried in repo devdocs/ that deserves promotion (or at least a wiki summary linking back):

Source Suggested wiki destination Status
RaftCore/devdocs/devbin-devjson-publish-format-changes.md Append to / supersede part of Device Data Publishing ⏳ Pending
RaftCore/devdocs/bus-addr-status-lightweight-refactor-plan.md, bus-device-tracking-refactor-plan.md Internal — keep in devdocs/; surface user-visible behaviour in I2C Device Scanning ⏳ Pending
RaftCore/devdocs/ble-data-drop-investigation.md Internal; user-visible findings into BLEManager troubleshooting section ⏳ Pending
RaftI2C/devdocs/i2c-adaptive-yield-plan.md, i2c-bus-speed-change-investigation.md Surface tunables in I2C Bus; keep rationale in devdocs/ ⏳ Pending
RaftSysMods/devdocs/BLE-publish-throughput-analysis.md Add a "Throughput and tuning" subsection to BLEManager ⏳ Pending
RaftSysMods/devdocs/LoggerLoki_Implementation.md Linked from RaftLogManagerSysMod; not promoted in full ✅ Linked

Convention proposal: devdocs/ = working notes / decision records (kept in each repo). Wiki = stable user-facing docs. When content is promoted, leave a one-line pointer in the original devdocs/ file to the wiki page.

Execution plan (phased)

The work broke naturally into six phases. All six are now complete; remaining items are tracked in § Remaining work.

Phase A — Quick wins — ✅ Done

Phase B — Restructure navigation — ✅ Done

  • ✅ New Home with the 9-section layered ToC and repository map
  • ✅ New _Sidebar reflecting the structure on every page
  • ⏳ Renames of RaftCoreAPI.md, RaftRestAPIEndpoints.md, RaftFullAPIList.md, RaftSysModAPI.md were deliberately not done to preserve external links — H1 titles and intros were clarified instead. A future tidy-up rename map is recorded near the bottom of this document.

Phase C — Communications stack docs — ✅ Done

Phase D — Missing built-in SysMod pages — ✅ Done

Phase E — Tutorials and architecture — ✅ Done

Phase F — Helpers, ops, polish — ✅ Done

  • ✅ RaftHelperClasses index of utility classes
  • ✅ Final cross-link audit (internal targets verified; external URLs HTTP-checked — 23/24 return 200, the embedded.com link is bot-blocked but reachable from a browser)
  • ⏳ WebUI build pipeline page — not yet written (see remaining work)
  • ⏳ Partitions / flash layout page — not yet written (see remaining work)
  • ⏳ Promote selected devdocs/ content — LoggerLoki linked; the rest still pending

Maintainer decisions (recorded May 2026)

  1. Hierarchy. Move toward a hierarchical naming/sidebar layout. GitHub wiki URLs are flat, so hierarchy is expressed via (a) _Sidebar.md groupings (already done) and (b) an optional topic prefix on new filenames going forward. Existing well-known filenames are kept to preserve external links.
    • Recommended prefixes for new pages:
      • GettingStarted-*.md — tutorials, quick starts
      • Architecture-*.md — cross-cutting design pages
      • SysMod-*.md — one per built-in SysMod (future renames optional)
      • Comms-*.md — communications / protocol pages
      • Device-*.md — devices and buses
      • Helper-*.md — helper-class indexes
      • Reference-*.md — settings, API listings
    • Existing files retain their current names; if a future rename is unavoidable, add a stub redirect page at the old name pointing to the new one.
  2. External-link preservation. Strict — no mass renames. Only rename when the gain is large and add a stub at the old name when we do.
  3. Audience. App-developer is the primary audience. Pages should lead with how-to and configuration; internal/contributor detail goes later in the page or into devdocs/. (This guidance has been applied to the new pages added in Phases C–F.)
  4. Repository scope. Keep all docs consolidated under RaftCore.wiki. RaftSysMods, RaftI2C and RaftWebServer continue to use their devdocs/ folders for internal notes only.
  5. API churn. No planned API changes are blocking documentation; proceed against current main.

Tracking

This document lives at RaftCore.wiki/devdocs/documentation-improvement-plan.md and should be updated as phases progress.

Status as of May 2026: Phases A–F all complete. New pages added during the project follow the prefixed-naming convention only when newly created (Phases C–F pages were created before the naming policy was set; renaming them now would invalidate the freshly-built sidebar/Home links and would not yet have any external links to preserve, so they are left in place — see the rename log below if a future tidy-up is undertaken).

External-link audit (May 2026): all 24 outbound links verified with HTTP 200, except https://www.embedded.com/implementing-finite-state-machines-in-embedded-systems/ (referenced from SysMods.md) which returns 000 to curl due to bot-blocking but appears reachable from a real browser; left unchanged.

Remaining work

Grouped by priority. None of these block the framework being usable from the docs as they stand today; they are nice-to-haves and ongoing maintenance items.

High value

  1. Adding a new I2C device type — tutorial. ✅ Done — AddingAnI2CDeviceType (HDC1080 worked example, with addExtendedDeviceTypeRecord runtime path also covered).
  2. Advanced RaftJson page. ✅ Done — RaftJsonAdvanced (chaining, prefixed views, NVS persistence, change callbacks, generation helpers, immediate static accessors, memory/copy semantics).
  3. Expand Configuration. ✅ Done — three layers, RaftCoreApp wiring, prefixed per-SysMod views, runtime updates and change callbacks documented.
  4. Expand JSON. ✅ Done — full constructor surface, all simple/array/key/type accessors, generation helpers, escaping rules, size/performance/NUL/thread-safety notes.
  5. Expand RegisteringAsADataSource. ✅ Done — full coverage of API, message-generator and state-detector callbacks, trigger / rate / minMs interaction, threading and lifetime, JSON and devbin worked examples, plus tips/pitfalls and cross-links.
  6. Cross-link SysMods. ✅ Done — every method bullet in the interface overview now links to its focused page or section (RaftSysModAPI, JSON, RaftJsonAdvanced, RegisteringAsADataSource, ProtocolExchange, FileDownloadProtocol, etc.).

All High-value items are now complete.

Medium value

  1. OTAUpdateFlow consolidation page. ✅ Done — OTAUpdateFlow covers the HTTP-direct path, the file-stream path through ProtocolExchange + FileDownloadProtocol, shared invariants, partition layout, failure modes and diagnostics.
  2. Verify RaftStatePublisherSysMod / RaftStatePublisherSettings. ✅ Done — stale pubList references replaced; pages already document the "register dynamically + subscribe per channel" behaviour.
  3. Expand RaftBLEManagerSysMod. ✅ Done — full rewrite covering peripheral / central roles, GATT layout, comms-channel queueing, named values, and a Throughput-and-tuning section that distils the BLE-publish-throughput devdoc.
  4. WebUI build pipeline page. ✅ Done — WebUIBuildPipeline documents UI_SOURCE_PATH, GenWebUI.py, gzip + FS-image merge, dev workflow and troubleshooting.
  5. Partitions and flash layout page. ✅ Done — PartitionsAndFlashLayout covers OTA-capable vs single-app layouts, alignment rules, sizing, NVS encryption and rollback.

All Medium-value items are now complete.

Low value / housekeeping

  1. Promote remaining devdocs/ content per the table above (devbin format change notes → DeviceDataPublishing; I2C tunables → I2CBus; BLE data-drop notes → RaftBLEManagerSysMod).
  2. One-paragraph teaser at the end of QuickStart ✅ Done — final "Next: build your first SysMod" paragraph points at WritingYourFirstSysMod, Configuration, StatePublisher, SysTypes, RaftArchitecture and SysMods.
  3. Optional uniform rename pass using the rename map below — only worth doing if the wiki gains many more pages and the mixed naming becomes painful. Would require a stub-redirect page at every old name.

Optional future rename map (not executed)

If a uniform-prefix tidy-up is ever desired, the following mapping is recommended:

Current Proposed
QuickStart.md GettingStarted-QuickStart.md
PlatformIOArduinoQuickStart.md GettingStarted-PlatformIOArduino.md
WritingYourFirstSysMod.md GettingStarted-WritingYourFirstSysMod.md
AddingACommsChannel.md GettingStarted-AddingACommsChannel.md
RaftArchitecture.md Architecture-AtAGlance.md
RaftCLI.md Build-RaftCLI.md
SysTypes.md Build-SysTypes.md
TopLevelSysTypeConfig.md Build-TopLevelSysType.md
RaftBuildProcess.md Build-Process.md
LocalRaftLibraries.md Build-LocalDevLibraries.md
SysMods.md Architecture-SysMods.md
RaftSysManager.md Architecture-SysManager.md
Configuration.md Architecture-Configuration.md
JSON.md Architecture-RaftJson.md
CommunicationsStackOverview.md Comms-StackOverview.md
CommsChannels.md Comms-Channels.md
ProtocolExchange.md Comms-ProtocolExchange.md
RICRESTProtocol.md Comms-RICREST.md
RaftRestAPIEndpoints.md Comms-AddingRESTEndpoints.md
RaftCoreAPI.md Reference-BuiltinRESTEndpoints.md
RaftFullAPIList.md Reference-FullAPIList.md
FileDownloadProtocol.md Comms-FileDownload.md
RaftNetworkManagerSysMod.md SysMod-NetworkManager.md
RaftBLEManagerSysMod.md SysMod-BLEManager.md
RaftWebServerSysMod.md SysMod-WebServer.md
RaftMQTTManagerSysMod.md SysMod-MQTTManager.md
RaftFileManagerSysMod.md SysMod-FileManager.md
RaftLogManagerSysMod.md SysMod-LogManager.md
RaftSerialConsoleSysMod.md SysMod-SerialConsole.md
RaftCommandSerialSysMod.md SysMod-CommandSerial.md
RaftCommandSocketSysMod.md SysMod-CommandSocket.md
RaftCommandFileSysMod.md SysMod-CommandFile.md
RaftESPOTAUpdateSysMod.md SysMod-ESPOTAUpdate.md
RaftStatePublisherSysMod.md SysMod-StatePublisher.md
RaftRemoteLogging.md SysMod-RemoteLogging.md
RegisteringAsADataSource.md SysMod-DataSourceRegistration.md
RaftSysModSettings.md Reference-SysModSettings.md
RaftNetworkManagerSettings.md Reference-NetworkManagerSettings.md
RaftBLEManagerSettings.md Reference-BLEManagerSettings.md
RaftStatePublisherSettings.md Reference-StatePublisherSettings.md
RaftSysManagerSettings.md Reference-SysManagerSettings.md
DeviceManagerSettings.md Reference-DeviceManagerSettings.md
DeviceManager.md Device-Manager.md
DeviceManagerRESTAPI.md Device-ManagerRESTAPI.md
DeviceFactoryAndClasses.md Device-FactoryAndClasses.md
DeviceTypeRecordFormat.md Device-TypeRecordFormat.md
DeviceDataPublishing.md Device-DataPublishing.md
DataLogger.md Device-DataLogger.md
I2CBus.md Device-I2CBus.md
I2CDeviceScanning.md Device-I2CScanning.md
I2CDeviceIdentificationAndPolling.md Device-I2CIDPolling.md
RaftMotorControlOverview.md Device-MotorControlOverview.md
RaftMotorControlConfiguration.md Device-MotorControlConfig.md
RaftMotorControlCommands.md Device-MotorControlCommands.md
RaftCoreLEDPixels.md Helper-LEDPixels.md
LEDPixelsJSONConfig.md Helper-LEDPixelsJSON.md
RaftHelperClasses.md Helper-Index.md

Performing this rename would also require updating every internal [text](Name) link plus _Sidebar.md and Home.md, plus stub redirect pages at every old name to preserve any external blog/forum links pointing to the current names.

Home

Getting Started

Scaffolding & Building

Architecture

Built-in SysMods

Comms & Protocols

Devices & Buses

ROS 2

Helpers

Reference

Clone this wiki locally