Skip to content

STARDOS Architecture

Jump to bottom
Richard Snider edited this page Sep 22, 2023 · 5 revisions

Purpose

The purpose of this document is to detail the software structures that make up a STARDOS system. It includes details on ROS2 topic and namespace names, configuration formats, and other key details necessary for implementing STARDOS software.

Terminology

This document will use the following terminology specific to the STARDOS system:

  • Sensor -- A source of data, accessed via a node. Usually a piece of hardware.
  • Node -- A STARDOS modular program.
  • Coprocessor -- A monolithic program that interfaces with the rest of the STARDOS network via a node, either as a separate process or integrated into the coprocessor. Not technically STARDOS software.
  • Computer -- A single computer, as commonly understood, with nodes and coprocessors installed and running.
  • System -- A STARDOS system is a collection of computers that should be grouped logically, and share a root ROS2 namespace. Examples:
    • An aircraft, for example Phoenix: /phoenix
    • The ground station: /gcs
  • Network -- A STARDOS network is a collection of systems that interact through a ROS2 network and share a root namespace.
  • Processing Chain -- A series of linked nodes, beginning with a data source node and ending with a data sink node, containing any number of data processing nodes, through which data flows and is processed.

Topology

What follows is a graph of the network topology that STARDOS uses to communicate.

STARDOS Topology

ROS2 Namespace and Node Structure

Every STARDOS system will have a top-level ROS2 namespace. For vehicles, this should be the name of the vehicle in snake_case. For systems of which there can only be one (such as the ground station), the namespace will simply be named for the type of system it represents (such as /ground for the ground station).

Becuase ROS does not support hyphens in namespace and node names, hyphens in config files are converted into underscores

An example complete namespace structure for the reference config example is as follows. Nodes are highlighted in green, and topics are highlighted in red:

/phoenix/
- ├─ heartbeat
- ├─ attitude
- ├─ gps
  │  ├─ phoenix_copilot/
+ │  ├─ control
+ │  ├─ system_status
+ │  ├─ logging
+ │  ├─ mavlink_copilot
- │  └─ system_status
  └─ APL-004/
+    ├─ control
+    ├─ system_status
+    ├─ logging
-    ├─ system_status
-    ├─ set_config
-    ├─ start_mission
-    ├─ end_mission
     └─ main/
+       ├─ timer_capture_group
-       ├─ activate
-       ├─ deactivate
-       ├─ capture
-       ├─ secondary
        └─ lucam0/
+          ├─ lucam_single
-          │  ├─ heartbeat
-          │  └─ data
+          ├─ jack
-          │  ├─ heartbeat
-          │  └─ data
+          ├─ hammer
-          │  ├─ heartbeat
-          │  └─ data
+          └─ write_to_disk
-             ├─ heartbeat
-             └─ data

STARDOS Nodes

A STARDOS network is composed of nodes which perform various tasks, from capturing and processing data, to controlling other nodes. This makes STARDOS a modular and easily extended system; each data source (usually a physical sensor) has its own node controlling it, making the addition of new data sources require zero modification of existing code.

The Control Node

The Control node is the most important node of a STARDOS computer. Every STARDOS computer will have an instance of the Control node running. In many respects, it functions as the init system of STARDOS. It is started by the computer's service manager (likely systemd), and is primarily what makes a STARDOS computer a STARDOS computer.

Capture Group Nodes

A capture group node manages a set of data source nodes, telling them when to capture. The simplest form runs on a timer, instructing data source nodes to capture at a fixed interval while above a certain altitude. More complex forms might have different operations.

Data Source Nodes

A data source node collects data and metadata from a sensor and makes it available to other nodes on the same system.

Data Processing Nodes

A data processing node receives data from other nodes on the same system, performs some transformation operation, and makes the resulting data available to other nodes on the same system. These are a type of pipeline node.

Data Sink Nodes

A data sink node receives data from other nodes on the same system* and removes it from the control of STARDOS. The most basic forms of this are writing it to disk, or deleting it. Additionally, many interface nodes for coprocessors* will be data sink nodes. These are a type of pipeline node.

Datalink Node

The datalink node sits at the logical "edge" of a system* and enables inter-system communication. If running on an aircraft, it also relays certain autopilot messages needed by the other nodes on the system.

Intra-System Communication

STARDOS nodes communicate within a system via ROS 2 using a standardized language, as detailed below.

Data Messages

Data messages are the primary way that nodes communicate with each other. They carry data throughout a processing chain

string[] content;     // A list of data or paths to the data in temporary storage
string metadata;      // The metadata associated with the data, encoded as JSON
int64_t collected_at; // 64-bit timestamp containing the time the data was collected
uint32_t index;       // 32-bit mission-local and source-local index

Control Messages

Control messages are a secondary means of communication for nodes. They are sent to instruct a node to do something, or change its mode of operation. The most common use is for a capture group node to instruct its data source nodes to capture, however data source nodes may implement a secondary operation that capture groups may instruct them to perform at certain times.

Generally, the presense of a control message on a specific topic should be sufficient information to act on, but in the event that it is not, the control message specification includes a string field to convey additional information.

string options;

Heartbeat Messages

Heartbeat messages are sent on a timer -- 1 second is ideal -- and provide information about the status of the node that sends them. They should always be logged.

int8_t state;         // The state integer of the node sending the heartbeat. Negative numbers are error states.
uint32_t errors;      // Error bitmask, with the errors being defined by the node
uint16_t requests;    // Number of default operation requests this node has received
uint16_t failures;    // Number of operations this node has failed to complete
uint16_t performance; // Processing time in milliseconds of most-recently-completed operation
uint8_t queue_length; // How many operations are in this node's processing queue
uint8_t[16] data;     // Arbitrary data fields defined in the node's definition file

Autopilot Relay Messages

Autopilot relay messages are sent by the datalink node at the logical edge of the system. They contain flight telemetry of interest to various nodes. The format will depend on the autopilot message being relayed, check MAVLink documentation for details.

Inter-System Communication

STARDOS systems communicate within a network via a custom low-level protocol carried via a MAVLink messages. Because of the limited bandwidth of this protocol (we should limit ourselves to around 1.625 Kbps, or 1 full-sized MAVLink message per second), only heartbeat messages and necessary control messages will be sent through this link.

Directory Structure

Installation

STARDOS software should be installed to /opt/stardos/. This should contain the following:

  • workspace/ -- ROS2 workspace directory
  • data/ -- Mount point for persistent data drive (removable)
  • tmp/ -- Mount point for temporary data filesystem (RAM disk)
  • scripts/ -- Additional STARDOS scripts
  • services/ -- SystemD service definition files
  • files/ -- Additional STARDOS configuration files
  • computer.json -- A configuration file identifying the computer to STARDOS.

Data directory

STARDOS persistent data is stored under /opt/stardos/datasets/. This directory should be a mount point, mounting an external drive connected via USB >= 10Gbps (USB 3.1 / USB 3.2 gen 2).

A mission-unique directory will be created on that drive, and a bind mount pointing to it will be created at /opt/stardos/data. This bind mount will be removed on mission shutdown.

Runtime directory

STARDOS temporary runtime data is stored under /opt/stardos/tmp/. A tmpfs filesystem will be mounted here by the control node, with a size of 128 MiB per node plus a flat 1 GiB, when a mission is started. It will be unmounted automatically on mission shutdown, and any data left in it will be copied to /opt/stardos/data/tmp_backup.

In the event that a payload consists of multiple computers, only a single tmpfs filesystem should be created on the primary payload computer. Other computers should then mount that directory using NFS. When using such an architecture, it will be important to keep in mind the speed limitations of Ethernet, and only offload tasks which will see a benefit running on separate hardware in excess of the penalty incurred by involving a network. Generally, it is advisable to avoid running image capture and processing on a separate system, but more lightweight data such as atmospheric sensors or light detectors should be fine.

Clone this wiki locally