Skip to content

Design Decisions

Jump to bottom
James Isnor edited this page Jul 6, 2026 · 5 revisions

Formulatel Design

This page talks about the decisions made when designing formulatel.

Overview

formulatel consists of three parts:

  1. ingest - the application responsible for receiving telemetry from a title, transforming it into the formulatel protobuf format, and queuing it for the backend.
  2. persist - the service responsible for writing telemetry from ingest into a persistent datastore.
  3. Visualization - the front-end component that displays data from the datastore and allows ad-hoc analysis.
---
config:
  theme: redux
  look: neo
title: formulatel
---
flowchart LR
  subgraph Sources [Sim Racing Games]
    T1[Title A]
    T2[Title B]
    TN[Title C]
  end
  Ingest[ingest<br/>Receive, Transform, Publish]
  subgraph PubSub [Pub/Sub Topics by Type]
    TypeA[Topic: Vehicle]
    TypeB[Topic: Motion]
    TypeN[Topic: Laps]
  end
  G[Grafana<br/>Visualization]
  T1 --> Ingest
  T2 --> Ingest
  TN --> Ingest
  Ingest --> TypeA
  Ingest --> TypeB
  Ingest --> TypeN
  Persist[persist<br/>Persistence Layer]
  DB[(Datastore)]
  Persist --> DB
  PubSub --> Persist
  PubSub <--Live Visualization--> G
  G <--Analysis--> DB
Loading

A Short View Back to the Past

The formulatel service was originally a project to help me learn about OpenTelemetry, so the first design focused on streaming data from the title with the intention of then adding observability on those streams and their data. The plan was to create an RPC per-title where ingest would translate the title's native format into a protocol buffer and send it to persist, and then have Grafana build the same dashboard for multiple titles, allowing the driver to see the same dashboard in different cars and giving them the ability to create ad-hoc visualizations of the telemetry.

block-beta
columns 3
tel>"telemetry data"]:3
space down1<[" "]>(down) space

block:ingest_servcice:3
    i1["ingest"]
    i2["ingest"]
    i3["ingest"]
end

space down2<[" "]>(down) space

block:rpc_service:3
    r1["rpc"]
    r2["rpc"]
    r3["rpc"]
end

space down3<[" "]>(down) space

block:otel_col:3
    otel1["otel-col"]
    otel2["otel-col"]
    otel3["otel-col"]
end

space down4<[" "]>(down) space

block:datastore:3
    influxdb
    prometheus
    opensearch
end

up1<[" "]>(up) up2<[" "]>(up) up3<[" "]>(up)

block:visualize:3
    grafana
end
Loading

I eventually realized that I didn't need an OpenTelemetry collector at all, and that I would need to use something like MQTT to make use of Grafana Live. That was a lot easier for me to wrap my head around, MQTT is very small and simple whereas OTcol is fairly complex, and it made writing the first title-implementation of ingest and persist easier to reason about.

Transport

formulatel uses MQTT to transport data from the driver's machine to persist. The motivating factor behind choosing MQTT was simply that our visualization layer, Grafana, requires such a stream for Grafana Live. That's also the reason we're using v3 instead of v5.

I decided on protocol buffers as the encoding format because it is:

  • flexible - support new titles and schema evolution of existing ones
  • efficient - variable-sized field encoding and a binary format make protobufs fast and scale up well

I imagined RPC streams being the "best" way for us to get data from ingest -> persist: we could enforce authentication and even setup oauth. The protobufs can also be compiled into many languages for drivers to write their own ingest for yet-to-be-supported titles. The main reason formulatel uses MQTT instead is due to the Grafana Live plugin landscape: there is already an MQTT plugin created for Grafana Live, and a live dashboard was the first major milestone after getting over some of the initial hurdles.

Future Plans

It would be interesting to build a streaming data plugin for either our (yet-to-be created) gRPC streams, MQTT v5, or another type of data stream.

Data Model

When I was deciding how to model and store vehicle telemetry, my primary considerations were extensibility efficient time-series querying. I had a pretty good idea from the beginning that I wanted to use protocol buffers to encode the telemetry, in no small part because I thought gRPC was going to be the best way to stream telemetry from ingest->persist, but also because protocol buffers can be compiled to many languages and I thought that was pretty neat, as it offered drivers a wider vector to extend to other titles.

As for how to model the data, I was conflicted between creating a protobuf for each title or trying to create a standard format. I originally thought it would be easiest to define every title's native format 1:1 and create a different message and rpc for each title, but after thinking about which goals I wanted to tackle first, I decided to forgo initially implementing multiple title support until I had something half-decent that was worth refining for multiple titles. Instead, formulatel defines a "root" GameTelemetry message, containing the fields present in all formulatel telemetry (timestamp, title, session_id, and user_id), as well as a subset of the F123 model whose fields are common enough to be expected in other sim-racing titles. The idea was that we'll use this sort of "general" format, and potentially even standardize vehicle telemetry across many titles, at the expense of being able to see every single sensor from every title.

There are clear drawbacks to this design, the most significant being that we're hoping most racing-sim titles will send the same data in the same way; i.e. the Speed of a vehicle will always be sent as kilometres per hour. ingest doesn't do any transformation of the data, it only repackages them as a protobuf message, and I wouldn't want to add any conditional title-specific calculation to standardize the units of a field. I decided to do this in Grafana because it is the only system that needs to care about units, and it maintained the design that Grafana would be where telemetry was parsed.

Another problem with this design is that some titles might not group the data described in the messages together. The F123 title packages Speed, RPM, Throttle, and Brake into one packet, so we can store those in a single row (timestamp) in the database that Grafana can visualize easily. Once we want to support a title that doesn't agree with this grouping, we'll end up with rows that only have partial data. That means we'll need to have another DB table to store the new grouping or our dashboards and drivers will need to do some fairly complex querying with a significant understanding of the title's telemetry format.

The good thing about this design is that it has allowed me to implement something that I can use as a driver, and as the only current user of formulatel, my opinion is very important. For now, this works well-enough for F123.

Future Plans

I'm not satisfied with the data model of formulatel and fully expect it to evolve. The data that we send, store, and query is largely based on a single title's format, and so was the schema of the database tables so that we could make DB writes as efficient as possible. In the future, I'm considering trying an OLAP datastore like ClickHouse and creating streams for a single sensor value. It turns out that the MQTT plugin for Grafana Live can parse JSON, but it can be a performance issue if it does too much, making our telemetry streams heavier than need be for the live visualization component. This would make supporting new titles straightforward and easy to extend, and it makes the database easier to reason about because each sensor gets its own column. On the other hand, using single-value streams would increase the number of connections to the broker more than tenfold, which could pose an interesting scaling challenge in the backend where persist is limited by open files, ports, and network bandwidth rather than CPU or memory.

Datastore

The criterion for a datastore boiled down to "write-efficient time-series that Grafana plays well with". Originally I wanted to provide a dashboard that could query "any telemetry" and thought I could use something like InfluxDB or OpenTSDB, but I quickly realized once I started reading their documentation that they wouldn't suit our use-case well at all: the former is fairly limiting with its query language, the latter much too large and heavy for this project.

formulatel doesn't use a time-series "native" datastore; instead, we opted for old reliable postgres and timescaleDB hypertables to efficiently store and query time-series data. The plan is to make use of the COPY protocol to efficiently batch telemetry from persist and tune the batch size and flush interval along with the CPU, memory, and network in order to size and scale persist and, eventually, the datastore.

Cloud Deployment

formulatel is deployed as a cloud service and can be reached at https://dashboard.formula.tel .

formulatel is deployed to the OCI cloud, making use of the generous Ampere free-tier (thus the ARM64 builds). The terraform directory contains the IaC used for this, as well as a setup-server.sh script that runs as part of the cloud-init when the VM is first provisioned, and a db-init.sql script that can be used to provision users and databases before the postgres instance is used by formulatel. Anybody should be able to deploy formulatel to their own cloud, completely for free.

The kubernetes directory contains the formulatel helm chart and a README.md with as much of the manual setup instructions as possible (e.g. creating secrets in kubernetes for the DB users we added with the db-init.sql script or installing cert-manager in a fresh cluster).

Infrastructure

The infrastructure for our free cloud deployment is predictably small and not at all production-like, using a single VM with 4 threads and 24GB memory to run kubernetes (k3s) and postgres. All of formulatel is deployed in kubernetes via the helm chart in kubernetes/charts/formulatel.

It's (perhaps obviously) not ideal to run our public services - Grafana and an MQTT broker - on the same instance that we are running postgres. Doing so means that we have to open ports on the PG instance that we normally would not need to, and generally databases are provisioned in their own cloud-specific hardware and services so that we don't need to think about iptables, pg_hba.conf, or any other server-management specific DB config. This is something to improve in the future with a larger budget or a free-er datastore.

Clone this wiki locally