Network Tables Protocol Specification, Version 4.0

A pub/sub WebSockets API for accessing NetworkTables.

Table of Contents

Motivation
References
Design
WebSockets Protocol Configuration
Supported Data Types
Properties
Text Data Frames
Binary Data Frames
Drawbacks
Alternatives
Trades
- JSON option for value updates (rejected)
- Timestamp format
Unresolved Questions

Motivation

Currently in NetworkTables there is no way to synchronize user value updates and NT update sweeps, and if user value updates occur more frequently than NT update sweeps, the intermediate values are lost. This prevents NetworkTables from being a viable transport layer for data logging at rates higher than the NetworkTables update rate (e.g. for capturing high frequency PID changes). While custom code can work around the second issue, it is more difficult to work around the first issue (unless full timestamps are also sent).

Adding built-in support for capturing and communicating all timestamped data value changes with minimal additional user code changes will make it much easier for inexperienced teams to get high resolution, accurate data to dashboard displays with the minimal possible bandwidth and airtime usage. Assuming the dashboard performs record and playback of NT updates, this also meets the desire to provide teams a robust data capture and playback mechanism.

Migration to a pub/sub style protocol layer where the client can request only its desired topics and the frequency in which it wants to receive value changes provides some mitigation for increased network utilization due to the data logging changes.

Using WebSockets (RFC 6455) as the transport layer along with JSON and MessagePack data encoding enables browser-based dashboards and other web technologies to easily access the protocol and minimizes the amount of custom wire protocol implementation required. Support for this protocol can easily be added in parallel to the existing NetworkTables protocol stack in order to maintain compatibility with older clients/servers and facilitate a gradual transition.

References

RFC 6455, The WebSocket Protocol, https://tools.ietf.org/html/rfc6455

RFC 7159, The JavaScript Object Notation (JSON) Data Interchange Format, https://tools.ietf.org/html/rfc7159

MessagePack, MessagePack Specification, https://github.com/msgpack/msgpack/blob/master/spec.md

Network Tables Protocol Specification, Version 3.0

Cristian’s algorithm, https://en.wikipedia.org/wiki/Cristian%27s_algorithm

Design

Both text and binary WebSocket frames are used for transport. Text frames are JSON, and binary frames are MessagePack. JSON is used for control messages for human readability and ease of debugging. MessagePack is used only for time and space efficient encoding of values, and the MessagePack subset used in this protocol is small enough that a custom encoder/decoder can be easily implemented for languages that lack good MessagePack libraries.

Consistent with pub/sub nomenclature, this spec uses the term "topic" for what was called an "entry" or "key" in the NT 3.0 spec.

MessagePack messages use numeric IDs for topics to reduce data size; these IDs have a 1:1 mapping to a full topic name (a string). This mapping is communicated via the JSON Topic Announcement Message (announce).

Topic Names

Topic have UTF-8 string names. Names starting with $ are reserved for use by the server (see Server-Published Meta Topics). Otherwise names are unrestricted, but convention is for them to start with / and be /-separated (without consecutive //) to make a Unix-path-like structure.

Timestamps

All MessagePack messages are timestamped. Timestamps are specified in unsigned integer microseconds—for robot programs, microseconds are preferred as the FPGA provides time in microseconds, and integers result in a more compact wire encoding (in typical robot use, integer microseconds timestamps will be half the size of double timestamps until about T=70 minutes, and in the worst case will be no larger).

Timestamps in messages always use the server time base. The server time base is synchronized across the network by the means described in this section (an implementation of Cristian’s algorithm).

Implementations should expose the actual timestamps for messages as well as provide methods to convert from server time base to local time to facilitate applications that prefer the server time base (e.g. a dashboard showing robot events might prefer to use server time instead of wall clock time).

The topic ID of -1 is reserved for timestamp communication. Clients shall periodically (e.g. every few seconds) send, in a manner that minimizes transmission delays, a MessagePack message with ID of -1, a timestamp of 0, and an implementation-selected type and data value (typically int or float 64) containing its (the client’s) current local time.

When the server receives an -1 ID message from a client, it shall respond to the client, in a manner that minimizes transmission delays, with a MessagePack message with ID=-1, a timestamp of its (the server’s) current local time (in microseconds), and the client-provided data value.

When the client receives an -1 ID message from the server, it shall compute the round trip time (RTT) from the delta between the message’s data value and the current local time. If the RTT is less than that from previous measurements, the client shall use the timestamp in the message plus ½ the RTT as the server time equivalent to the current local time, and use this equivalence to compute server time base timestamps from local time for future messages.

Due to the fact there can be multiple publishers for a single topic and unpredictable network delays / clock drift, there is no global total order for timestamps either globally or on a per-topic basis. While single publishers for real-time data will be the norm, and in that case the timestamps will usually be in order, applications that use timestamps need to be able to handle out-of-order timestamps.

Caching and Reconnection Handling

Servers shall keep a retained value for each topic for the purposes of both Get Values Message (getvalues) and Subscribe Message (subscribe) requests; the retained value shall be the value in the largest timestamp (greater-than or equal-to) message received for that topic. This retained value is deleted if the topic is deleted (e.g. there are no more publishers).

Clients may similarly keep a retained value for each topic for ease of use by user code. If this is done, this retained value shall be updated by both locally published values and received messages for that topic with greater-than/equal-to timestamps, and the retained value shall be deleted when a Topic Removed Message (unannounce) is received.

Clients should support a "set default" operation for a topic. This is a "weak" value update that publishes a value with a timestamp of 0 (thereby not causing the retained value of the server or other clients to be updated if they have a current value update with a timestamp > 0). Typically the "set default" operation should also result in the retained property being set on the topic.

Clients may accept application commands to publish and subscribe while disconnected. If a client does so, in addition to maintaining a retained value as described above, it must keep track for each application-published topic whether any of the locally published values were "strong" (via a "set" operation), or all of them were "weak" (via a "set default" operation). While disconnected, there is no reference clock; "strong" timestamps shall be set to 1 and "weak" timestamps shall be set to 0.

When the client disconnects, the client shall delete any topics that are not published by the application and do not have the retained or persistent properties set. The client shall also reset the application-published retained value timestamps to 0 and 1 as per the previous paragraph.

When the connection to the server is established (either reconnect or initial connection), the client shall publish and send only the retained values to the server that are in application-published topics (those with timestamps of 0 and 1, per above). Only the values with timestamp 0 may be sent immediately upon reconnection. The values with timestamp 1 must wait until the client clock is synchronized with the server clock; the timestamps for these values when sent to the server must be either the current server time or, if possible, an estimation of server time when the values were actually written.

Note: the previous paragraphs enable offline, multi-publisher operation under network/server reboot conditions without creating zombie topics, assuming clients use "set default" and the retained property appropriately. This is achieved mainly via the use of timestamps 0 and 1 to enable tie breaks such that normally-set values (timestamp X) are used in preference to retained values (timestamp 1), and retained values are used in preference to weakly set values (timestamp 0). An example use case is as follows:

Server starts
Dashboard client connects
Coprocessor client connects
Coprocessor client publishes configuration topic, sends an initial value using "set default", and subscribes to the topic (to detect configuration changes)
Dashboard client sees configuration topic published and subscribes to it
Dashboard user changes configuration value—dashboard client publishes to the topic and sends the user value
Coprocessor receives the user value and updates its retained value
Server reboots (this also disconnects the dashboard and coprocessor clients)
If the dashboard reconnects first:
- The user value was published and cached (retained value) on the dashboard client, so the dashboard client re-publishes and sends the cached data with timestamp 1.
- The coprocessor client reconnects later. It also published and cached, but it only ever called "set default" and sends the cached data (which is also the user value) with timestamp 0. It receives the retained value from the server with timestamp 1, and updates locally.
- The server propagates the timestamp 0 message, but since it has a retained value with timestamp 1, as do other clients, the retained value is not updated and the user value remains active.
If the coprocessor reconnects first:
- The coprocessor client only ever called "set default", so it sends the cached data (the user-set value) with timestamp 0.
- If the dashboard never reconnects, no new values are published, so the user-set value is active
- If the dashboard reconnects, it sends a message with timestamp 1 ("strong" set). This propagates but does not change the value (it’s the same user-set as before).
If the dashboard updates the value while offline, it’s still a "strong set" and wins the tie

A second use case:

Server starts
Dashboard client connects
Dashboard client publishes configuration value with the retained property set, and publishes an initial value
Dashboard client disconnects. The topic is not deleted on the server because the retained property is set.
Coprocessor client connects
Server sends announce message for topic and sends retained value (with timestamp 1)
Coprocessor client publishes, uses "set default", and subscribes to the topic. Since "set default" uses a timestamp of 0, it loses to the retained value with timestamp 1, and the coprocessor subscriber will see the value previously set by the dashboard.
Dashboard client reconnects

Server Behavior

Topic IDs may be common across all client connections or be connection-specific. If they are common, the server needs to be careful regarding topic ID reuse due to deleted topics, as the protocol provides no way to change a client topic ID. Requests (e.g. subscribe or publish) are always specific to a single client connection.

When a client initially connects, the server shall send it announce messages for each existing topic.

The server shall keep a publisher count for each topic. Persistent and retained topics have an additional implicit publisher. When the publisher count reaches zero (which only happens for non-persistent and non-retained topics), the server shall delete the topic (including its retained value). When a client connection is lost, the server shall handle that as an implicit unpublish for all topics currently published by that client.

The server may operate in-process to an application (e.g. a robot program). In this case, the application operationally behaves like a client (e.g. it sends publish requests and receives topic announcements), but of course does not need to estimate delta time, create JSON/MessagePack messages, etc, as all of the necessary operations can be performed programmatically within the same process.

Client Behavior

Clients are responsible for keeping server connections established (e.g. via retries when a connection is lost). Topic IDs must be treated as connection-specific; if the connection to the server is lost, the client is responsible for sending new publish and subscribe messages as required for the application when a new connection is established, and not using old topic IDs, but rather waiting for new announce messages to be received.

Except for offline-published values with timestamps of 0, the client shall not send any other published values to the server until its clock is synchronized with the server per the Timestamps section.

Server-Published Meta Topics

The server shall publish a standard set of topics with information about server state. Clients may subscribe to these topics for diagnostics purposes. These topics are hidden—they are only announced to clients that have subscribed to them.

Topic Name	Data Type	Description
`$clients`	`msgpack`	Connected clients
`$clientsub$<client>`	`msgpack`	Client `<client>` subscriptions
`$serversub`	`msgpack`	Server subscriptions
`$sub$<topic>`	`msgpack`	Subscriptions to `<topic>`
`$clientpub$<client>`	`msgpack`	Client `<client>` publishers
`$serverpub`	`msgpack`	Server publishers
`$pub$<topic>`	`msgpack`	Publishers to `<topic>`