Understanding EOS and Sysdb

An introduction to EOS's architecture

EOS is composed of a number of processes, called "agents," each performing one specific role. These processes are just regular Linux processes. Even the "ASIC drivers", which manage the hardware chips that actually process packets, are userland processes. No interesting logic is implemented in the kernel, because tampering with the kernel significantly increases the likelihood of in-kernel bugs, which are generally not very forgiving.

Now all those processes need to work in concert so that, for instance, when a port goes down, the LED for that port on the front panel gets switched off. So the "LED Agent" process needs to know whenever a port goes up or down, which is something managed by the "ASIC Driver" process.

A Video Introduction

A Written Description

First, the wrong way to solve inter-process communication

Having one part of the system (e.g. the ASIC Driver) explicitly call another part of the system (e.g. the LED Driver) to alert it that something changed is one solution to the problem:

However, this quickly leads to unmaintainable spaghetti code, inconsistent APIs between the various agents, and complex dependencies regarding which agent should call which other agent under what circumstances. Writing software this way is not only tedious and error prone, it also just doesn’t scale as the number of agents keeps increasing.

This sort of software also is much worse at fault containment. Processes that are slow or buggy cause hangs and crashes in entirely unrelated agents. For example, if the Routing agent is busy processing updates from BGP, static route updates from the CLI may be stuck in a queue waiting to be handled. If one of the processes crash, that user's configuration is lost. Attempts to fix this with more intricate message passing only intertwines the two distinct processes even further and leads to a less reliable OS; meaning a crash in process A will cause a crash in process B, and a delay in process X will cause a hang in process Y. Because switches maintain very complex state machines, the web of process dependencies is dense and intertwined, and these cascading failures can completely disrupt operation of the device.

EOS's architecture

Enter Sysdb, a small in-memory system that’s somewhat like a database, or a file system, or a message bus, depending on how you see it. All the state that needs to be acted on by agents is stored in Sysdb, and each agent materializes a view of a subset of Sysdb that is relevant to its activities. For instance, the LED Driver cares about the status of the ports, but doesn’t care about the routing table, therefore it loads only the portion of the state hierarchy that contains the interface statuses.

The goal of Sysdb is to synchronize state across the different agents, so that the LED agent can receive a notification when ASIC driver marks an interface as "down". This process works in the following steps:

1. The ASIC driver notices that, say, Ethernet1's transceiver has been removed.
2. It then updates its local state hierarchy, setting interface/status/Ethernet1/oper_status to not connected
3. The underlying agent infrastructure enqueues a state update message on its connection to Sysdb.
4. Sysdb reads the state update off of its end of the connection and updates its state hierarchy appropriately. At this point, both Sysdb and the ASIC driver know that Ethernet1 is not connected
5. Sysdb sends this state update to every other agent who has registered interest in that state update.
6. The LED agent, which registered interest in interface state at start-up, reads the update off of its connection to Sysdb and updates its copy of the interface state hierarchy.
7. This triggers a callback so the LED agent can switch off the little light, indicating that signal on Ethernet1 was lost.

State update notifications are thus propagated across the system in an eventually consistent fashion, meaning that when one agent updates its own data, the rest of the system will get the memo “eventually” (within a certain bound of time) and ultimately everybody should converge and be on the same page. However, all of these updates happen extremely quickly, so the LED agent is notified of the new state in fractions of a moment after the ASIC driver notices that the transceiver was removed. However, this process also allows the system to operate at scale. If any agent gets wedged, crashes, or just operates inefficiently, no other process is affected. Instead, the agent is quietly restarted (thanks to ProcMgr, a simple daemon on EOS which monitors agents' health), and resumes normal operation without any other agent noticing.

This state update notification mechanism is at the heart of all the software we write, because all of the interesting “business logic” happens as a side effect of some piece of state changing. The LED Driver switches the LED off as a result of handling a state update notifying it that the port went down, not because anything else in the system otherwise called or poked the LED Driver in any explicit way. In other words, the state is the API. Everything happens as a side effect of state changing.

Storing all the state in Sysdb also brings in another significant advantage: when an agent crashes or otherwise restarts, most of its state is preserved, by virtue of being in Sysdb. When the agent starts up again, it will reconnect to Sysdb, and retrieve its state however it was left in Sysdb before restarting. Additionally this restart is completely invisible to any other agent, since no two agents communicate directly. This means that agent crashes are significantly less severe, and that the software encoding the agent's logic can be seamlessly updated with no downtime.

Now that you have a better understanding of EOS's architecture, go ahead and read about an EOS SDK agent's lifecycle