Skip to content

Commit

Permalink
State reconciliation with dependency graphs
Browse files Browse the repository at this point in the history 
This commit adds packages "depgraph" and "reconciler" under EVE/libs, implementing
a dependency graph [1] and a state reconciliation algorithm, respectively.
It will be used in EVE to generically solve the problem of reconciliation
between the actual/current (configuration) state and the latest intended/desired
state. Every time the desired state changes, a certain sequence of Create, Modify
and Delete operations has to be scheduled and executed to update the current
state accordingly. The order of operations often matters as it has to respect
all dependencies that exist between configuration items.
The plan is to first refactor NIM and use dependency graph and reconciler for
configuration items that device networks are built with.
Based on how NIM refactoring goes, this solution could be used for zedrouter next.

For more detailed reasoning and motivation behind depgraph+reconciler, please refer
to the proposal presented on the LF Edge wiki [2].

[1]: https://en.wikipedia.org/wiki/Dependency_graph
[2]: https://wiki.lfedge.org/display/EVE/Dependency+graph+for+configuration+items

Signed-off-by: Milan Lenco <milan@zededa.com>
  • Loading branch information
@milan-zededa
milan-zededa committed Jan 27, 2022
1 parent 100c3af commit 76bb8fa
Show file tree
Hide file tree
Showing 34 changed files with 9,120 additions and 0 deletions.
300 changes: 300 additions & 0 deletions libs/depgraph/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,300 @@
# Dependency Graph (depgraph)

## Use-case

This package implements a [dependency graph](https://en.wikipedia.org/wiki/Dependency_graph).
The main use-case is to represent configuration items (network interfaces, routes,
volumes, etc.) or any managed stateful objects (incl. processes, containers, files, etc.)
as graph nodes, here called *items*, and their dependencies as directed graph edges.

For example, if there are items `A` and `B` with edge (dependency) `A->B`,
it means that `B` should be created before `A`. Conversely, the removal of these
two items should proceed in the opposite order, i.e. `A` should be removed first
(think of the dependency as "`A` cannot exist without `B`"). Edges of this dependency
graph slightly diverge from the standard definition as it is allowed for an edge
to point to an item which is not present in the graph, representing a scenario
of a missing dependency.

The graph can be for example used to model the intended or the current state
of a managed system. Note that depgraph alone only provides a data structure
to store the modeled state. However, in combination with the [Reconciler](../reconciler/README.md),
it can help to solve the challenge of state reconciliation. A management
agent will typically maintain two graphs, one for the intended state, updated
based on the input from a user/controller, and the other for the current state.
The agent will use APIs of the managed system to learn the actual state
and will update the graph accordingly. The reconciler will take both of these
graphs as an input and will perform all state transitions needed to reach
the intended state, updating the graph representing the current state
in the process. For more information on the topic of state reconciliation,
please refer to the [readme file of the Reconciler](../reconciler/README.md).

## Subgraphs

Apart from items and edges, the graph also supports a notion of subgraphs.
A subgraph is a subset of graph items, including all edges that originate or point
from/to any of these items (again, a slight deviation from the standard definition).
Subgraph is given a name and optionally also a description (just like the top-level
graph). The main use-case is to group related items and allow to select and edit
them together.

Here is a visual depiction of an example graph with some subgraphs:

![Subgraphs](./pics/subgraphs.png)

For example, all components of a virtualized network (bridge, routes, dns server, etc.)
can be grouped into one subgraph with the logical name of the network:

```go
virtNetwork := depgraph.New(
depgraph.InitArgs{
Name: virtNetworkName,
Items: []depgraph.Item{bridge, route1, route2, dnsServer}
})
```

Then, to add or replace all the components of the network in the graph, only one function
call is needed:

```go
fullStateGraph.PutSubGraph(virtNetwork)
```

Also, the entire content of the subgraph can be removed with just:

```go
fullStateGraph.DelSubGraph(virtNetworkName)
```

Subgraphs can be also nested and thus compose a hierarchical tree structure.
This is very similar to directory structure of a filesystem if you think of subgraphs
as directories and items as files.
Currently, subgraphs are not related to and does not affect dependencies.

Note that in terms of API, subgraph is also a graph - it implements the `Graph`
interface. Single item can also be viewed as a subgraph - `Graph.ItemAsSubGraph(item)`
returns a (read-only) graph handle.

## API & Usage

Configuration items modeled as graph nodes (in API simply called items) should
implement the `Item` interface.
This means that for every distinct item type, there needs to be a structure
with methods as required by the interface. For example, it is required to provide
a name for every item instance (based on the item configuration) through the method
`Name() string`. Two distinct items of the same type should have different names.
Distinct here means that the manifestation of these items are two separate objects
in the managed system. A graph-wide unique item identifier/reference is therefore
a combination of the type (returned by `Type() string`) and the name. This is captured
by the `ItemRef` structure. To obtain a reference for an item (which is needed for
several graph methods), call `Reference(Item)`.
Another notable method defined by the `Item` interface is `Dependencies()`. It lists
all the dependencies of an item and therefore determines the outgoing edges of the item.

Here is a simplified example for Items representing Linux interfaces and routes.
Only `Name()`, `Type()` and `Dependencies()` methods are shown here. For other
required methods, please see the `Item` interface definition.

```go
import "github.com/lf-edge/eve/libs/depgraph"

type LinuxInterface struct {
name string
}

func (intf LinuxInterface) Name() string {
return intf.name
}

func (intf LinuxInterface) Type() string {
return "Linux interface"
}

func (intf LinuxInterface) Dependencies() []depgraph.Dependency {
// no dependencies
return nil
}

// Other Item methods for LinuxInterface are omitted.

type LinuxRoute struct {
via string
dst string
metric int
}

func (route LinuxRoute) Name() string {
// Both dst and via should be included to uniquely identify the route
// among all the routes.
return route.dst + " via " + route.via
}

func (route LinuxRoute) Type() string {
return "Linux route"
}

func (route LinuxRoute) Dependencies() []depgraph.Dependency {
return []depgraph.Dependency{
{
Item: depgraph.RequiredItem{
Type: "Linux interface", // can also use LinuxInterface{}.Type()
Name: route.via, // can also use LinuxInterface{name: route.via}.Name()
},
Description: "Route requires outgoing interface to be configured first",
},
}
}

// Other Item methods for LinuxRoute are omitted.
```

With each item instance the graph also allows to store some state data, implementing
the `ItemState` interface. This can be for example used to store information about the last
state transition performed for the item and any errors that resulted from it.
Think of `Item` as the configuration and `ItemState` as a run-time state.
State data are mostly transparent to the graph with only few methods defined to help to enhance
the [visualization of the graph](#visualization).
Item state data are completely optional and can be passed as nil. They are typically omitted
when the graph is used to model the intended state and provided when the currently running
state is modeled.

Once all needed `Item` (and possibly `ItemState`) implementations are available, you are ready
to build a dependency graph to model some kind of system state.
Graph with some initial content is created using `New()`:

```go
import "github.com/lf-edge/eve/libs/depgraph"

// using LinuxRoute and LinuxInterface defined above

g := depgraph.New(depgraph.InitArgs{
Name: "MyGraph",
Description: "This is my example graph",
ItemsWithState: []ItemWithState{
{Item: LinuxInterface{name: "eth0"}, State: LinuxInterfaceState{lastOpErr: nil}},
},
// For items without state data you can use Items, avoiding to pass ItemState as nil
// and therefore making the code shorter and easier to read.
// But do not put the same Item into both Items and ItemsWithState.
Items: []Item{
LinuxRoute{via: "eth0", dst: "10.10.0.0/16", metric: 10},
LinuxRoute{via: "eth0", dst: "192.168.16.0/24", metric: 10},
}
})
```

Graph will automatically build edges for all items based on their dependencies.
In the example above, there will be one edge from each route pointing to the `eth0` interface.
`InitArgs` can also contain the initial content of subgraphs.

A single item is manipulated using `Item()`, `PutItem()` and `DelItem()` methods:

```go
import (
"fmt"
"github.com/lf-edge/eve/libs/depgraph"
)

// Add new Linux route (nil ItemState).
item := LinuxRoute{via: "eth0", dst: "10.20.0.0/16", metric: 100}
g.PutItem(item, nil)

// Update the item.
item.routeMetric = 50
g.PutItem(item, nil)

// Delete the item.
itemRef := Reference(item)
g.DelItem(itemRef)

// Read single graph item.
item, found := g.Item(itemRef)
if found {
fmt.Printf("Linux interface %+v\n", item)
}
```

To iterate all items in the graph:

```go
inclSubGraphs := true
iter := g.Items(inclSubGraphs)
for iter.Next() {
item, state := iter.Item()
fmt.Printf("Item: %+v, with state: %+v\n", item, state)
}
```

To iterate all edges originating from an item:

```go
iter := g.OutgoingEdges(itemRef)
for iter.Next() {
e := iter.Edge()
fmt.Printf("Edge from %s to %s for dep: %+v\n",
e.FromItem, e.ToItem, e.Dependency)
}
```

Lastly, sub-graphs are manipulated using `SubGraph()`, `PutSubGraph()`, `DelSubGraph()`
and `EditSubGraph()` methods:

```go
item1 := LinuxInterface{name: "eth0.1"}
item2 := LinuxInterface{name: "eth0.2"}
subG := depgraph.New(depgraph.InitArgs{
Name: "MySubGraph",
Description: "This is my example sub-graph",
Items: []Items{item1, item2},
})

// Add new subgraph
g.PutSubGraph(subG)

// Can use the same handle to edit the subgraph...
subG.DelItem(Reference(item1))

// ... or a read-write handle can be retrieved:
readSubG := g.SubGraph("MySubGraph") // can only read nodes, edges, etc.
subG = g.EditSubGraph(readSubG) // elevate to read-write
subG.Putitem(item1, nil)

// Remove the subgraph:
g.DelSubGraph("MySubGraph")
```

## Visualization

The graph content can be exported into [DOT](https://en.wikipedia.org/wiki/DOT_(graph_description_language))
using `DotExporter` and then visualized for example using [Graphviz](https://graphviz.org/).
Subgraphs are drawn as [clusters](https://graphviz.org/Gallery/directed/cluster.html),
i.e. items they contain are plotted near each other and contained within a rectangle.

Example usage (incl. Graphviz with a wrapper for Go):

```go
// import "github.com/goccy/go-graphviz"

// Initialize DOT exporter
exporter := &DotExporter{CheckDeps: true}

// Render DOT representation of the dependency graph.
dot, err := exporter.Export(graph)
if err != nil {
log.Fatalf("depgraph DOT rendering failed: %v", err)
}

// Use go-graphviz - a Graphviz wrapper for Go.
gvizGraph, err := graphviz.ParseBytes([]byte(dot))
if err != nil {
log.Fatalf("failed to parse DOT: %v", err)
}
gviz := graphviz.New()
err = gviz.RenderFilename(gvizGraph, graphviz.PNG, "/path/to/graph.png")
if err != nil {
log.Fatal(err)
}
```

Example of a rendered depgraph:

![graph visualization example](./pics/graph-example.png)

0 comments on commit 76bb8fa

Please sign in to comment.