Skip to content

Commit

Permalink
Merge pull request #593 from srl-labs/ceos-postdep-scrapli
Browse files Browse the repository at this point in the history
use scrapligo for ceos mgmt provisioning
  • Loading branch information
hellt committed Aug 23, 2021
2 parents 7162e33 + 8e43cbe commit e562d6f
Show file tree
Hide file tree
Showing 7 changed files with 118 additions and 79 deletions.
16 changes: 1 addition & 15 deletions clab/clab.go
Original file line number Diff line number Diff line change
Expand Up @@ -279,21 +279,7 @@ func (c *CLab) CreateLinks(ctx context.Context, workers uint, postdeploy bool) {
}

for _, link := range c.Links {
// skip the links of ceos kind
// ceos containers need to be restarted in the postdeploy stage, thus their data links
// will get recreated after post-deploy stage
if !postdeploy {
if link.A.Node.Kind == "ceos" || link.B.Node.Kind == "ceos" {
continue
}
linksChan <- link
} else {
// postdeploy stage
// create ceos links that were skipped during original links creation
if link.A.Node.Kind == "ceos" || link.B.Node.Kind == "ceos" {
linksChan <- link
}
}
linksChan <- link
}
// close channel to terminate the workers
close(linksChan)
Expand Down
3 changes: 0 additions & 3 deletions cmd/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -179,9 +179,6 @@ var deployCmd = &cobra.Command{
}
}

// run links postdeploy creation (ceos links creation)
c.CreateLinks(ctx, linkWorkers, true)

log.Info("Adding containerlab host entries to /etc/hosts file")
err = clab.AppendHostsFileEntries(containers, c.Config.Name)
if err != nil {
Expand Down
26 changes: 4 additions & 22 deletions docs/manual/kinds/ceos.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,7 @@ topology:
## Features and options
### Node configuration
cEOS nodes have a dedicated [`config`](../conf-artifacts.md#identifying-a-lab-directory) directory that is used to persist the configuration of the node. It is possible to launch nodes of `ceos` kind with a basic config or to provide a custom config file that will be
used as a startup config instead.
cEOS nodes have a dedicated [`config`](../conf-artifacts.md#identifying-a-lab-directory) directory that is used to persist the configuration of the node. It is possible to launch nodes of `ceos` kind with a basic config or to provide a custom config file that will be used as a startup config instead.

#### Default node configuration
When a node is defined without `startup-config` statement present, containerlab will generate an empty config from [this template](https://github.com/srl-labs/containerlab/blob/master/nodes/ceos/ceos.cfg) and copy it to the config directory of the node.
Expand Down Expand Up @@ -144,7 +143,7 @@ With such topology file containerlab is instructed to take a file `myconfig.conf

It is possible to change the default config which every ceos node will start with with the following steps:

1. Save the [default configuration template](https://github.com/srl-labs/containerlab/blob/master/nodes/ceos/ceos.cfg) under some local file name[^2] and add the necessary changes to it
1. Craft a valid startup configuration file[^2].
2. Use this file as a startup-config for ceos kind:
```
name: ceos
Expand All @@ -167,25 +166,8 @@ It is possible to change the default config which every ceos node will start wit
- endpoints: ["ceos1:eth1", "ceos2:eth1"]
```

#### Configuration persistency

It is important to understand how configuration persistency behaves when a single lab is going through rounds of `deploy->destroy` actions.

When the lab with cEOS nodes gets deployed for the first time the configuration file is generated with the IPv4/6 address assigned to `Ma0` management interface. These management interface addresses match the IP addresses that docker has assigned to cEOS containers. This makes it possible to have the cEOS nodes to start up with Management interface already correctly addressed.

When a user later configures the nodes during the lab exercise and saves it with `wr mem` or similar, the changes will be written to `startup-config` file of cEOS.

User then may destroy the lab and the config changes will persist on disk, this is done with `destroy` command. During this operation the containers will be destroyed, but their configuration files will still be kept in the lab directory by the path `clab-$labName`.

If a user then desires to start this lab once again it may lead to a problem. Since docker may assign new IP addresses to the cEOS nodes of the lab, the configuration saved on disk may not match those new docker-assigned addresses, and that will result in an incorrect management interface configuration.

To avoid this, and be able to start the nodes with the previously saved configuration, users may do the following:

1. Address the nodes explicitly via [user defined addresses](../network.md#user-defined-addresses). This will instruct docker to use the addresses as specified by a user in a clab file.
2. Leverage [user defined config](#user-defined-config), if all you need is to have a startup config.

#### Saving configuration
In addition to cli commands such as `write memory` user can take advantage of the [`containerlab save`](../../cmd/save.md) command. It saves running cEOS configuration into a file by `conf-saved.conf` path in the relevant node directory.
In addition to cli commands such as `write memory` user can take advantage of the [`containerlab save`](../../cmd/save.md) command. It saves running cEOS configuration into a startup config file effectively calling the `write` CLI command.

## Container configuration
To start an Arista cEOS node containerlab uses the configuration instructions described in Arista Forums[^1]. The exact parameters are outlined below.
Expand Down Expand Up @@ -246,5 +228,5 @@ As of this writing (22-June, 2021), ceos-lab image requires a cgroups v1 environ
Consult your distribution's documentation for details regarding configuring cgroups v1 in case you see similar startup issues as indicated in [#467](https://github.com/srl-labs/containerlab/issues/467).
[^1]: https://eos.arista.com/ceos-lab-topo/
[^2]: do not remove the template variables from the `Management0` interface, otherwise the nodes will not apply the IP address from docker IPAM service.
[^2]: feel free to omit the IP addressing for Management interface, as it will be configured by containerlab when ceos node boots.
[^3]: if startup config needs to be enforced, either deploy a lab with `--reconfigure` flag, or use [`enforce-startup-config`](../nodes.md#enforce-startup-config) setting.
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ require (
github.com/olekukonko/tablewriter v0.0.5-0.20201029120751-42e21c7531a3
github.com/opencontainers/runtime-spec v1.0.3-0.20210303205135-43e4633e40c1
github.com/pkg/errors v0.9.1
github.com/scrapli/scrapligo v0.0.0-20210704164516-6c3b4e74cfad
github.com/scrapli/scrapligo v0.0.0-20210822185345-c949ba367b79
github.com/sirupsen/logrus v1.8.1
github.com/spf13/cobra v1.0.0
github.com/srl-labs/srlinux-scrapli v0.2.0
github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5
github.com/weaveworks/ignite v0.9.1-0.20210705155449-2dbcdd663727
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b
Expand Down
8 changes: 6 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ github.com/bugsnag/osext v0.0.0-20130617224835-0dd3f918b21b/go.mod h1:obH5gd0Bsq
github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0/go.mod h1:D/8v3kj0zr8ZAKg1AQ6crr+5VwKN5eIywRkfhyM/+dE=
github.com/c2h5oh/datasize v0.0.0-20200112174442-28bbd4740fee h1:BnPxIde0gjtTnc9Er7cxvBk8DHLWhEux0SxayC8dP6I=
github.com/c2h5oh/datasize v0.0.0-20200112174442-28bbd4740fee/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/carlmontanari/difflibgo v0.0.0-20210718170140-424f52054f94/go.mod h1:+3MuSIeC3qmdSesR12cTLeb47R/Vvo+bHdB6hC5HShk=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/certifi/gocertifi v0.0.0-20180118203423-deb3ae2ef261/go.mod h1:GJKEexRPVJrBSOjoqN5VNOIKJ5Q3RViH6eu3puDRwx4=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
Expand Down Expand Up @@ -919,8 +920,9 @@ github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD
github.com/safchain/ethtool v0.0.0-20190326074333-42ed695e3de8 h1:2c1EFnZHIPCW8qKWgHMH/fX2PkSabFc5mrVzfUNdg5U=
github.com/safchain/ethtool v0.0.0-20190326074333-42ed695e3de8/go.mod h1:Z0q5wiBQGYcxhMZ6gUqHn6pYNLypFAvaL3UvgZLR0U4=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/scrapli/scrapligo v0.0.0-20210704164516-6c3b4e74cfad h1:eb+6BSk5gTJ3rsQ1CfZGfMsoxvo7ZYwKlCTszU1CtJs=
github.com/scrapli/scrapligo v0.0.0-20210704164516-6c3b4e74cfad/go.mod h1:+csimZHh80jQXjdDdHmAIKCwiXPZvXQ7ZgKEQWmFpK8=
github.com/scrapli/scrapligo v0.0.0-20210814224131-df0e66d7cd23/go.mod h1:0tHMgiCiTuWOvSceFU7klaYThXvRZNvc7k+fmQrtH54=
github.com/scrapli/scrapligo v0.0.0-20210822185345-c949ba367b79 h1:fFnWvBZu5CLbZ5lKP7HJzOygxDQFWoDC6pVs1Yc44RQ=
github.com/scrapli/scrapligo v0.0.0-20210822185345-c949ba367b79/go.mod h1:0tHMgiCiTuWOvSceFU7klaYThXvRZNvc7k+fmQrtH54=
github.com/seccomp/libseccomp-golang v0.9.1/go.mod h1:GbW5+tmTXfcxTToHLXlScSlAvWlF4P2Ca7zGrPiEpWo=
github.com/securego/gosec v0.0.0-20191002120514-e680875ea14d/go.mod h1:w5+eXa0mYznDkHaMCXA4XYffjlH+cy1oyKbfzJXa2Do=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
Expand Down Expand Up @@ -967,6 +969,8 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
github.com/srl-labs/srlinux-scrapli v0.2.0 h1:uBvD7E326ucH1AKfe0ufo9063MAa+rTeIJSeBBZXy6o=
github.com/srl-labs/srlinux-scrapli v0.2.0/go.mod h1:j4SjAR3WX5OdjTSTaU8IJn4V7Hv6ateazBhI36AyKCk=
github.com/stefanberger/go-pkcs11uri v0.0.0-20201008174630-78d3cae3a980/go.mod h1:AO3tvPzVZ/ayst6UlUKUv6rcPQInYe3IknH3jYhAKu8=
github.com/stretchr/objx v0.0.0-20180129172003-8a3f7159479f/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down
71 changes: 35 additions & 36 deletions nodes/ceos/ceos.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
package ceos

import (
"bytes"
"context"
_ "embed"
"errors"
"fmt"
"net"
"os"
Expand Down Expand Up @@ -38,7 +38,7 @@ var (
//go:embed ceos.cfg
cfgTemplate string

saveCmd = []string{"Cli", "-p", "15", "-c", "copy running flash:conf-saved.conf"}
saveCmd = []string{"Cli", "-p", "15", "-c", "wr"}
)

func init() {
Expand Down Expand Up @@ -88,7 +88,7 @@ func (s *ceos) Deploy(ctx context.Context) error {
}

func (s *ceos) PostDeploy(ctx context.Context, ns map[string]nodes.Node) error {
log.Debugf("Running postdeploy actions for Arista cEOS '%s' node", s.cfg.ShortName)
log.Infof("Running postdeploy actions for Arista cEOS '%s' node", s.cfg.ShortName)
return ceosPostDeploy(ctx, s.runtime, s.cfg)
}

Expand Down Expand Up @@ -124,12 +124,12 @@ func createCEOSFiles(node *types.NodeConfig) error {
if err != nil {
return err
}
tpl := string(c)
cfgTemplate = string(c)
}

err = node.GenerateConfig(node.ResStartupConfig, tpl)
if err != nil {
return err
}
err := node.GenerateConfig(node.ResStartupConfig, cfgTemplate)
if err != nil {
return err
}

// sysmac is a system mac that is +1 to Ma0 mac
Expand All @@ -142,47 +142,46 @@ func createCEOSFiles(node *types.NodeConfig) error {
return nil
}

// ceosPostDeploy runs postdeploy actions which are required for ceos nodes
func ceosPostDeploy(ctx context.Context, r runtime.ContainerRuntime, node *types.NodeConfig) error {
// post deploy actions are not needed if a user specified startup config was provided
// and it doesn't have templation vars for ipv4 management address
if node.StartupConfig != "" {
c, err := os.ReadFile(node.StartupConfig)
if err != nil {
return err
}
if !bytes.Contains(c, []byte("{{ if .MgmtIPv4Address }}")) {
return nil
}

cfgTemplate = string(c)
}

// regenerate ceos config since it is now known which IP address docker assigned to this container
err := node.GenerateConfig(node.ResStartupConfig, cfgTemplate)
d, err := utils.SpawnCLIviaExec("arista_eos", node.LongName)
if err != nil {
return err
}

err = r.StopContainer(ctx, node.ContainerID)
if err != nil {
return err
defer d.Close()

cfgs := []string{
"interface management 0",
"no ip address",
"no ipv6 address",
}
// remove the netns symlink created during original start
// we will re-symlink it later
if err := utils.DeleteNetnsSymlink(node.LongName); err != nil {
return err

// adding ipv4 address to configs
if node.MgmtIPv4Address != "" {
cfgs = append(cfgs,
fmt.Sprintf("ip address %s/%d", node.MgmtIPv4Address, node.MgmtIPv4PrefixLength),
)
}

err = r.StartContainer(ctx, node.ContainerID)
if err != nil {
return err
// adding ipv6 address to configs
if node.MgmtIPv6Address != "" {
cfgs = append(cfgs,
fmt.Sprintf("ipv6 address %s/%d", node.MgmtIPv6Address, node.MgmtIPv6PrefixLength),
)
}
node.NSPath, err = r.GetNSPath(ctx, node.ContainerID)

// add save to startup cmd
cfgs = append(cfgs, "wr")

resp, err := d.SendConfigs(cfgs)
if err != nil {
return err
} else if resp.Failed() {
return errors.New("failed CLI configuration")
}

return utils.LinkContainerNS(node.NSPath, node.LongName)
return err
}

func (s *ceos) GetImages() map[string]string {
Expand Down
70 changes: 70 additions & 0 deletions utils/networkcli.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package utils

import (
"time"

"github.com/scrapli/scrapligo/driver/base"
"github.com/scrapli/scrapligo/driver/core"
"github.com/scrapli/scrapligo/driver/network"
"github.com/scrapli/scrapligo/transport"
log "github.com/sirupsen/logrus"
"github.com/srl-labs/srlinux-scrapli"
)

var (
// map of commands per platform which start a CLI app
NetworkOSCLICmd = map[string]string{
"arista_eos": "Cli",
"nokia_srlinux": "sr_cli",
}
)

// SpawnCLIviaExec spawns a CLI session over container runtime exec function
// end ensures the CLI is available to be used for sending commands over
func SpawnCLIviaExec(platform, contName string) (*network.Driver, error) {
var d *network.Driver
var err error

switch platform {
case "nokia_srlinux":
d, err = srlinux.NewSRLinuxDriver(
contName,
base.WithAuthBypass(true),
// disable transport timeout
base.WithTimeoutTransport(0),
)
default:
d, err = core.NewCoreDriver(
contName,
platform,
base.WithAuthBypass(true),
base.WithTimeoutTransport(0),
)
}

if err != nil {
log.Errorf("failed to create driver for device %s; error: %+v\n", err, contName)
return nil, err
}

// TODO: implement for ctr (containerd)
execCmd := "docker"
openCmd := []string{"exec", "-it"}

t, _ := d.Transport.(*transport.System)
t.ExecCmd = execCmd
t.OpenCmd = append(openCmd, contName, NetworkOSCLICmd[platform])

transportReady := false
for !transportReady {
if err := d.Open(); err != nil {
log.Debugf("%s - Cli not ready (%s) - waiting.", contName, err)
time.Sleep(time.Second * 2)
continue
}
transportReady = true
log.Debugf("%s - Cli ready.", contName)
}

return d, err
}

0 comments on commit e562d6f

Please sign in to comment.