Monitoring

[Muddyblack]

So just a quick idea — I think this could add more maintenance, but I just propose it as an idea. Maybe when there is time, it might actually be pursued! :D

Does netlab already offer something to monitor a network?

Why not add a new plugin, or more likely a netlab core feature, that allows users to monitor their network? Something like injecting Grafana and Prometheus.

The thing is, I guess this would require Jinja templates for the Grafana dashboard.json files.

I actually have such a plugin working, but it is currently hardcoded for FRR (specifically for BGP and OSPF). Of cause :D
And requires also presetup on the docker images as well a own collector docker for frr as frr exporter itself sadly does not expose everything one is interested in. So this is also ugly for now sth one does not really want to deal with else he would do it on his own anyways.

So for now not the best solution but I mean there will be better ones existing making it clean to use. I am sure of that it will just take time to correctly think it through and implement such.

What it tracks:

• System: CPU, RAM, per-interface traffic
• OSPF: Total lab LSAs, OSPF adjacencies, per-node LSA count, LSA type distribution, OSPF neighbor states over time, SPF calculation rates, and SPF calculation time
• BGP: Peers, BGP prefixes, BGP session uptime, BGP messages, message rate, and convergence
• General Routing: Learned routes, current neighbors, convergence, and flapping

And this alone is a 3k lines JSON file, so it would be a bigger project I think!

How it works right now:

• I can choose any lab (if its folder still exists), and within each lab, select any device.
• Yet again, I only have it done for containerlab. Each containerlab deployment runs its own monitoring stack and saves the data directly in its own lab folder.

SO its cool as you just have all your network data in one place visually allowing to faster cleaner analyse it.
Sth else cool would be to offer sth with that you could automate some tests easy and directly have it visually also. Liek doing flapping tests etc.

[DanPartelly]

SuzieQ tool ?

[Muddyblack]

Seems interesting for gnmi etc it requires enterprise pricing tho?
But it has already all vendors etc ^

[DanPartelly]

It's a free open source product, so you can add other NOSes or add features and contribute them upstream. Just evaluate the tool, see if it fits your needs.

Do one thing and doit well. Netlab is the ultimate "network as code" tool with unmatched device abstraction capabilities. It is long since it surpassed the level of "lab" tool. I personally see 0 reason to try to fit it in that weird shaped hole network monitoring is. That's for another product to cover.

[Muddyblack]

You are right it is a lot and again a lot of different varieties.

[ipspace]

Something like injecting Grafana and Prometheus.

You're more than welcome to add Grafana or Prometheus as tools to netlab. Check the existing tool definitions; I'm pretty sure you'll nail it on the first try.

The core idea is to specify the docker commands to start the tools, and the templates used to render tool configuration files from transformed netlab data. Unfortunately, there's no documentation (yet).

[Muddyblack]

Alright I will start a branch for it having it as a new tool besides graphite and suzieq.
So the first structure stands now I have to make sure it takes all my old graphana dashboard options over into this new setup.

Before in my plugin injector variant I dumped all in one Dashboard Item. Now here I separate Dashboards for each Protocol and its own Dashbaord for System metrics etc.

First structure (did not copy my original variant as expected but proofs it to work) :

[Muddyblack]

This uses the out of band management bridge. My plugin variant also had an aggregator docker that was able to reach devices when the out of band management was deactivated but for the first version it might be too much for netlab?

But I will tinker about it for some time :)

[ipspace]

This uses the out of band management bridge.

Yes, the tools don't have an in-lab connection and are supposed to reach the devices over the management interface.

My plugin variant also had an aggregator docker that was able to reach devices when the out of band management was deactivated but for the first version it might be too much for netlab?

If you want to have an in-lab management device, then use use Linux node with a different image, and clab.config_templates to generate the config files.

[ipspace]

So the first structure stands now I have to make sure it takes all my old graphana dashboard options over into this new setup.

It might be worthwhile to have a simple dashboard config shipped with netlab and explain in the tool documentation how a user can create their own config templates and where to put them (within the lab topology directory and within the ~/.netlab/ directory)

[sdargoeuves]

this discussion + the Network Services for syslog could be getting us closer to having this and replace Sonda with the actual netlab devices.

As mentioned above, it's beyond the scope of netlab, but what a cool set of tools it could be!

[ipspace]

I think it's even simpler than that. Modify their "docker-compose.yml" to use the containerlab management network, replace the collection configuration, and you have it ;)

[Muddyblack]

Or maybe even link it with the netlab-ui later. Adding in there then also possibility doing automatic repetitive tests like flapping and so on with timing reproduceable if possible. In that case as Dan proposed it would be another product covering it XD

But for now that is future "Hirngespinnste" we will see.