data model augmentation for devices with centralized configuration

[a-v-popov]

It would seem that netlab data model is designed around Cisco IOS mental picture that everything is configured on interfaces.
As result every other device has to loop in templates again and again over full interface list to build a configuration.
Specifically I am looking at VLAN configuration and it seems there is no independent bridges object or anything like this, only vlans which doesn't seem to have any member list.

Would you be willing to consider augmenting data provided to templates to accommodate platforms driving configuration from centralized objects?

[ipspace]

It would seem that netlab data model is designed around Cisco IOS mental picture that everything is configured on interfaces.

Not exactly. The data is attached to interfaces (because that's where it's usually generated), how you use them is a different story. As you know, we support several devices that have drastically different configuration approaches (for example, Junos).

As result every other device has to loop in templates again and again over full interface list to build a configuration.

Yes. Where's the problem in that? Time spent generating device configurations is trivial compared to everything else we have to go through to have a lab started (even when using fast containers like FRR)

Specifically I am looking at VLAN configuration and it seems there is no independent bridges object or anything like this, only vlans which doesn't seem to have any member list.

Correct. Having duplicate data is usually a bad idea, and there are only a few exceptions where we do that (VRF interfaces inside the routing protocol VRF instances -- honestly, can't remember why exactly I went with that approach).

However, we use per-VLAN bridges in FRRouting, and apart from a few loops across the interfaces the result is not too shabby.

Would you be willing to consider augmenting data provided to templates to accommodate platforms driving configuration from centralized objects?

Only when the resulting Jinja2 template would otherwise become too convoluted (and a few extra loops is not nearly convoluted enough ;).

For example, we eventually added OSPF areas to OSPF data structure for devices that have process/area/interface configuration hierarchy because it was too cumbersome to build area list in Jinja2, but did not copy OSPF interface data to OSPF areas because it's trivial to just loop over the interfaces and check their area ID. Effectively, we added an equivalent to the already-existing vlans dictionary to OSPF.

Similarly, we added few attributes to BGP neighbors because every single Jinja2 template used the same convoluted logic to figure out (for example) whether the neighbor is a BGP RR client.

Every time we create duplicate data we risk that we'll break something that changes the original data structure after we create the duplicate (not to mention all the other related technical debt), so there has to be a really good reason to add more data to the data model.

[a-v-popov]

It is not about performance, it is about were your business logic lives.

Having duplicate data is usually a bad idea...

We are not talking about committing data in a database, we are talking about preparing for templating.

[ipspace]

It is not about performance, it is about were your business logic lives.

As I wrote, if the business logic becomes too convoluted, we can discuss. Right now, we have ~15 different VLAN implementations, and we managed to implement them all with the existing data model. Is it perfect? Of course not. Does it work? Yes. Are there tradeoffs? Always.

We are not talking about committing data in a database, we are talking about preparing for templating.

And where's the difference? If you have the duplicate information in several places and you only modify it in one, it doesn't matter whether it's in memory or on disk.

Finally, if you really want to have a bespoke data model, you could always create it as a device quirk. We had to do that to cope with Junos and IOS XR routing policies... and OF COURSE I broke stuff when I tried to change Junos routing policies because the routing policies were modified in several templates, and I forgot about one of them. At that moment, I was so glad we have so many integration tests...