[SONIC] Management VRF design doc #226

vharish02 · 2018-08-06T19:50:55Z

This pull request has Management VRF design.

Signed-off-by: Harish Venkatraman Harish_Venkatraman@dell.com

This pull request has Management VRF design. Signed-off-by: Harish Venkatraman <Harish_Venkatraman@dell.com>

prsunny · 2018-08-07T17:36:47Z

Questions:

For sflow type usecase, where sample packets received on FEP and forwarded to a collector via mgmt. port, how is this handled with the Mgmt VRF (l3mdev) approach?
In general, does this require any Netlink related changes for l3mdev?

shine4chen · 2018-09-05T06:05:39Z

@prsunny
I try to answer your two questions.

I don't think sflow packet should be handled in mgmt vrf. It should be done in default vrf and other data-plane vrf.
In linux 4.9 netlink has supported vrf master and slave notification.

lguohan · 2018-09-21T00:28:18Z

mgmt-vrf-design-doc.md

+mgmt traffic and data traffic. This is acheived by setting the vrfname to mgmt-vrf in the minigraph.py file as default setting.
+The interfaces.j2 file checks for the vrfname tag to see if it is set to mgmt-vrf. If the vrfname is set then it creates the
+mgmt-vrf configuration by executing the lniux commands. Debian 4.9 kernel does not support running services per VRF, by enabling
+the tcp_l3mdev_accept=1 the services will work across all VRF's. Below diagram shows the flows of event for creating mgmt-vrf on


is this only for tcp, do we need udp_l3mdev_accept as well?

udp_l3mdev_accept is not there in 4.9 kernel need to backport the kernel code.

since snmpd must be accessible from mgmt network, this is a blocker for management vrf feature.

here is the kernel upstream patch. torvalds/linux@63a6fff

can you test it and take it to 4.9 kernel?

Sure will do

nikos-github · 2018-09-25T04:43:51Z

mgmt-vrf-design-doc.md

+    "MGMT_INTERFACE": {
+        "eth0|10.11.150.19/24": {
+            "gwaddr": "10.11.150.254"
+            "vrfname": "mgmt-vrf"


Please change to "vrf": "mgmt" or "vrf": "mgmt-vrf" to be consistent with the vrf proposal.

nikos-github · 2018-09-25T04:44:45Z

mgmt-vrf-design-doc.md

+    }
+
+The vrfname if set to mgmt-vrf will program the linux kernel with the vrf configuration and create a mgmt-vrf for the mgmt
+port eth0. If it is set to "None" then the mgmt-vrf will not be configured on the system. Management VRF will be enabled by default when SONiC boots. 


Default configuration should be none or it will break existing deployments.

I believe the requirement was to bring up with mgmt-vrf, But this can be changed.

If no "vrfname": "mgmt-vrf" in config_db.json, mgmt vrf is enabled by default? I don't see mgmt vrf enabled without "vrfname" config or CLI "config vrf add mgmt" with 201911 image.

nikos-github · 2018-09-25T04:45:16Z

mgmt-vrf-design-doc.md

+-----
+
+By default, mgmt-vrf will be configured in Debian 4.9 kernel, when SONiC boots there will be isolation of traffic between
+mgmt traffic and data traffic. This is acheived by setting the vrfname to mgmt-vrf in the minigraph.py file as default setting.


I believe minigraph.py is getting removed completely.

This if for phase-1 only. in phase 2 we will use config command to configure.

nikos-github · 2018-09-25T04:46:20Z

mgmt-vrf-design-doc.md

+Flow
+-----
+
+By default, mgmt-vrf will be configured in Debian 4.9 kernel, when SONiC boots there will be isolation of traffic between


See above on why it can't be the default.

nikos-github · 2018-09-25T08:01:20Z

mgmt-vrf-design-doc.md

+The config_db.json schema is updated to have the vrfname in MGMT_INTERFACE. The schema representation will look like below
+
+    "MGMT_INTERFACE": {
+        "eth0|10.11.150.19/24": {


Please outline in your design document how this will work when eth0 ip address and gateway are obtained via dhcp and not statically configured.

will update

nikos-github · 2018-09-25T08:02:19Z

mgmt-vrf-design-doc.md

+         }
+    }
+
+The vrfname if set to mgmt-vrf will program the linux kernel with the vrf configuration and create a mgmt-vrf for the mgmt


How will the old vrf cleanup happen if user specifies in config_db.json a new vrf for mgmt and does a config reload?

Thereis a scirpt to cleanup the old vrf.

lguohan · 2018-09-26T08:34:44Z

mgmt-vrf-design-doc.md

+    | 6      | Enable DHCP services to run in Management VRF context                                                      |          |          |
+    |        | ˇ       Dhcpd                                                                                              |          |          |
+    |        | ˇ       Dhcrelay  Default VRF support                                                                     |          |          |
+    |        | ˇ       dhclient                                                                                           |          |          |


it is unclear from the design doc that how dhclient can be running in management vrf context and what is the implication for that.

Will update.

nikos-github · 2018-10-03T21:52:25Z

mgmt-vrf-design-doc.md

+-------
+
+In this Phase mgmt-vrf feature will be enabled by default when SONiC boots with Debian 4.9 kernel, configuration of mgmt-vrf will be
+done using linux commands.


Need to describe how DNS resolution will work with eth0 being in the mgmt vrf.

nikos-github · 2018-10-03T21:54:16Z

mgmt-vrf-design-doc.md

+![](https://i.imgur.com/Ge6SNX4.png)
+
+
+Phase-2


NTP and DNS need to work when eth0 is placed in the mgmt vrf in Phase-1.

a-barboza · 2019-11-13T18:37:24Z

mgmt-vrf-design-doc.md

+      Packet coming via Front end port	  All IP services listed in Table 1 should          Treat the packet like any other data traffic. 
+                                          be supported.                                     To be sent out as per the routing table lookup
+
+     2.In band management


Is there an update to this document ? Apparently per "sonic-net/sonic-utilities#463" only the Management Port (eth0) is part of the management VRF. This description here seems to indicate that Front Panel/In-Band/Front End ports can be part of the management VRF.

a-barboza · 2019-11-13T18:40:30Z

mgmt-vrf-design-doc.md

+    show vrf config
+    show vrf brief
+    show vrf <vrfname>
+


Adding some comments on behalf of the Dilip Kumar/VRF Team:

CLI Syntax in Mgmt VRF HLD Feedback/suggestions Notes

config vrf add mgmt config vrf add Vrf-mgmt VRF name should be prefixed by "Vrf"

config vrf del mgmt config vrf del Vrf-mgmt

config interface ip add eth0 ip/mask gatewayIP config interface ip add eth0 <IP/Mask> The default GW must be configured as a regular static IP route command (ip route add 0.0.0.0/0 GW)

config interface ip remove eth0 ip/mask

show mgmt-vrf show vrf Vrf-mgmt [--verbose] This command should display the data from ConfigDB instead from kernel, like for other VRFs

show mgmt-vrf routes show ip route vrf Vrf-mgmt This should display routes from Zebra instead from Kernel. Behavior should be similar to other user VRFs

wendani · 2020-01-15T06:49:40Z

mgmt-vrf-design-doc.md

+Introduction
+------------
+
+Management VRF is a subset of VRF (virtual routing tables and forwarding) and provides a separation between the out-of-band management network and the in-band data plane network. For all VRFs, the main routing table is the default table for all of the data plane switch ports. With management VRF, a second table, mgmt, is used for routing through the Ethernet ports of the switch. The following design for mgmt-vrf uses the l3mdev approach for implementating mgmt-vrf on SONiC.


Need some changes in wording as the current one is similar to https://docs.cumulusnetworks.com/cumulus-linux/Layer-3/Management-VRF

xianlongqin · 2020-01-15T19:14:59Z

mgmt-vrf-design-doc.md

+                                          To be sent out as per the routing table lookup    To be sent out as per the routing table lookup
+                                          Dest IP = FEP IP in Default VRF                   Dest IP ≠ FEP IP in Default VRF
+
+      Packet coming via mgmt. port        Packet dropped                                    Packet dropped


Confusing. Line 40 is also "Packet coming via mgmt. port", why the actions are different.

The table is not formatted well. There are two scenarios discussed:

Packets coming with 1) Dest IP == Mgmt IP; 2) Dest IP == front-end-port IP. A line with "------------------" should be drawn below line 45

https://github.com/Azure/SONiC/pull/226/files/310de5a9d649481e23ec3b048cbe90242c6e063f#r367057554

xianlongqin · 2020-01-15T19:22:59Z

mgmt-vrf-design-doc.md

+    }
+
+The vrfname if set to mgmt-vrf will program the linux kernel with the vrf configuration and create a mgmt-vrf for the mgmt
+port eth0. If it is set to "None" then the mgmt-vrf will not be configured on the system. Management VRF will be enabled by default when SONiC boots. 


If no "vrfname": "mgmt-vrf" in config_db.json, mgmt vrf is enabled by default? I don't see mgmt vrf enabled without "vrfname" config or CLI "config vrf add mgmt" with 201911 image.

prsunny · 2020-04-16T18:51:37Z

Revised document is available at - #603

rajkumar38

A general question. Does SWSS 00-copp.config.json will be applicable to traffic received on in-band-mgmt interface ?

[SONIC] Management VRF design doc

310de5a

This pull request has Management VRF design. Signed-off-by: Harish Venkatraman <Harish_Venkatraman@dell.com>

lguohan reviewed Sep 21, 2018

View reviewed changes

nikos-github reviewed Sep 25, 2018

View reviewed changes

lguohan reviewed Sep 26, 2018

View reviewed changes

nikos-github suggested changes Oct 3, 2018

View reviewed changes

a-barboza reviewed Nov 13, 2019

View reviewed changes

wendani reviewed Jan 15, 2020

View reviewed changes

xianlongqin reviewed Jan 21, 2020

View reviewed changes

rajkumar38 reviewed May 25, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SONIC] Management VRF design doc #226

[SONIC] Management VRF design doc #226

vharish02 commented Aug 6, 2018

prsunny commented Aug 7, 2018

shine4chen commented Sep 5, 2018

lguohan Sep 21, 2018

vharish02 Sep 22, 2018

lguohan Sep 22, 2018

lguohan Sep 22, 2018

vharish02 Sep 24, 2018

nikos-github Sep 25, 2018

vharish02 Sep 26, 2018

nikos-github Sep 25, 2018

vharish02 Sep 26, 2018

xianlongqin Jan 15, 2020

nikos-github Sep 25, 2018

vharish02 Sep 26, 2018

nikos-github Sep 25, 2018

nikos-github Sep 25, 2018

vharish02 Sep 26, 2018

nikos-github Sep 25, 2018

vharish02 Sep 26, 2018

lguohan Sep 26, 2018

vharish02 Sep 26, 2018

nikos-github Oct 3, 2018

nikos-github Oct 3, 2018

a-barboza Nov 13, 2019

a-barboza Nov 13, 2019

wendani Jan 15, 2020

xianlongqin Jan 15, 2020

wendani Jan 22, 2020

xianlongqin Jan 15, 2020

prsunny commented Apr 16, 2020

rajkumar38 left a comment

CLI Syntax in Mgmt VRF HLD	Feedback/suggestions	Notes
config vrf add mgmt	config vrf add Vrf-mgmt	VRF name should be prefixed by "Vrf"
config vrf del mgmt	config vrf del Vrf-mgmt
config interface ip add eth0 ip/mask gatewayIP	config interface ip add eth0 <IP/Mask>	The default GW must be configured as a regular static IP route command (ip route add 0.0.0.0/0 GW)
config interface ip remove eth0 ip/mask

show mgmt-vrf	show vrf Vrf-mgmt [--verbose]	This command should display the data from ConfigDB instead from kernel, like for other VRFs
show mgmt-vrf routes	show ip route vrf Vrf-mgmt	This should display routes from Zebra instead from Kernel. Behavior should be similar to other user VRFs

		![](https://i.imgur.com/Ge6SNX4.png)


		Phase-2

[SONIC] Management VRF design doc #226

Are you sure you want to change the base?

[SONIC] Management VRF design doc #226

Conversation

vharish02 commented Aug 6, 2018

prsunny commented Aug 7, 2018

shine4chen commented Sep 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prsunny commented Apr 16, 2020

rajkumar38 left a comment

Choose a reason for hiding this comment