management vrf using vrf function of new kernel

[batmancn]

• What I did

Implement management vrf function follow this spec, phase 1. This PR based on #2348 .

#2405 also define management vrf function, by namespace. This is another way to implement management vrf using vrf function in new kernel 4.5.

• How I did it

I use DB to set management vrf.
I use 'interfaces-config.sh' to trigger network restart.
For details, refer to 'How to verify it' section.

• How to verify it

1. use default DB settings to start SONiC
2. copy modified files as bellow

scp ./files/image_config/interfaces/interfaces.j2 root@172.18.8.213:/usr/share/sonic/templates/interfaces.j2

3. use these steps to add management vrf in DB

# cat /root/mgmt-vrf.del.json
{
    "MGMT_INTERFACE": {
        "eth0|172.18.8.213/24": null
    }
}

# config load -y /root/mgmt-vrf.del.json

# cat /root/mgmt-vrf.add.json
{
    "MGMT_INTERFACE": {
        "eth0|172.18.8.213/24": {
            "gwaddr": "172.18.8.1",
            "vrfname": "mgmtvrf"
        }
    }
}

# config load -y /root/mgmt-vrf.add.json

4. use this command to restart network and set management vrf

bash /usr/bin/interfaces-config.sh

5. use this command to verify

root@sonic:~# ip route show table 1001
default via 172.18.8.1 dev eth0
broadcast 172.18.8.0 dev eth0 proto kernel scope link src 172.18.8.213
172.18.8.0/24 dev eth0 proto kernel scope link src 172.18.8.213
local 172.18.8.213 dev eth0 proto kernel scope host src 172.18.8.213
broadcast 172.18.8.255 dev eth0 proto kernel scope link src 172.18.8.213

6. use these steps to del management vrf in DB

# cat /root/mgmt-vrf.del.json
{
    "MGMT_INTERFACE": {
        "eth0|172.18.8.213/24": null
    }
}

# config load -y /root/mgmt-vrf.del.json

# cat /root/mgmt-vrf.old.json
{
    "MGMT_INTERFACE": {
        "eth0|172.18.8.213/24": {
            "gwaddr": "172.18.8.1"
        }
    }
}

# config load -y /root/mgmt-vrf.old.json

7. use this command to restart network and set management vrf

bash /usr/bin/interfaces-config.sh

8. use these command to verify

# ip route show table 1001

# ip route show
default via 172.18.8.1 dev eth0 proto zebra
...
172.18.8.0/24 dev eth0 proto kernel scope link src 172.18.8.213
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown

• Known issue

Services based on old socket will be break down in a short time, like sshd/ntpd/logd...

[lguohan]

@nikos-github , you are using ifupdown2 to bring up eth0, do you have that PR. i think that will simplify the template here?

[nikos-github]

@lguohan For the dhcp case with ifupdown2 and putting eth0 in the mgmt vrf by default, below are the changes I made. Nothing else was needed.

diff --git a/files/image_config/interfaces/interfaces.j2 b/files/image_config/interfaces/interfaces.j2
index da2499e..d34ce0a 100644
--- a/files/image_config/interfaces/interfaces.j2
+++ b/files/image_config/interfaces/interfaces.j2
@@ -17,6 +17,9 @@ iface lo {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 #
 {% endfor %}
 {% endblock loopback %}
+auto mgmt
+iface mgmt
+    vrf-table 1001
 {% block mgmt_interface %}
 
 # The management network interface
@@ -45,6 +48,7 @@ iface eth0 {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 {% endfor %}
 {% else %}
 iface eth0 inet dhcp
+    vrf mgmt
 {% endif %}
 #
 {% endblock mgmt_interface %}

[batmancn]

@lguohan For the dhcp case with ifupdown2 and putting eth0 in the mgmt vrf by default, below are the changes I made. Nothing else was needed.

diff --git a/files/image_config/interfaces/interfaces.j2 b/files/image_config/interfaces/interfaces.j2
index da2499e..d34ce0a 100644
--- a/files/image_config/interfaces/interfaces.j2
+++ b/files/image_config/interfaces/interfaces.j2
@@ -17,6 +17,9 @@ iface lo {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 #
 {% endfor %}
 {% endblock loopback %}
+auto mgmt
+iface mgmt
+    vrf-table 1001
 {% block mgmt_interface %}
 
 # The management network interface
@@ -45,6 +48,7 @@ iface eth0 {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 {% endfor %}
 {% else %}
 iface eth0 inet dhcp
+    vrf mgmt
 {% endif %}
 #
 {% endblock mgmt_interface %}

That's cool.

Actually I'm using this way at beginning (https://stackoverflow.com/questions/53946282/how-to-define-vrf-interfaces-in-etc-network-interfaces), but I got errors, so I change my code.

I will try this way again follow this patch.

[nikos-github]

@batmancn The table with ifupdown2 needs to be in the 1001 to 1255 range when you specify it. I picked 1001 above.

[batmancn]

@lguohan For the dhcp case with ifupdown2 and putting eth0 in the mgmt vrf by default, below are the changes I made. Nothing else was needed.

diff --git a/files/image_config/interfaces/interfaces.j2 b/files/image_config/interfaces/interfaces.j2
index da2499e..d34ce0a 100644
--- a/files/image_config/interfaces/interfaces.j2
+++ b/files/image_config/interfaces/interfaces.j2
@@ -17,6 +17,9 @@ iface lo {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 #
 {% endfor %}
 {% endblock loopback %}
+auto mgmt
+iface mgmt
+    vrf-table 1001
 {% block mgmt_interface %}
 
 # The management network interface
@@ -45,6 +48,7 @@ iface eth0 {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 {% endfor %}
 {% else %}
 iface eth0 inet dhcp
+    vrf mgmt
 {% endif %}
 #
 {% endblock mgmt_interface %}

Oh, this patch could not work. If it's my mistake?

This is the /etc/network/interfaces I process

root@sonic:~# cat /etc/network/interfaces
#
# =============== Managed by SONiC Config Engine DO NOT EDIT! ===============
# generated from /usr/share/sonic/templates/interfaces.j2 using sonic-cfggen
# file: /etc/network/interfaces
#
# The loopback network interface
auto lo
iface lo inet loopback
# Use command 'ip addr list dev lo' to check all addresses
iface lo inet static
    address 10.1.0.1
    netmask 255.255.255.255
#

auto mgmt
iface mgmt # this is line 16
    vrf-table 1001

# The management network interface
auto eth0
iface eth0 inet dhcp
    vrf mgmt
#

I use this command

/etc/init.d/networking restart

This is the error I got

-- The result is failed.
Nov 07 18:56:05 sonic systemd[1]: networking.service: Unit entered failed state.
Nov 07 18:56:05 sonic systemd[1]: networking.service: Failed with result 'exit-code'.
Nov 07 18:57:01 sonic CRON[6687]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 07 18:57:01 sonic CRON[6688]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1)
Nov 07 18:57:01 sonic CRON[6687]: pam_unix(cron:session): session closed for user root
Nov 07 18:58:01 sonic CRON[6705]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 07 18:58:01 sonic CRON[6706]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1)
Nov 07 18:58:01 sonic CRON[6705]: pam_unix(cron:session): session closed for user root
Nov 07 18:58:13 sonic kernel: perf: interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
Nov 07 18:58:14 sonic su[6443]: pam_unix(su:session): session closed for user root
Nov 07 18:59:01 sonic CRON[6723]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 07 18:59:01 sonic CRON[6724]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1)
Nov 07 18:59:01 sonic CRON[6723]: pam_unix(cron:session): session closed for user root
Nov 07 18:59:42 sonic systemd[1]: Starting Raise network interfaces...
-- Subject: Unit networking.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit networking.service has begun starting up.
Nov 07 18:59:42 sonic sh[6748]: ifquery: /etc/network/interfaces:16: unknown or no address type and no inherits keyword specified
Nov 07 18:59:42 sonic sh[6748]: ifquery: couldn't read interfaces file "/etc/network/interfaces"
Nov 07 18:59:42 sonic ifup[6751]: ifup: /etc/network/interfaces:16: unknown or no address type and no inherits keyword specified
Nov 07 18:59:42 sonic ifup[6751]: ifup: couldn't read interfaces file "/etc/network/interfaces"
Nov 07 18:59:42 sonic systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 18:59:42 sonic systemd[1]: Failed to start Raise network interfaces.
-- Subject: Unit networking.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit networking.service has failed.
--
-- The result is failed.
Nov 07 18:59:42 sonic systemd[1]: networking.service: Unit entered failed state.
Nov 07 18:59:42 sonic systemd[1]: networking.service: Failed with result 'exit-code'.

I use 'sonic-mellanox-201811-6f371.bin', this is my OS

root@sonic:~# uname -a
Linux sonic 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2015-12-19) x86_64 GNU/Linux

[nikos-github]

@batmancn Some changes from #2348 are also needed.

[batmancn]

@lguohan For the dhcp case with ifupdown2 and putting eth0 in the mgmt vrf by default, below are the changes I made. Nothing else was needed.

diff --git a/files/image_config/interfaces/interfaces.j2 b/files/image_config/interfaces/interfaces.j2
index da2499e..d34ce0a 100644
--- a/files/image_config/interfaces/interfaces.j2
+++ b/files/image_config/interfaces/interfaces.j2
@@ -17,6 +17,9 @@ iface lo {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 #
 {% endfor %}
 {% endblock loopback %}
+auto mgmt
+iface mgmt
+    vrf-table 1001
 {% block mgmt_interface %}
 
 # The management network interface
@@ -45,6 +48,7 @@ iface eth0 {{ 'inet' if prefix | ipv4 else 'inet6' }} static
 {% endfor %}
 {% else %}
 iface eth0 inet dhcp
+    vrf mgmt
 {% endif %}
 #
 {% endblock mgmt_interface %}

@batmancn Some changes from #2348 are also needed.

Still some error.

1. I add some feature from Initial changes for dhcp to support eth0 in a mgmt vrf #2348

apt-get install -y ifupdown2
apt-get install -y squashfs-tools

root@sonic:/# dpkg -l | grep ifupdown
rc  ifupdown                      0.8.19                         amd64        high level tools to configure network interfaces
ii  ifupdown2                     1.0~git20170314-1              all          Network Interface Management tool similar to ifupdown

root@sonic:/# reboot

2. I create '/etc/dhcp/dhclient-exit-hooks.d/vrf' same as 'files/dhcp/vrf'

3. This is management interface config in /etc/sonic/config_db.json

root@sonic:/# grep -r "MGMT" -A 5 /etc/sonic/config_db.json
    "MGMT_INTERFACE": {
        "eth0|172.18.8.213/24": {
            "gwaddr": "172.18.8.1"
        }
    },
    "PORT": {

4. I command /etc/init.d/networking restart

5. error ip route information

root@sonic:/# ip route show table 1001  ## here should be 172.18.8.213, but result is 172.18.9.213
broadcast 172.18.9.0 dev eth0 proto kernel scope link src 172.18.9.213
172.18.9.0/24 dev eth0 proto kernel scope link src 172.18.9.213
local 172.18.9.213 dev eth0 proto kernel scope host src 172.18.9.213
broadcast 172.18.9.255 dev eth0 proto kernel scope link src 172.18.9.213

5. I got error in /var/log/syslog ## why 'vrf_exists: not found', 'vrf_get_table_dev: not found'

Nov  8 14:38:04.012720 sonic INFO dhclient[29785]: DHCPDISCOVER on eth0 to 255.22
55.255.255 port 67 interval 18
Nov  8 14:38:05.022600 sonic INFO dhclient[29785]: DHCPREQUEST of 172.18.9.213 oo
n eth0 to 255.255.255.255 port 67
Nov  8 14:38:05.023441 sonic INFO dhclient[29785]: DHCPOFFER of 172.18.9.213 froo
m 172.18.9.106
Nov  8 14:38:05.025825 sonic INFO dhclient[29785]: DHCPACK of 172.18.9.213 from
172.18.9.106
Nov  8 14:38:05.045385 sonic INFO syncd#supervisord: syncd Nov 08 14:38:05 NOTICC
E  SAI_UTILS: mlnx_sai_utils.c[2290]- set_dispatch_attrib_handler: Set PACKET_ACC
TION, key:route 0.0.0.0 0.0.0.0, val:DROP
Nov  8 14:38:05.048648 sonic INFO networking[29726]: /sbin/dhclient-script: 22:
/etc/dhcp/dhclient-exit-hooks.d/vrf: vrf_exists: not found
Nov  8 14:38:05.049993 sonic INFO networking[29726]: /sbin/dhclient-script: 29:
/etc/dhcp/dhclient-exit-hooks.d/vrf: vrf_get_table_dev: not found
Nov  8 14:38:05.061603 sonic INFO dhclient[29785]: bound to 172.18.9.213 -- renee
wal in 296 seconds.
Nov  8 14:38:05.366402 sonic INFO systemd[1]: Started ifupdown2 networking initii
alization.

[nikos-github]

A couple of things. You can't have both ifupdown and ifupdown2. Also, when you specify dhcp for eth0 instead of the static IP that you have in config_db.json, you will clearly get the discrepancy you are observing. The point Guohan was making is that even with static IP assignment, the template you have for eth0 in the mgmt vrf can be simplified.

[nikos-github]

@batmancn The builds are failing because of unit tests. Have a look at src/sonic-config-engine/tests/test_j2files.py in def test_interfaces(self). I think we also want to parameterize this configuration unless the plan is to have it on by default for both dhcp and static address assignment. Please check with @lguohan.