Network macvlan for containers #12

stephdl · 2020-06-02T21:13:22Z

Macvlan must be created by filling some important parameters, the goal is to create a container with an IP on your network

macVlanGateway is the gateway of your network (192.168.12.1)
macVlanLocalNetwork is the full network of your router (192.168.12.0/24)
macVlanNetwork is the restricted IP for macVlan0 (192.168.12.192/27 here 32 IP for your container 192.168.12.193->192.168.12.222)
macVlanNic is the NIC where to run macvlan (eth0 here)

[root@ns7dev9 ~]# config setprop  docker macVlanGateway 192.168.12.1 macVlanLocalNetwork 192.168.12.0/24 macVlanNetwork 192.168.12.192/27 macVlanNic eth0
[root@ns7dev9 ~]# signal-event nethserver-docker-update

create the container with an IP on your network here 192.168.12.211

docker run --net=macvlan -dit --name nginx-test-01 --ip=192.168.12.211 --restart=unless-stopped nginx:alpine nginx-debug -g 'daemon off;'

if all is good, the container is reachable locally and externally

[root@ns7dev9 ~]# curl 192.168.12.211
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

some debug command

check macvlan0 is created
ip a
must be inet 192.168.12.192/27
check docker macvlan network is created
docker network ls
b4f0244f07cb macvlan macvlan local

nethbot · 2020-06-02T21:15:17Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.7.pr12.g5c56f3d.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-02T21:40:53Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.7.pr12.gf23fd8a.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-02T22:04:38Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.8.pr12.g20d3cfd.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-03T13:31:18Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.8.pr12.g315adba.ns7.noarch.rpm x86_64 armhfp aarch64

mrmarkuz

It's working as expected.
macvlan0 and docker macvlan network are created and the test container is curlable from host and local network.

stephdl · 2020-06-03T14:25:27Z

did you check after a server restart, does macvlan0 is created after the reboot, does the container is up ?

Did you check also after a shorewall restart, a docker restart ?

mrmarkuz · 2020-06-03T14:38:21Z

I need to recheck with a clean install, I tested too much and now the test machine hangs...
I can confirm that the route was recreated after reboot (I deleted it before) and macvlan0 was there...
EDIT:
The container is up and reachable from host and network even after restarting docker or shorewall or after reboot.

The ssh connection to the docker host freezes for some seconds about every 20 seconds after reboot of the docker host. When I restart shorewall the issue disappears.

A docker-ce update stops the test container and it's not startable anymore, I needed to remove and recreate.

Error response from daemon: failed to listen to abstract unix socket "/containerd-shim/moby/a9070b5aa314b7a4aa64b9d08263ca44a92dccb9ca1337c251278e6eece7d736/shim.sock": listen unix /containerd-shim/moby/a9070b5aa314b7a4aa64b9d08263ca44a92dccb9ca1337c251278e6eece7d736/shim.sock: bind: address already in use: unknown
Error: failed to start containers: a9070b5aa314

stephdl · 2020-06-03T16:56:34Z

I think we need to enable live-restore
https://docs.docker.com/config/containers/live-restore/

for what I tested it works well

nethbot · 2020-06-03T17:10:18Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.9.pr12.g5808f73.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-03T17:19:38Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.10.pr12.gb724ed3.ns7.noarch.rpm x86_64 armhfp aarch64

stephdl · 2020-06-03T17:21:18Z

confirmed with live-restore the container is not hanged during the service restart nor the service upgrade

mrmarkuz · 2020-06-03T18:42:48Z

Great! It's working now even after docker upgrade.
The ssh freeze issue disappeared too.

stephdl · 2020-06-03T18:44:25Z

I ask myself if it could solve our issue with aeria ?

pagaille · 2020-06-03T19:17:03Z

Found a bug (I think)

To reproduce : set the prop macVlanNetwork to some subnet, apply by firing the nethserver-docker-update, then set the prop to another subnet and fire the event again.

Expected : last subnet is reported with ip a
Instead, the two subnets are present :

24: macvlan0@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 76:e0:32:98:bc:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.192/27 scope global macvlan0
       valid_lft forever preferred_lft forever
    inet 10.0.1.224/27 scope global macvlan0
       valid_lft forever preferred_lft forever

stephdl · 2020-06-03T19:27:50Z

aeria and macvlan cannot share the same bridge br0 :(

mrmarkuz · 2020-06-03T19:36:59Z

I tested aeria, it seems we have some more issues:

Portainer refuses to connect locally to /var/run/docker.sock, even without aeria.
Starting pihole with aeria is not working:

docker: Error response from daemon: failed to create endpoint pihole on network aeria: NetworkDriver.CreateEndpoint: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused')).

Both works with the stable nethserver-docker version, I retested it.

stephdl · 2020-06-03T19:44:29Z

Portainer is not so stable I think, maybe a version related issue, for pihole I think it works I am playing with it, but aeria alone, I think macvlan and aeria are incompatible or at least not on the same bridge

stephdl · 2020-06-03T19:46:33Z

the live-restore solved the upgrade issue with aeria, the container, aeria, docker are up after the upgrade

but if I restart the server, the service is down.... I think I want to switch pihole to macvlan

pagaille · 2020-06-03T19:55:11Z

On my side pihole works on Aeria but sometimes stop resolving anything for an unknown reason. Did try on aqua as well without success. Maybe I should delete the persistent volumes, it could be a configuration error.

stephdl · 2020-06-03T19:56:47Z

Maybe I should delete the persistent volumes, it could be a configuration error.

Yes it could be related when a container doesn't want to start

mrmarkuz · 2020-06-03T19:59:03Z

If pihole accepts a fixed IP it should work with macvlan.

The same portainer version works with stable nethserver-docker so I think there's an issue in this pull request...

stephdl · 2020-06-03T20:04:59Z

hum

testing portainer with default docker, it doesn't start even without created another docker network

[root@ns7dev9 ~]# rpm -qa | grep docker
docker-ce-cli-19.03.8-3.el7.x86_64
nethserver-docker-1.0.1-1.ns7.noarch
docker-ce-19.03.8-3.el7.x86_64

[root@ns7dev9 ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
6d0b30b4d351        aqua                bridge              local
f5fa42c2197e        host                host                local
328b3560ff4f        none                null                local

803e86fada17 portainer/portainer "/portainer" 2 minutes ago Restarting (2) 51 seconds ago portainer

not sure my version has changed something

stephdl · 2020-06-03T20:09:40Z

Removing everything under [root@ns7dev9 ~]# rm /var/lib/nethserver/portainer/* -rf and the container can be created

nethbot · 2020-06-04T20:01:57Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.12.pr12.gba07003.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-04T20:06:13Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.11.pr12.g8304190.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-04T20:11:55Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.11.pr12.g749a290.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-06T12:56:26Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.9.pr12.g494500d.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-06T20:26:23Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.10.pr12.g4d452e6.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-10T19:37:13Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.10.pr12.g4d452e6.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-10T20:44:50Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.11.pr12.g7501b60.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-13T09:55:33Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.1-1.17.pr12.g0aa23d6.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-13T11:02:03Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.2-1.13.pr12.g66899bd.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-13T11:26:56Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.2-1.13.pr12.g641e870.ns7.noarch.rpm x86_64 armhfp aarch64

nethbot · 2020-06-13T13:06:00Z

in 7.8.2003/nethforge-autobuild:

nethserver-docker-1.0.2-1.14.pr12.g13a484b.ns7.noarch.rpm x86_64 armhfp aarch64

shorewall policy for macvlan

4f6a34a

stephdl added 5 commits June 2, 2020 23:39

esmith properties for macvlan

e3ade76

action for macvlan network

33907f3

remove martian messages for macvlan

e51d2cf

create networking for macvlan0

5f7b836

Createlinks create macvlan

9e70331

stephdl force-pushed the macvlan branch from 01eb177 to 9e70331 Compare June 2, 2020 21:39

Fix macvlan network creation

125895e

stephdl force-pushed the macvlan branch from 41ef0fa to 125895e Compare June 3, 2020 13:29

stephdl requested a review from mrmarkuz June 3, 2020 13:32

mrmarkuz approved these changes Jun 3, 2020

View reviewed changes

stephdl force-pushed the macvlan branch from 72faffa to e927de2 Compare June 4, 2020 20:04

stephdl force-pushed the macvlan branch from e927de2 to aa8dacc Compare June 4, 2020 20:10

stephdl force-pushed the macvlan branch from aa8dacc to 48c54b3 Compare June 6, 2020 12:49

Rename mac shorewall zone to macvl

b8286b0

stephdl force-pushed the macvlan branch from 48c54b3 to b8286b0 Compare June 6, 2020 12:55

Do not have twice IP setting for macvlan0

34152c1

Restart docker at the end of interface-update

ec3f771

stephdl force-pushed the macvlan branch from 6fd0752 to ec3f771 Compare June 10, 2020 20:43

stephdl added 2 commits June 13, 2020 13:24

macVlan requires a bridge

9a83ea2

Docker must be restarted with network service

eb3e170

stephdl force-pushed the macvlan branch from a899292 to eb3e170 Compare June 13, 2020 11:25

Restart container on macvlan after docker restart

8c21988

stephdl merged commit dca658b into NethServer:master Jun 13, 2020

stephdl deleted the macvlan branch June 13, 2020 14:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network macvlan for containers #12

Network macvlan for containers #12

stephdl commented Jun 2, 2020 •

edited

nethbot commented Jun 2, 2020

nethbot commented Jun 2, 2020

nethbot commented Jun 2, 2020

nethbot commented Jun 3, 2020

mrmarkuz left a comment

stephdl commented Jun 3, 2020 •

edited

mrmarkuz commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020

nethbot commented Jun 3, 2020

nethbot commented Jun 3, 2020

stephdl commented Jun 3, 2020

mrmarkuz commented Jun 3, 2020

stephdl commented Jun 3, 2020

pagaille commented Jun 3, 2020

stephdl commented Jun 3, 2020 •

edited

mrmarkuz commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020

stephdl commented Jun 3, 2020 •

edited

pagaille commented Jun 3, 2020

stephdl commented Jun 3, 2020

mrmarkuz commented Jun 3, 2020

stephdl commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 6, 2020

nethbot commented Jun 6, 2020

nethbot commented Jun 10, 2020

nethbot commented Jun 10, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

Network macvlan for containers #12

Network macvlan for containers #12

Conversation

stephdl commented Jun 2, 2020 • edited

nethbot commented Jun 2, 2020

nethbot commented Jun 2, 2020

nethbot commented Jun 2, 2020

nethbot commented Jun 3, 2020

mrmarkuz left a comment

Choose a reason for hiding this comment

stephdl commented Jun 3, 2020 • edited

mrmarkuz commented Jun 3, 2020 • edited

stephdl commented Jun 3, 2020

nethbot commented Jun 3, 2020

nethbot commented Jun 3, 2020

stephdl commented Jun 3, 2020

mrmarkuz commented Jun 3, 2020

stephdl commented Jun 3, 2020

pagaille commented Jun 3, 2020

stephdl commented Jun 3, 2020 • edited

mrmarkuz commented Jun 3, 2020 • edited

stephdl commented Jun 3, 2020

stephdl commented Jun 3, 2020 • edited

pagaille commented Jun 3, 2020

stephdl commented Jun 3, 2020

mrmarkuz commented Jun 3, 2020

stephdl commented Jun 3, 2020 • edited

stephdl commented Jun 3, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 4, 2020

nethbot commented Jun 6, 2020

nethbot commented Jun 6, 2020

nethbot commented Jun 10, 2020

nethbot commented Jun 10, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

nethbot commented Jun 13, 2020

stephdl commented Jun 2, 2020 •

edited

stephdl commented Jun 3, 2020 •

edited

mrmarkuz commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020 •

edited

mrmarkuz commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020 •

edited

stephdl commented Jun 3, 2020 •

edited