Phantom docker network #1743
Comments
|
Do you see any relevant warning/error logs in your daemon logs ? |
|
@aboch Nothing particularly enlightening: |
|
@aboch Not much more useful with debug enabled, either: |
|
I was referring to the logs when you remove the first macvlan network, if you can reproduce the issue. If this issue is blocking you, you need to remove the |
|
Yeah, I think what you mean is in the second log:
Unfortunately, I can't seem to remove the network it's referring to. I think your other solution may be the only option. |
|
I am referring to the first time the delete of the network |
|
Thanks. |
|
Having the same issue. I have a virtual interface, created by kvm, and a macvlan network created by compose with that interface VLAN.10 as its parent: Last week i having that issue while trying to But, removing the VLAN tag from parent interface, fixes the issue, but my project cant work with that network configuration.
Removing |
|
@sashgorokhov I was not using VLAN tagging in my environment. |
|
Unfortunately I'm encountering the same issue on Docker 17.07.0-ce, build 87847530f7 using the macvlan network that is linked to a TAP VPN (openvpn). The log files only mention: The original docker network was created using: This error started appearing after restarting the system (using systemd) while the docker network was fully configured and in use. The solution suggested by aboch, of removing My solution is to start docker before openvpn so that it will complain that it could not create the network but when I issue the |
|
This sounds a lot like a bug that I've squashed in my docker-ce fork, which had plagued me since first experimenting with ipvlan in global/swarm-scoped mode ~ docker-ce 17.06. The only semi-official workaround (from @mavenugo at Docker) was to nuke the libnetwork kv store file, so I went a bit deeper and found that cleanup is not occurring at or around shutdown, and so I decided to try cleanup of duplicates at creation time instead. My PR #2055 may be a good starting point for an officlal fix. The related issue I opened is #2056 (cleanup duplicate ipvlan and macvlan network IDs during createNetwork) It may not be the perfect fix, but it resolved this issue in my case -- I'm using ipvlan, but I suspect its the same issue with macvlan - |
|
@eyz yes, your PR fix this problem for me. |
|
We are facing the same issue. When is the fix planned? EDIT 1: EDIT 2: The above mentioned workaround of deleting "/var/lib/docker/network/files/local-kv.db" file and restarting docker has been tested for more than a fortnight now and we are not facing the original issue anymore. So, anyone facing this issue can probably use this workaround. This probably is a genuine issue in docker and needs to be fixed. |
|
experiencing the same issue on docker 18.06 |
|
@mavenugo @fcrisciani What is your opinion on the following diff. Gist with some comments in here. https://gist.github.com/rnataraja/d987b61d8acc8ac738641f55ebe5af0d |
|
@rnataraja thx! |
|
+1 |
|
still hitting this on |
|
still here in 18.09.5 |
|
+1 |
This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6f234db) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby/moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6f234db9fef23c591d8376f96db062e7107b658f) Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: b6190c2713623ab455d29da4771b684e4eafc63f Component: engine
|
well， same issue happen on 18.03.1-ce，power pc，I deleted the |
|
@dhxsy1994 Yes, but what version of docker are you running? The fix was only released a few days ago, so you'd have to be using a very cutting-edge release for it to be fixed for you. |
Thanks for reply，due to the IBM doing support for the power pc architecture，the docker for power |
|
@dhxsy1994 You're still well behind. You need to download and compile the latest version of Docker to get this. It's in version 19.03. |
ok！I will do it in a Non-production environment, do some tests and change the version of docker. |
full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: zach <Zachary.Joyner@linux.com>
|
I just got this again in docker v |
|
This happens again in Docker version 19.03.2, build 6a30dfca03 |
|
@sanga @lkytal can you please share the cmd to reproduce this issue |
|
Just came across this thread. Also happening to me in Docker version 19.03.5, build 633a0ea I deleted /var/lib/docker/network/files/local-kv.db after taking backup of it and rebooted the docker host. After reboot docker started properly and the macvlan network was able to get created and operate normally. |
|
Came across this with docker version 19.03.12, build 48a66213fe How robust is deleting /var/lib/docker/network/files/local-kv.db and restarting docker? Trying to reproduce. |
|
Having the same issue right now. |
|
Same jazz for me. |
This commit carries forward the work done in moby/libnetwork#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby/libnetwork#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
Deleting that file will also delete other bridge networks which you have created. Be careful while deleting it. Take a backup of the file. |
|
I also had this issue, after an abnormal shutdown, with |
|
I'm also experiencing this in 20.10.21 :( As mentioned here moby/moby#35101 (comment) |
|
Also experiencing this issue: Deleting Also mentioned this in moby/moby#35101 (comment) |
|
In my search to figure out about this phantom networking issue this was one of the only things to come up. Using: Docker version 24.0.7, build afdd53b My eth name updates every so often in debian 11. Caused an issue with adguard docker container not to start since specifically called for in macvlan. Well I guess on a restart it didn't close properly, or I didn't restart properly 1. (restarting debian properly is almost impossible at this point) caused a phantom network to lag around. Couldn't be found with 'docker network ls' couldn't force rm either. Couldn't be found. Deleting /var/lib/docker/network/files/local-kv.db and restarting docker fixed it. Will have to recreate networks. Make sure to take note/backup ranges if you have any specifics. ipv4/ipv6. Restarting docker can cause issues with containers returning. So if you have a lot of containers may have to start them back up also. Maybe I could have restarted PC, but that would mean more downtime. Hope this helps someone else in their searchings. My original error: |
and others
I've created a docker network with a
-o parent=option, and then removed it and rebooted the machine, and now I cannot add the network back. Instead I get the error:I checked to make sure that the network was not listed in the docker networks:
I tried again removing the non-existent docker network:
And I tried again:
How could a network exist in the network test but not be listed in the network list? How do I reset it?
The text was updated successfully, but these errors were encountered: