Cannot remove network due to task #31068
Comments
|
We are experiencing the same with the error message of:
Grepping the /var/log/messages we have found a reference to the task which says the following:
docker version docker info docker network inspect |
|
Yes @thinkhard-j-park report looks a duplicate of #31066, I am assuming a manual docker run was attempted on that attachable network and failed the allocation. In @silveraid report instead, I am not sure why a task allocation would fail with an error that should happen during a network allocation phase only(subnet chosen for the network overlap with an existing one). I need to double check the code. |
|
I'm experiencing exactly same issue as @silveraid |
|
@ahmedsajid are you still experiencing this issue on the current (17.03) release? |
|
I'm experiencing same issue running root@dk1w:~# docker info
Containers: 14
Running: 10
Paused: 0
Stopped: 4
Images: 21
Server Version: 17.03.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 255
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: active
NodeID: l5ug7tsu7wyjd2n1qeersvo0u
Is Manager: false
Node Address: 192.168.100.211
Manager Addresses:
192.168.100.201:2377
192.168.100.202:2377
192.168.100.203:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-64-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.823 GiB
Name: dk1w
ID: ......
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: sitamet
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: true
Insecure Registries:
registry.......
127.0.0.0/8
Live Restore Enabled: false |
|
Update: my bad, actually, I had a container (not started with
Note that I removed all the services and then inspecting the task gives me: |
|
We are seeing the same behavior immediately after an upgrade from 17.03 to 17.06: No fixes have been discovered yet. Will comment if we manage to figure out a way. |
|
Be sure to check if the task is not running on a different node; i'm not sure if it's shown in the inspect output if it is |
|
I have the same issue on a single node swarm, with no container running. Using 17.06.1-ce. It happened when it failed to connect an existing container (that was removed later) to an overlay network. The only way I found to remove the network was to re-initialize the swarm (leave then init) |
|
Sorry @thaJeztah I missed your comment. Yes, the task is not defined on any of our manager nodes; it seems to simply not exist. We can't inspect tasks from worker nodes but have confirmed that the Right now our only workaround has been to deploy our stack to a second |
thaJeztah
added
the
kind/bug
|
Same issue with dtr-ol network and it is preventing me from (re)installing dtr. Seeing others have this issue makes it seem like Docker Datacenter is not production ready. |
|
@thaJeztah is there any fix expected for this issue ? Or the only painful workaround is to reinitialize the swarm ? |
|
I had to restart the docker daemon on the swarm master to get rid of the task |
Reproduce issue of network deletiondocker network rm dtr-ol Server Version: 17.06.2-ee-6 Expected solution to allow user to remove network by force and allow user to inspect task: docker network rm -fThis PR #35246 looks make sense, would you merge into docker EE 17.06.2-ee-6. Workaroud
|
|
Facing the same issue. Error response from daemon: rpc error: code = FailedPrecondition desc = network bbosggv6eg8o3342py6w5acsa is in use by task 1qpmy1luiijre5m11nus6m8td |
|
@ashish235 is the network "attachable", and is a container (that's not part of a swarm service) attached to that network? |
|
@thaJeztah , yes. The n/w is an attachable one nut no other service was attached to it. All the stacks running on cluster, I removed them. So there were 0 containers running . |
|
Hello @thaJeztah |
|
I have inspected task but cannot delete. As workaround i destroyed swarm cluster and recreated again |
|
After recreating swarm issue is present again Will downgrade docker |
|
After downgrade to 17.12.1-ce i can delte and create attachable swarm network but other error is present Also i:
And issue is still exists. |
|
Issue also reproduced after switching to weave network |
|
I resolved issue after switching IP network mask from /27 to /23 |
|
Just reproduced with 18.03: From the worker itself, the task has been running for some time (the service was removed more than 20 minutes ago): Version: |
|
I have same issue in Docker 18.06 (( |
|
Experiencing this on 18.09.0 as well. We attached a manually started container to the network Is there any fix or workaround to get rid of this? |
|
Is that container still attached / running? If so, the error is legitimate, as it won't remove networks if they're still in use. |
No, the container is neither running nor attached. In fact it doesn't even exist anymore. My only chance was killing that task using tasknuke: https://hub.docker.com/r/dperny/tasknuke/ |
|
I upgrade docker engine from 18.09 to 18.09.3 have same issue. Version Network Issue problem network inspect docker object info No any container is neither running nor attached. |
|
Been having occasionally the same issue when working with stacks. I can reproduce this when running script that starts swarm stack and then connects few containers to this stack's network. This issue rises when I CTRL-c out of this script after stack has been created and maybe one or two container has been connected to the stack's network. After CTRL-c, I can then prune everything related to stack and these spawned containers, but just can't delete the network even though there are no containers or services running related to it. I can write a minimal script for helping to debug this, but since this occurs on a work-related script I need to rewrite something similar. Necessary info: |
Did you check if there's a stopped container that's attached? |
|
@thaJeztah Yeah. When this happens and there are some containers running or close, I usually just run |
|
I see your daemon version is a few versions behind; might be worth upgrading to the latest version to be sure it's not an issue that was fixed already 🤔 |
|
Decided to tackle with this on my home computer, and seems like that I can reproduce this with the latest versions and also in a new freshly installed docker. I try to write some MVP reproducible script for debugging this, since currently I'm not entirely sure if there is something funky happening in the container itself or is this related to somewhere else. |
|
had same issue on macbook, just had to restart Docker Desktop, then And recreate whatever thereafter. |
|
Same on recent docker-ce 19.03.11! Looks like some stale info. However, no traces of running container on node xc7nyfx1lyyunfty4vwvwjje0, no files, only those records in logfile: You can't delete this container or task, but can inspect it: Version: |
|
Same on docker-ce 19.03.11 & 19.03.12, no running containers $docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ep27n7o2tet0iov37de2ykq0u vm-kvm6-app Ready Active 19.03.11
9ry5t46doovjch00c6hr55jgt * vm-kvm7-app Ready Active Leader 19.03.11
nijew7qb4ayiv0bomb67sx6ku vm-kvm8-app Ready Active 19.03.12$docker network rm hub-network
Error response from daemon: rpc error: code = FailedPrecondition desc = network gvo1cdcball903nh7izvw62zh is in use by task hwhfxvwgp4cq8t9yzynqmwuqc |
I removed a node from the swarm network, then I successfully removed the network. |
|
so this bug existed for four years now, still not fixed? |
|
The problem is not yet solved I have this problem regularly, I start the containers with fixed IP. After some time of cleaning all containers and then deploy again some IP addresses stay allocated giving the error: |
|
Issue still persists in 20.10.8. Currently we restart the docker daemons and that will allow us to remove the network. Really only useful in testing and is no real solution long term. |
|
Same issue here |
|
Same issue here with 20.10.14 |
|
Yep, bringing this up too. Still exists in Ubuntu 22.04. What's the workaround to fix this? Do I REALLY have to recreate my cluster? O.o |
|
First You have to remove all the service attached to the network Next you have to remove the network you want to delete |
|
Same issue with 20.10.24. Using swarmctl, I can retrieve many tasks with service name
|
|
Witnessing same issue using Docker 24.0.7. I was able to remove network after restarting Docker daemon (in single node swarm) |
The only thing that works...Thanks. |
and others
Description
Steps to reproduce the issue:
Describe the results you received:
docker network rm cluster-network
Error response from daemon: rpc error: code = 9 desc = network qytxrqgp7pw1915tqhdnkd4si is in use by task 8ruj7pjh65g9du0m1y7ce476i
Describe the results you expected:
Want to delete network with proper descritpion. what is the task?
Additional information you deem important (e.g. issue happens only occasionally):
[ { "Name": "cluster-network", "Id": "qytxrqgp7pw1915tqhdnkd4si", "Created": "0001-01-01T00:00:00Z", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": true, "Containers": null, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4097" }, "Labels": null } ]Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
VMware vshpere, ubuntu16.04 host
The text was updated successfully, but these errors were encountered: