docker service create doesn't work when network and generic-resource are both attached #44378
Comments
Dogfalo
added
kind/bug
|
This looks simple enough to fix, but probably has to happen over at swarmkit |
AssignedGenericResources in constraint_enforcer.go were falsely checked inside a case that enforced Reservations to be set Furthermore, the if statement had a missing ! Signed-off-by: Martin Braun <braun@neuroforge.de
AssignedGenericResources in constraint_enforcer.go were falsely checked inside a case that enforced Reservations to be set Furthermore, the if statement had a missing ! Signed-off-by: Martin Braun <braun@neuroforge.de>
|
Good find! Pretty sure we ran into this as well I submitted a PR over at swarmkit :) |
|
This has been bugging me, thanks for finding it! |
…-generic-resources Fixes moby/moby#44378
|
The fix for this should be included in 23.0.2. |
|
We still encounter this bug on Docker Engine Version Here is an excerpt from Background: We run on Ubuntu20.04 with an apt-installed docker engine. Any suggestions, or anyone else having this behaviour? |
|
Hmm, this looks unrelated though @mrnicegyu11 - but can you try to share a reproducer here? I tried the original steps on a single node (local devenv) with the following and it worked: docker network create -d overlay --scope swarm test-network
docker service create --network test-network --generic-resource "gpu_test=1" --name test-service quay.io/centos/centos:stream8 bash -c "
env && sleep infinity"
docker service ps --no-trunc test-service
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
k3lrk5xjiorlorb7m7f9znyxn test-service.1 quay.io/centos/centos:stream8@sha256:59460e4360f0657a24ce56339b117f14ac236dd51e1cf33f30ed5725ba1b4429 ubuntu Running Running 16 seconds ago
docker service inspect test-service
[
{
"ID": "df0915k0vefz3f1u6zvafy2fq",
"Version": {
"Index": 5584
},
"CreatedAt": "2023-08-11T14:12:43.079778404Z",
"UpdatedAt": "2023-08-11T14:12:43.080247586Z",
"Spec": {
"Name": "test-service",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "quay.io/centos/centos:stream8@sha256:59460e4360f0657a24ce56339b117f14ac236dd51e1cf33f30ed5725ba1b4429",
"Args": [
"bash",
"-c",
"env \u0026\u0026 sleep infinity"
],
"Init": false,
"StopGracePeriod": 10000000000,
"DNSConfig": {},
"Isolation": "default"
},
"Resources": {
"Limits": {},
"Reservations": {
"GenericResources": [
{
"DiscreteResourceSpec": {
"Kind": "gpu_test",
"Value": 1
}
}
]
}
},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Platforms": [
{
"Architecture": "arm64",
"OS": "linux"
},
{
"Architecture": "ppc64le",
"OS": "linux"
},
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "vv3h7voq9m7b4k4rzovv8x9mf"
}
],
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip"
}
},
"Endpoint": {
"Spec": {
"Mode": "vip"
},
"VirtualIPs": [
{
"NetworkID": "vv3h7voq9m7b4k4rzovv8x9mf",
"Addr": "10.0.2.2/24"
}
]
}
}
]The generic resource is set up like this: "node-generic-resources": ["gpu_test=GPU-1"] |
This looks to me like an issue with your networks. Do you by any chance have a lot of overlay networks? How many containers do you have? Maybe the default network size is not big enough? |
|
@s4ke thanks a lot already for the rapid response, let me provide you with some more details about our networking in docker swarm: However, this error also occurs on an "empty" swarm with your basic example on your swarm (7 ubuntu machines). I can reproduce it with your given More background: At the time when the error happened on docker engine major versions I have ran your minimal PoC code on our cluster, for docker engine version For docker engine version Just to be clear: I reproduced the issue on a docker swarm where nothing except your given Thanks for the help / communication! :--) |
|
@dperny ideas? |
|
@mrnicegyu11 happy to help. Just as additional infos so its easier to reproduce. Can you share more about your env? Which nodes have which docker version running? I am wondering if we can build a unit test out of your Situation in moby/swarmkit. Also, I am unsure if the behaviour differs between having only a single node vs multiple. EDIT: just reread... You have no Version mismatch. Hmm |
|
@s4ke Yes, we keep our machines' docker engine version always in sync :) To progress maybe I can suggest the following: We provision our machines with ansible playbooks, so it is probably feasible for me within a reasonable time to reproduce this on a two-machine swarm (1 manager, 1 worker with generic resource), that we for example spin up vanilla on some cloud provider. If it also happens there, we would have an accessible and secluded environment to have a look at :) I will report back. If it doesnt happen there, it would be interesting to see if a full reinstall of the operating system will fix the issue on the machines of our swarm. After all, it is very weird that for us the versions where other people are affected run just fine, while the never versions have the issue even though it should be fixed. I will also mention that I just up- and downgraded the docker engine version, so the |
|
If you can try to reproduce that would be great. I will try to do the same next week or so. |
|
@mrnicegyu11 have you managed to reproduce it? |
Sorry I had to temporarily drop this for some other tasks, thanks for the gentle reminder I am trying to reproduce it now once again :) |
|
@s4ke I have managed to reproduce the problem on an isolated docker swarm created in AWS. The problem occurs with docker engine version 20, but not with version 24. I'll PN you some details for now :) Sorry for the delay! |
|
@s4ke Here is terraform & ansible code that makes the issue reproducible, at least for me :) https://github.com/ITISFoundation/minimal-example-docker-custom-constraint-bug |
|
The fix was not merged until 23.0.2, so this is expected. You'll want to update to a supported engine version to benefit from recent fixes (currently 24.0.z). I'm guessing that the "works" or "does not work" got inverted from this original statement:
|
|
@neersighted thanks for replying :--). In fact, the wording did not get mixed up. Let me clarify: In an exactly opposite scenario to what is expected from the docker engine changelogs, we seem to encounter the issue not on docker engine version 20.x, but we do encounter it on docker engine version 24.x. We are very confused by this as well, and probably I am overlooking something somewhere. Nevertheless, with our machine setup (swarm on ubuntu 20.04, with nvidia docker runtime), the problem is reproducible for me, even when spinning up cloud VM machines. We are provisioning them with ansible, so everything should be noted in code in the link I have posted in a previous comment. Let me know if I can provide some more specifics, thanks. |
|
@mrnicegyu11 Before I try to reproduce this on multiple nodes, can you verify whether this also happens on a single node for you? |
|
This is from a single node Swarm with GPU: We use this daemon.json: |
|
Force updating also works: But since there is only one resource present and the first task of the service is still running the force update actually starts with reporting "no suitable node" But it recovers from this nicely. |
|
@s4ke I will give it a shot! |
|
@s4ke I have been testing what you propose and indeed that works. but it helped me narrowing our problem. with the following: for example, which could mean that we have 5 GPUs in the node right? I would expect that everyone of my service would decrease the number of available GPUs by 1 ok? now running your example code shows ubuntu@ip-10-0-2-119:~$ cat /etc/docker/daemon.json
{
"node-generic-resources": ["NVIDIA-GPU=5"]
}
ubuntu@ip-10-0-2-119:~$ docker --version
Docker version 24.0.7, build afdd53b
ubuntu@ip-10-0-2-119:~$ docker network create -d overlay --scope swarm test-network
ir8hbeen70wdduvektz3q3xyp
ubuntu@ip-10-0-2-119:~$ docker service create --network test-network --generic-resource "NVIDIA-GPU=1" --name test-service quay.io/centos/centos:stream8 bash -c "env && sleep infinity"
wp59zh4y58fo8rrsmnnrm90xy
overall progress: 0 out of 1 tasks
1/1: assigned node no longer meets constraintsThis does not work. I cannot even start 1 service this way. What is wrong here? is the way I define the generic resources wrong? or the way the service is started? |
|
and I may add, that not attaching the network does indeed work as well docker node inspect self | jq
[
{
"ID": "296wpf736ezdstsx57djzoax1",
"Version": {
"Index": 340
},
"CreatedAt": "2024-01-18T11:38:43.97870017Z",
"UpdatedAt": "2024-01-18T11:42:19.505035908Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "ip-10-0-2-119",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 8000000000,
"MemoryBytes": 33160314880,
"GenericResources": [
{
"DiscreteResourceSpec": {
"Kind": "NVIDIA-GPU",
"Value": 5
}
}
]
},
"Engine": {
"EngineVersion": "24.0.7", |
|
Have you tried listing all GPUs separately? I will check the discrete example and get back to you though. Unsure though, how the code changed by me should have broken this. |
|
@s4ke the point is in our working system we do not list the GPUS, but the available VRAM amount. therefore we are very interested in having it working with the numbers again. Basically listing them is not realistic for us. The idea being that we have a machine with a GPU that has 11000 MB VRAM, then a user can start 1 service with 400VRAM, another with 200VRAM and a last one with 10500VRAM. Unless you have a way to do that and maybe we can adjust the syntax. |
|
I got that. What is the last version this worked on? |
|
same as what @mrnicegyu11 said, docker 20.x |
|
You are correct. If you use discrete resources, this is causing an issue. Will try to see what I can do, but this is maybe something we need the cavalry for :D @dperny |
|
I am confused. After playing around with unit tests, it looks like HasResource in validate.go has the boolean query the wrong way around for discrete resources: It is: case *api.GenericResource_DiscreteResourceSpec:
if res.GetDiscreteResourceSpec() == nil {
return false
}
if res.GetDiscreteResourceSpec().Value < rtype.DiscreteResourceSpec.Value {
return false
}It should be: if res.GetDiscreteResourceSpec() == nil {
return false
}
if res.GetDiscreteResourceSpec().Value > rtype.DiscreteResourceSpec.Value {
return false
}HasResources has the following documentation: |
|
This kinda explains that it started working for people with named resources but it stopped working for people with discrete resources as my fix inverted the boolean logic for the caller (which was correct) but I missed the wrong logic in HasResource. |
|
If the values match everything succeeds in scheduling: martinb@ubuntu:~$ docker service create --network test-network --generic-resource "gpu_test=2" --name test-service-2 quay.io/centos/centos:strea
m8 bash -c "env && sleep infinity"
m5mv5ubs1tpu7lhmrnzci0lno
overall progress: 0 out of 1 tasks
1/1: assigned node no longer meets constraints
^COperation continuing in background.
Use `docker service ps m5mv5ubs1tpu7lhmrnzci0lno` to check progress.
martinb@ubuntu:~$ docker ^C
martinb@ubuntu:~$ docker service rm test-service-2
test-service-2
martinb@ubuntu:~$ docker service create --network test-network --generic-resource "gpu_test=5" --name test-service-2 quay.io/centos/centos:strea
m8 bash -c "env && sleep infinity"
2k1vivjkv0yf0k5suzqgansbu
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged |
|
I will prepare a fix and PR it over at swarmkit. |
|
Fix is found and PR'ed over at moby/swarmkit. Thanks for the investigation @sanderegg @mrnicegyu11 |
|
Hey @s4ke thank you very much for super fast reaction time. looking forward. 💯 |
|
@s4ke do you have an idea when it would be released ? ETA? thanks again! |
|
I am merely a contributor to this project, so I don't know. Usually this is taken up quite fast for these kinds of small issues. I hope to get this into 25.0.x but this is more a question to the folks at docker/mirantis. |
|
PTAL @neersighted @thaJeztah |
|
Update: PR over at swarmkit is merged. |
|
Very strange. We actually have a situation with Docker 24.0.2 where we create the service with discrete resources and it works if we createit via a stack file. If we create the service via CLI it does not work and your issue is reproducible @sanderegg |
|
It's released. Please check if your issue is fixed @sanderegg |
|
In my tests, the issue seems to be fixed now. |
Description
Issue
In docker swarm,
--generic-resourcedoes not work when it is used alongside--network. This is due to an incorrect condition,if genericresource.HasResource(ta, available.Generic), in theconstraint_enforcer.gocode when the service is brought up.The code should read
if !genericresource.HasResource(ta, available.Generic) {so that the task which has an assigned and available GenericResource is not removed.This is bug is important as it prevents the usage of generic resources in Docker Swarm; this is particularly relevant for assigning services to nodes based on GPU availability.
The generic-resources feature used to work properly in swarm in version
18.06.1.Reproduce
Bug Investigation + Reproduction steps
This functionality was working in version
18.06.1but not in any version afterwards.Each release was tested through these steps. An additional condition that is required for the bug to occur is that the service must be being brought up on a non-manager swarm node:
/etc/docker/daemon.jsonon the worker to add an item tonode-generic-resources. Restart the docker serviceAdd a worker node to the swarm.
Create a service that with the network attached as well as a generic-resource. This step will fail and the service will never get to the running state.
docker service ps. These errors continue in a loop where a new task is created and subsequently rejected. This error does not resolve by itself and the service never reaches theRunningstate.Expected behavior
docker service createshould create a service with anetworkandgeneric-resourceattached.docker version
docker info
Additional Info
No response
The text was updated successfully, but these errors were encountered: