Unable to run service with network_mode "service:NAME"

[nickschuch]

Description

This ticket is a duplicate of #8766 given it was marked as "closed" but is not resolved.

Unable run to execute compose run <service> when a service is using network_mode: service:nginx

Steps to reproduce the issue:

This is a docker-compose.yml file which can replicate the issue.

version: '3'

services:
  nginx:
    image: nginx

  cli:
    image: alpine
    network_mode: service:nginx

Commands which replicate the issue.

$ compose up -d
$ compose run cli sh
Error response from daemon: network service:nginx not found

Describe the results you received:

Error response from daemon: network service:nginx not found

Describe the results you expected:

The service container to execute within the same network as the "nginx" service in this case.

This worked with Compose v1.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version v2.2.2

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 129
  Running: 1
  Paused: 0
  Stopped: 128
 Images: 169
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.5-76051505-generic
 Operating System: Pop!_OS 21.10
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.32GiB
 Name: batman
 ID: QAXJ:HDRA:DGM3:DR75:OBYA:VOV7:EEOE:K5IG:QD57:54I7:SVH6:5EQ3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: nickschuch
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

N/A

[ndeloof]

just tested with Docker Compose version v2.2.2
"works for me"

 → docker-compose run cli sh
[+] Running 7/7
 ⠿ nginx Pulled                                                                                                                                                                                                     44.6s
   ⠿ e5ae68f74026 Pull complete                                                                                                                                                                                     [+] Running 2/2
 ⠿ Network truc_default    Created                                                                                                                                                                                   0.0s
 ⠿ Container truc-nginx-1  Created                                                                                                                                                                                   0.1s
[+] Running 1/1
 ⠿ Container truc-nginx-1  Started                                                                                                                                                                                   0.4s
[+] Running 2/2
 ⠿ cli Pulled                                                                                                                                                                                                        5.9s
   ⠿ 59bf1c3509f3 Pull complete                                                                                                                                                                                      2.8s
/ # wget localhost
Connecting to localhost (127.0.0.1:80)
saving to 'index.html'
index.html           100% |**************************************************************************************************************************************************************************|   615  0:00:00 ETA
'index.html' saved
/ # cat index.html 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

[nickschuch]

Looks like it works if you skip straight to running docker-compose run cli sh on a stack which has not been up'd yet.

If you follow the steps to reproduce (documented in description) its:

• Up the stack
• Then run the CLI

[nickschuch]

Is there somewhere I can start looking to work on this issue?

[vrusua]

I have exactly the same issue, Docker Desktop for Mac (Engine: 20.10.12, Compose: v2.2.3). Did anyone sorted this? Thanks.

[vrusua]

@nickschuch

just tried to add the custom network name and add one to both services in networks:

version: '3.9'

services:
  nginx:
    image: nginx
    networks:
      - nginx-public
      
  cli:
    image: alpine
    networks:
      - nginx-public
    network_mode: service:nginx

networks:
  nginx-public:
      name: nginx-public

Please check over how it works on your side.

[kevin-bockman]

For some reason, I don't see a +1 emoji here..

I have a user experiencing the same issue (replicated using the example initially provided in the issue)

Docker Compose version v2.2.3
Docker server version 20.10.12

They're using Ubuntu under WSL2.

I just had them revert to using the separately installed docker-compose CLI (1.29.2, build 5becea4) and that's working.

@vrusua example is invalid because you can't specify networks and network_mode together in the same service. Switching to using networks isn't a viable workaround for us either, as that's a completely different method. We want to be able to use network_mode as a service so the services can be addressed using localhost to each other.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jbrwilkinson]

We also have this issue especially on Docker Desktop for macOS; colleagues with Docker Compose v1.* can work fine, Docker Compose v2.* fails with: Error response from daemon: network service:<service name> not found

[jbrwilkinson]

Unchecking [ ] Use Docker Compose v2 in macOS Docker Desktop preferences works around this when using docker-compose, but docker compose still fails.

[arren-ru]

I just found the same issue with that test env

version: "3"
services:
  pod:
    image: k8s.gcr.io/pause:3.8
  cli:
    image: busybox
    depends_on: [nginx]
    network_mode: service:pod
  nginx:
    image: nginx:alpine
    network_mode: service:pod

docker-compose V1.29 runs it ok

docker-compose run cli
Creating network "test_default" with the default driver
Pulling pod (k8s.gcr.io/pause:3.8)...
3.8: Pulling from pause
c7170791d244: Pull complete
Digest: sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d
Status: Downloaded newer image for k8s.gcr.io/pause:3.8
Pulling nginx (nginx:alpine)...
alpine: Pulling from library/nginx
9b18e9b68314: Pull complete
51048c2d0108: Pull complete
3897c47760f3: Pull complete
f61a6b717778: Pull complete
56ed090e0ec8: Pull complete
e5a0a61a0146: Pull complete
Digest: sha256:082f8c10bd47b6acc8ef15ae61ae45dd8fde0e9f389a8b5cb23c37408642bf5d
Status: Downloaded newer image for nginx:alpine
Pulling cli (busybox:)...
latest: Pulling from library/busybox
4189ef966ca4: Pull complete
Digest: sha256:20142e89dab967c01765b0aea3be4cec3a5957cc330f061e5503ef6168ae6613
Status: Downloaded newer image for busybox:latest
Creating test_pod_1 ... done
Creating test_nginx_1 ... done
Creating test_cli_run ... done
/ #

but V2 fails

docker compose run cli
[+] Running 3/3
 ⠿ Network test_default    Created             0.0s
 ⠿ Container test-pod-1    Created             0.0s
 ⠿ Container test-nginx-1  Created             0.0s
[+] Running 2/1
 ⠿ Container test-pod-1    Started             0.2s
 ⠿ Container test-nginx-1  Started             0.1s
Error response from daemon: network service:pod not found

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[kevin-bockman]

Hi @ndeloof - this has been an issue for almost a year - see #8766 - we've had this issue as well. I did some digging and I found out some additional info. Hope this helps.

This fails on v2 if the network (EDIT: IS) already setup, so just doing a docker compose run first will not trigger this issue. Even adding a depends_on doesn't help because the container actually does get created either way, so that's not the issue. This solely seems to be related to whether or not the network is created at the same time as the docker compose run.

I've tested this to be an issue on all docker-compose version v2, including the latest 2.11.1. It works on the latest v1 (1.29.2)

Test file:

version: '3'

services:
  nginx:
    image: nginx

  cli:
    image: alpine
    network_mode: service:nginx

idunno% /usr/local/bin/docker-compose run cli sh
Creating docker-compose-test_nginx_1 ... done
Creating docker-compose-test_cli_run ... done
/ # exit
idunno% /usr/local/bin/docker-compose run cli sh
Creating docker-compose-test_cli_run ... done
/ # exit
idunno% /usr/local/bin/docker-compose --version 
docker-compose version 1.29.2, build 5becea4c

Docker Compose version v2.11.1
WORKS: start directly with a run so it creates the network
idunno% docker compose run cli sh
[+] Running 2/2
 ⠿ Network docker-compose-test_default    Created                                                                                                                                                     0.1s
 ⠿ Container docker-compose-test-nginx-1  Created                                                                                                                                                     0.2s
[+] Running 1/1
 ⠿ Container docker-compose-test-nginx-1  Started                                                                                                                                                     0.7s
/ # exit

FAIL: trying to run with the network already setup
idunno% docker compose run cli sh
[+] Running 1/0
 ⠿ Container docker-compose-test-nginx-1  Running                                                                                                                                                     0.0s
Error response from daemon: network service:nginx not found

idunno% docker compose stop      
[+] Running 1/1
 ⠿ Container docker-compose-test-nginx-1  Stopped                                                                                                                                                     0.4s
idunno% docker compose rm -f -v
Going to remove docker-compose-test-nginx-1
[+] Running 1/0
 ⠿ Container docker-compose-test-nginx-1  Removed                                                                                                                                                     0.1s
idunno% docker compose down -v 
[+] Running 1/1
 ⠿ Network docker-compose-test_default  Removed                                                                                                                                                       0.3s

WORKS: trying again after doing a cleanup (remove container & network)
idunno% docker compose run cli sh 
[+] Running 2/2
 ⠿ Network docker-compose-test_default    Created                                                                                                                                                     0.0s
 ⠿ Container docker-compose-test-nginx-1  Created                                                                                                                                                     0.0s
[+] Running 1/1
 ⠿ Container docker-compose-test-nginx-1  Started                                                                                                                                                     0.4s
/ #

[hendrics]

Just a note, It has been working on v1 for many years, afaik. This is a blocker using v2.

[i-ky]

Here is v1 code:

compose/compose/service.py

Lines 1480 to 1482 in 30fcb72

	containers = self.service.containers()
	if containers:
	return 'container:' + containers[0].id

and here is v2 code:

compose/pkg/compose/convergence.go

Lines 129 to 140 in 7cf5940

	cnt := cnts[0]
	serviceName := cnt.Labels[api.ServiceLabel]
	if d := getDependentServiceFromMode(service.NetworkMode); d == serviceName {
	service.NetworkMode = types.NetworkModeContainerPrefix + cnt.ID
	}
	if d := getDependentServiceFromMode(service.Ipc); d == serviceName {
	service.Ipc = types.NetworkModeContainerPrefix + cnt.ID
	}
	if d := getDependentServiceFromMode(service.Pid); d == serviceName {
	service.Pid = types.NetworkModeContainerPrefix + cnt.ID
	}

At a glance they do the same - take the first container from array and use its ID to replace network_mode: service:x with network_mode: container:y, but important difference is that in v1 it is an array of containers belonging to service x in contrast to array of all containers belonging to the project in v2. If "correct" container is on top of the list, then v2 code works just fine, but fails otherwise.

I will submit a PR in a moment.

[ndeloof]

cnts variable here is obtained by filtering "observed state" (i.e all project containers) by service label

compose/pkg/compose/convergence.go

Line 118 in 7cf5940

cnts := c.getObservedState(serviceName)

also, replacement only take place if network_mode: service:??? matches selected container's service name

    serviceName := cnt.Labels[api.ServiceLabel]

	if d := getDependentServiceFromMode(service.NetworkMode); d == serviceName {
		service.NetworkMode = types.NetworkModeContainerPrefix + cnt.ID
	}

[i-ky]

cnts variable here is obtained by filtering "observed state" (i.e all project containers) by service label

But not here:

compose/pkg/compose/run.go

Lines 85 to 89 in 7cf5940

	observedState, err := s.getContainers(ctx, project.Name, oneOffInclude, true)
	if err != nil {
	return "", err
	}
	updateServices(&service, observedState)

[ndeloof]

oh right, nice catch

[i-ky]

Submitted #10036. It is probably not the cleanest solution, but it seems to work.