New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The order in which networks are assigned to interfaces does not match the order they are listed in compose file #4645

Closed
larsks opened this Issue Mar 21, 2017 · 14 comments

Comments

Projects
None yet
@larsks
Copy link

larsks commented Mar 21, 2017

The following report was produced using Docker version 17.03.0-ce and compose version docker-compose version 1.11.2, build dfed245.

Given a docker-compose.yml that looks like this:

version: "2"

services:
  server:
    image: alpine
    command: sleep 999
    networks:
      - nw0
      - nw1
      - nw2
      - nw3

networks:
  nw0:
  nw1:
  nw2:
  nw3:

I would expect the networks to be assigned to interface in order, such that eth0 is attached to nw0, eth1 is attached to nw1, etc. This is exactly the behavior I see if I start a container using docker run and then attach additional networks with docker network connect.

However, when using docker-compose and the above compose file, the ordering of networks and interfaces appears to be inconsistent. Assuming that the above compose file is in a directory named nwtest, this script will demonstrate the problem:


#!/bin/sh

docker-compose up -d

for nw in 0 1 2 3; do
	nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
		nwtest_nw${nw})
	if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
		awk '$1 == "inet" {print $2}')

	nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
	if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

	echo "nw${nw} $nw_net eth${nw} ${if_net}"

	if [ "$if_net" != "$nw_net" ]; then
		echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
	fi
done

docker-compose stop

On my system, that produces as output:

Starting nwtest_server_1
nw0 192.168.32.0 eth0 192.168.32.0
nw1 192.168.48.0 eth1 192.168.48.0
nw2 192.168.64.0 eth2 192.168.80.0
MISMATCH: nw2 = 192.168.64.0, eth2 = 192.168.80.0
nw3 192.168.80.0 eth3 192.168.64.0
MISMATCH: nw3 = 192.168.80.0, eth3 = 192.168.64.0
Stopping nwtest_server_1 ... done

For comparison, here is a script that performs the same test using docker run and manual attachment:

#!/bin/sh

docker rm -f nwtest_server_1
docker run -d --name nwtest_server_1 --network nwtest_nw0 \
	alpine sleep 999

for nw in 1 2 3; do
	docker network connect nwtest_nw${nw} nwtest_server_1
done

for nw in 0 1 2 3; do
	nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
		nwtest_nw${nw})
	if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
		awk '$1 == "inet" {print $2}')

	nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
	if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

	echo "nw${nw} $nw_net eth${nw} ${if_net}"

	if [ "$if_net" != "$nw_net" ]; then
		echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
	fi
done

docker rm -f nwtest_server_1

This always runs without error.

@larsks larsks changed the title The order in which networks are assigned to interfaces does not the order they are listed in compose file The order in which networks are assigned to interfaces does not match the order they are listed in compose file Mar 21, 2017

@johnharris85

This comment has been minimized.

Copy link
Contributor

johnharris85 commented Mar 31, 2017

Initial guess would be networks is implemented as a dict. Will take a look this evening.

@tgogos

This comment has been minimized.

Copy link

tgogos commented Mar 31, 2017

Thanks @johnharris85
There is also an open bounty at SO (link): , where you can also add your answer.

@an-tex

This comment has been minimized.

Copy link

an-tex commented Apr 5, 2017

Same issue. Gets tricky when you have to rely on the order e.g. for iptables rules like "iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE"

@akshshar

This comment has been minimized.

Copy link

akshshar commented Apr 7, 2017

+1. Trying to spin up some router topologies using docker-compose. This completely screws up the connections for the protocols.

@narayan-iyengar

This comment has been minimized.

Copy link

narayan-iyengar commented May 24, 2017

+1 form me as well.
Editing my response to add some details. I thought about what @johnharris85 mentioned, so I numbered my network names in the order I want them to be connected and I can get around the issue for now.

For example part of my compose looks like this:
networks:
-- 1firstnw
-- 2nw
-- 3nw

Hope this helps someone.

@LuisPiedra

This comment has been minimized.

Copy link

LuisPiedra commented Jun 16, 2017

+1

Workaround of setting alphabetically order is not working for me in docker-compose version 1.13.0, build 1719ceb, it seems to be adding ifaces randomly.

In any case, even if alphabetically works, I still need to have the same network assigned to different interfaces in different containers, so still need to force the ordering.

As @johnharris85 mentioned, network configuration is passed to docker engine as an unordered dict if networks are created with the container. One possible implementation would be to first create the container and then connect the container to each network one by one. This behaviour could be also configurable.

@LuisPiedra

This comment has been minimized.

Copy link

LuisPiedra commented Jun 18, 2017

I have been playing with the scripts provided by @larsks , and docker engine keeps the order of the interfaces only when networks are connected to a running container. Otherwise, it will also create the interfaces randomly.

If the container is started after connecting the networks, it will also have mismatches

docker rm -f nwtest_server_1
docker create --name nwtest_server_1 --network nwtest_nw0 \
	alpine sleep 999

for nw in 1 2 3 4 5 6 7 8 9; do
	docker network connect nwtest_nw${nw} nwtest_server_1
done

docker start nwtest_server_1

Same happens if container is restarted

docker rm -f nwtest_server_1
docker run -d --name nwtest_server_1 --network nwtest_nw0 \
	alpine sleep 999

for nw in 1 2 3 4 5 6 7 8 9; do
	docker network connect nwtest_nw${nw} nwtest_server_1
done

docker stop nwtest_server_1
docker start nwtest_server_1

Anyway, even if docker engine offers support for this behaviour (this is already adressed: moby/moby#25181), docker-compose won't respect the definition order since original network list is converted to an unordered list.

In addition, if connections include parameters like a fixed ip address, networks is not a list at all, it is implemented as a dictionary. Maybe some additional parameter like order or priority would be needed if docker engine implements this behaviour at some point.

@mwiget

This comment has been minimized.

Copy link

mwiget commented Jun 27, 2017

I thought I had an easy fix for this, but it turns out docker network connect plays its own game, as @LuisPiedra pointed out correctly. It is not enough to sort the calls to connect_container_to_network() ...

@Ageraluon

This comment has been minimized.

Copy link

Ageraluon commented Mar 31, 2018

+1, any progress on this yet?

@shin-

This comment has been minimized.

Copy link
Contributor

shin- commented May 3, 2018

Issue should be fixed by #5566 - at least the Compose aspect of it.

@shin- shin- closed this May 3, 2018

@eugenepark1

This comment has been minimized.

Copy link

eugenepark1 commented Aug 5, 2018

@shin can you elaborate what exactly is the fix? there werent much details on the link

@gentunian

This comment has been minimized.

Copy link

gentunian commented Feb 9, 2019

extending @larsks example I've created this gist that demonstrate that ordering is dict-based and not yaml-based.

The gist include everything you need to run the test. Just chmod+x it.

@shin is this working on docker swarm?

@shin-

This comment has been minimized.

Copy link
Contributor

shin- commented Feb 9, 2019

https://docs.docker.com/compose/compose-file/compose-file-v2/#priority

Swarm features should be requested on the Swarm repo or Moby repo.

@gentunian

This comment has been minimized.

Copy link

gentunian commented Feb 9, 2019

@shin it seems that priority is missing in version 3, so my guess is that is deprecated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment