compose scale (on swarm) with volumes_from

[arthurtsang]

hi all,

I have the following docker-compose.yml

dataVolume:
  image: data-volume
  command: sh -c "while true; do echo "hello"; sleep 1d; done;"

app:
  image: app
  volumes_from:
    - dataVolume
  ports:
    - 9600:9600

I have a 2-node docker swarm setup using consul as backend. when i issue, docker-compose up -d, i would have both dataVolume & app running on the same node.

then, when i tries to scale it, docker-compose dataVolume=2 app=2, it started another container in the 2nd node (appears to be by chance, cause it could have started on the first node too) and when trying to start app on node 2 (i've specified the port so it has to start in node 2), i got an error like this.

Service app specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating app_app_2...
Unable to find a node fulfilling all dependencies: --volumes-from=e7fa944015a72d9b53f4fd43ec4a88747136dbaad30b531907bd7be2d7d651b4 --volumes-from=6871736bb6f1380a31ccb631d88b0a0b260bb80aec7c2ec18487056917c624ed

and i've verified that those 2 volumes-from are the container id of the two dataVolume but that are on different nodes, i guess that's why it couldn't find it.

Now the question is, how to get it to work? the scenario is, I have an application that needs to read in the SSL certificates & keys. I'm trying to put that onto a container and when i scale it, i'd need the data container (ssl certs) to be scaled too.

Thanks,
Arthur Tsang

[aanand]

Yeah, this won't work with Swarm's scheduling constraints as they currently function.

The problem is that the scheduling constraint you want to express is "each app container should be on the same host as a dataVolume container". Furthermore, you probably want your app containers spread across hosts, and a 1-1 pairing of app and dataVolume containers.

This is the use case for the "pods" concept discussed in moby/moby#8781. It's tough to achieve with Compose and Swarm as they currently work.

As a workaround for now, you could try using an explicit host path instead of volumes_from, which you'd bind-mount into both services, and scale the dataVolume service up to the number of nodes, so as to ensure there's one on every node (as long as you're using the default "spread" strategy, that is).

ping @vieux and @aluzzardi who might have better ideas.

[arthurtsang]

thanks @aanand . instead of using data volume, i've decided to use consul/vault, esp i'm already using consul for swarm.

but it is an odd behavior when i use volume from, and scale together, it somehow not satisfy with finding one of the data volume but tries to find all data volumes.

[aluzzardi]

Unable to find a node fulfilling all dependencies: --volumes-from=e7fa944015a72d9b53f4fd43ec4a88747136dbaad30b531907bd7be2d7d651b4 --volumes-from=6871736bb6f1380a31ccb631d88b0a0b260bb80aec7c2ec18487056917c624ed

Looks like the app container was started with two --volumes-from (one per dataVolume) when scaling up.

@aanand How does volumes from and scaling work together?

[aanand]

I don't think it's ever been formally stated how volumes_from and scaling interact, but it looks like doing volumes_from every container in the service is indeed the current behaviour.

Which is nonsense - the volumes from each container would just override the previous one! So we should definitely change it to pick just one. But which one?

[arthurtsang]

how about the first one it could mount?