Define schema for mapping service resources into DNS

[stevvooe]

While there has been discussion in moby/moby#24973 and
#192, the adoption of a clear schema for mapping service resources into the DNS space is unclear.

The following presents a schema for mapping cluster-level FQDNs from various components:

Resource	Component	Structure	Examples
Cluster	<cluster>	<cluster>	local, cluster0
Namespace	<namespace>	<namespace>.<cluster>	production.cluster0, development.local, system
Node	<node>	<node>.<cluster>	node0.local
Job	<job>	<job>.<namespace>.<cluster>	job0.production.cluster0
Slot	<slot>	<slot id>.<job>.<namespace>.<cluster>	1.job0.production.cluster0
Task	<task>	<task id>.<slot id>.<job>.<namespace>.<cluster>	abcdef.1.job0.production.cluster0

@mavenugo @mrjana @aluzzardi

[stevvooe]

@aluzzardi This is related to what we discussed today.

@mavenugo Is there any progress on making this happen?

[aluzzardi]

@stevvooe How does this map with #1193 since we've adopted <e.ServiceAnnotations.Name>.<Slot>.<TaskID> as a model?

[stevvooe]

@aluzzardi That is just the naming convention, which goes left to right. DNS goes right to left. Completely compatible.

The only open item is the consistency of active slots. We need a way for tasks to discover all of the hostnames of other tasks, via DNS or other, regardless of DNS-RR or VIP mode. This will allow us to support host-identity based services, like zk, etcd, nats, etc.

[mostolog]

Perhaps I'm confused, but shouldn't DNS SRV records will perfectly match for this use case?

Regarding zookeeper config, as stated by documentation, the config file file should look like:

...
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
...

A Zookeeper image with a defined entrypoint, could execute:

nslookup -querytype=srv _zookepeer._tcp.swarmmode.com

# _service._proto.name.  TTL   class SRV priority weight port target.
_zookepeer._tcp.swarmmode.com.   100 IN    SRV 10       10     2888 zookeper.1.swarmmode.com
_zookepeer._tcp.swarmmode.com.   100 IN    SRV 10       10     2888 zookeper.2.swarmmode.com
_zookepeer._tcp.swarmmode.com.   100 IN    SRV 10       10     2888 zookeper.3.swarmmode.com

And echo records for each server to config file.

Each time a service-task is executed, it should publish SRV entries at swarm internal DNS, and when stopped, removed from it. Of course, I dare to say this without having any clue about Docker internal DNS (or whatever it uses)

Regards.

[stevvooe]

@mostolog That is somewhat the goal here but we first need a plan for mapping these into the SRV records in a consistent manner. At this point, it is fairly ad-hoc, which is very disappointing.

[mostolog]

I have been reading #192 and trying to reply, but at the end I just think I probably lack the needed knowledge to discuss this.

I'll only say that, IMHO, it makes much more sense to have abcdef-1.job-0.cluster-0 rather than abcdef.1.job0.production.cluster0.

[mostolog]

@stevvooe Somehow related to this:

docker run -h myhost...

adds an entry on /etc/host with: ip container-name
Is there any way to specify a domain or set this entry to FQDN? ie: ip myhost.domain.com on /etc/host

Seems doing:

docker run -h myhost.domain.com...

it's not very polite, cause hostname should be only myhost

Is this a missing feature I could request on https://github.com/docker/docker ? Am I missing something?
Thanks

[stevvooe]

#192 is a little ambitious. I wrote that proposal very early on. We mostly keep it open to consider some of the namespacing concepts, but let's not consider that a hard fast model.

Like I've said before, this isn't about a single fix. There needs to be a concerted effort to manage the DNS name mapping.

[vovimayhem]

Will this allow the routing mesh to route requests to different services running on the same port? Is that even being planned for the routing mesh dns stuff???

[stevvooe]

@vovimayhem No, this is more about mapping services into DNS. To multiplex services on the same port, you'll need to introduce an L7 load balancer to manage that in your infrastructure.

[vovimayhem]

Thank you for clarifying that! I've been searching for clear info (a confirmation) about this for almost a month now!

[mostolog]

@stevvooe I'm back! 😝
Have you considered enabling DNS registering in out-of-docker DNS server?

It will also be interesting being able to register under a specific domain tree
Background-related to this: could a docker node create multiple swarm clusters (each having a domain scope)?

[stevvooe]

@mostolog Let's keep the discussion focused on the proposal at hand. The presented ideas are interesting but they are orthogonal to the goal of creating a clear DNS-based service discovery, which is the topic at hand.

[doxxx]

I like your naming scheme proposal. What I'm wondering is if a container's canonical fully-qualified hostname could be the proposed task DNS name instead of simply the container ID? For example, if some application in the container does the equivalent of gethostname() and then getaddrinfo() it should return abcdef.1.job0.production.cluster0 instead of simply abcdef.

This would allow applications in containers to provide more useful hostnames to other applications, e.g. Apache Spark.

[stevvooe]

@doxxx Currently, docker containers have an expectation that their hostname is the container id (not the task id). This is insufficient, as it is not unique (truncated to only 48 bits of entropy!) and provide no notion of location. I am not sure if we can change this to something more correct inside the container.

With the way UTS namespace work, I would expect us to set the hostname to the abcdef and the domainname to 1.job0.production.cluster0 (corresponding to the slot). Assembling these would require calling gethostname and getdomain, resulting in the FQDN for the task.

[mostolog]

Don't know if close-related or related-enough to be considered...

I'm starting to get some "unfriendly" experiences with containers having too-long hostnames like: project-service-swarmnodeidwhichactuallyisquitelong-#slot created using 2.13 template naming. eg: {{.Node.ID}}.

Perhaps it would be great to be able to allow "nearest" container resolution a partial name (not FQDN).
ie: A configuration asking for "mysql" running on a container under "com.domain.app" should look for "com.domain.mysql" while another running under "com.domain.sub.whatever.app" "com.domain.sub.whatever.mysql"

Is that already designed that way? Does it make sense? It is not related at all? Thanks ;)

[stevvooe]

@mostolog I think that is a reasonable assumption. One example might be having task 1 hit task 2 with just 2, since they are in the same job. We already do this to some degree but there will need to be clever setup in the domain naming approach.

[mostolog]

Thanks!

[mostolog]

Hi again.

Reviewing my notes I just confirmed docker stack deploy --compose-file stack.yml mystack creates services named like:

mystack_fooservice
mystack_barservice

Already asked if dash can be configured instead of underscore on forums, but I was wondering if this won't have any effects on this issue (aka: define a FQDN for host containing "_") or something else.
Just to let you know.

[stevvooe]

Already asked if dash can be configured instead of underscore on forums, but I was wondering if this won't have any effects on this issue (aka: define a FQDN for host containing "_") or something else.
Just to let you know.

This is a HUGE bug.

@dnephin Are you guys going to fix this? Supporting _ in service names is a huge nono and will break any hope for a reasonable future. I'm surprised these passed validation.

[dnephin]

Until we have server side stacks I think it's a mistake to change this. We kept it consistent with Compose knowing that either namespaces or server side stacks would be a major change, and we didn't want to change it twice.

Services in a stack should be referenced by the scoped name (without the underscore) anyway, so the underscore shouldn't be relevant for anything except for the CLI.

[stevvooe]

Services in a stack should be referenced by the scoped name (without the underscore) anyway, so the underscore shouldn't be relevant for anything except for the CLI.

Phew!

Are there underscores in the actual service names? Because we to avoid having anything in the resolution path not be supportable in the future.

[mostolog]

@kleinsasserm Thanks for such a clarifying explanation.
Indeed, everything lead to that (now scientifically-supported) conclusion.

I guess the main concern is not registering hostnames, as they can be non-unique, they can change IPs along time...but still it makes much sense to me to merge your PR.
Or maybe it's just an Imbusywithmoreimportantstuffatthismoment.

Anyway, I think we still have to way for an answer from @docker-team

[m4r10k]

Thank you for your response, you are welcome! True, we will have to wait. 😇

[stevvooe]

@mostolog This issue is about the schema of mapping tasks to DNS in a more sane manner. For discussions regarding the current behavior, open another issue in moby.

[m4r10k]

😄 I have already opened one here moby/moby#34239 and there #2325 which is referenced in the PR moby/libnetwork#1855 that I mentioned here. No need to open up another one which will probably get closed as duplicate but what it really needs is, that someone of the maintainers should have a look there.

[mostolog]

@stevvooe IMHO it's related enough to be discussed here, as we are discussing why node names whould be added into DNS tree.

AKA: if I mess my containers having the same names, docker could complaint about it, or even dont work at all, but if they are going to be unique (AKA: using {{.Task.Slot}} template)...let me do it!
#1242 (comment)
#1242 (comment)
#1242 (comment) (bullet #3)

[stevvooe]

@mostolog @kleinsasserm Sorry but this is not a general hostname discussion thread. This issue is about the schema for mapping names to DNS. If there are other relevant discussions already open, have them there. I know it is annoying, but we need to keep the discussion topical.

The issues you are describing should be discussed on the issues in moby. If you want me to join those discussions, pull me in. I have no clue why your PR isn't merged and I am not familiar with the current behavior. I'll try to help as much as I can, but please be patient.

As far as getting this done, I am not really working in this area any more. If someone wants to take on a more complete proposal, I would be more than willing to support and advise.

[m4r10k]

OK.

[nicodmf]

I agree with the schema proposed here in the first message and close moby/moby#30546, As i say in the last issue, i think each component should be multi naming (ex: service.taskid, cluster.taskid, etc.)

[farfeduc]

When will this feature be available ? I would like to use the <Slot ID>.<Service> or <Service>.<Slot ID>.

[kinghuang]

I'd also like to echo support for this. It's been 3 years since this issue was opened, and there's still no easy way to configure things like ZooKeeper.

[antoineco]

If I understand this issue correctly, there is still no way to assign predictable DNS names to tasks inside a Swarm. Is that correct?

I was expecting to be able to use the {{.Task.Name}} template and resolve my tasks to mytask.1, mytask.2... but those task names are always suffixed with a random id (mytask.1.p8d7aufb80h8f8dwtfcmsyzy4).

[Penagwin]

@antoineco An upstream patch was merged in moby/moby#39204 on June 2, that will allow us to set the hostname and have it accessible from other containers in the network.

So now we're just waiting for that to make it upstream into docker I guess?

[antoineco]

That only makes the host name resolvable, which is useful but doesn't help to generate predicable names for task replicas.

Task.Slot may help though. I haven't noticed that parameter before so it may be new (?)

Example:

hostname: "myapp.{{.Task.Slot}}"

should set hostnames like myapp.1, myapp.2, ...

edit: it works 👌

[alen-z]

Can’t wait for this massive feature.

[deftdawg]

@antoineco what the version on the build you have that works? one of the nightlies?

[antoineco]

@deftdawg no, Docker 19.03. Anything recent enough should do, the feature seems older than I though.

[deftdawg]

Oh you were saying resolution by hostname worked, rather you are saying setting hostname now works...

Can now both set hostname (i.e. myapp-{{.Task.Slot}}), and resolve by {{.Task.Name}} neither of which worked on 18.x I came from...

My goal is to get DB nodes to cluster with each other without requiring an external service like etcd or consul - to do that they need to resolve by something-{{ Task.Slot }} because that is both predictable and stable (task slot 1 is respawn if it dies; where as taks.name will be something completely random)... this is so close to actually being usable...

Here's what I did to test:

$ docker --version
Docker version 19.03.6, build 369ce74a3c
$ cat ssh-cluster.yaml 
version: '3.7'
services:
    ssh:
        # use ssh containers because they are easy to shell into to poke around
        image: rastasheep/ubuntu-sshd:latest
        hostname: 'myapp-{{.Task.Slot}}'
        ports:
            - '2222:22'
        logging:
            driver: json-file
        deploy:
            replicas: 3
        # Debug stuff below 'docker inspect <container> | grep X_' to see values
        environment:
            X_NODE_ID: '{{.Node.ID}}'
            X_NODE_HOSTNAME: '{{.Node.Hostname}}'
            X_NODE_PLATFROM: '{{.Node.Platform}}'
            X_NODE_PLATFROM_ARCHITECTURE: '{{.Node.Platform.Architecture}}'
            X_NODE_PLATFROM_OS: '{{.Node.Platform.OS}}'
            X_SERVICE_ID: '{{.Service.ID}}'
            X_SERVICE_NAMES: '{{.Service.Name}}'
            X_SERVICE_LABELS: '{{.Service.Labels}}'
            X_SERVICE_LABEL_STACK_NAMESPACE: '{{index .Service.Labels "com.docker.stack.namespace"}}'
            X_SERVICE_LABEL_STACK_IMAGE: '{{index .Service.Labels "com.docker.stack.image"}}'
            X_SERVICE_LABEL_CUSTOM: '{{index .Service.Labels "service.label"}}'
            X_TASK_ID: '{{.Task.ID}}'
            X_TASK_NAME: '{{.Task.Name}}'
            X_TASK_SLOT: '{{.Task.Slot}}'
                    
$ docker stack deploy ssh --compose-file ssh-cluster.yaml
Updating service ssh_ssh (id: 7iim2c7je2wati3bwo4j6odcs)
$ docker stack ps -f "desired-state=running" ssh
ID                  NAME                IMAGE                           NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
vishpj0pwrnk        ssh_ssh.1           rastasheep/ubuntu-sshd:latest   Chromebox           Running             Running 50 seconds ago                       
f7d3cxvlar1r        ssh_ssh.2           rastasheep/ubuntu-sshd:latest   Chromebox           Running             Running 32 seconds ago                       
m7v3mmouhplb        ssh_ssh.3           rastasheep/ubuntu-sshd:latest   Chromebox           Running             Running 41 seconds ago                       
$ docker ps -f name=ssh
CONTAINER ID        IMAGE                           COMMAND               CREATED              STATUS              PORTS               NAMES
2b869dab3a94        rastasheep/ubuntu-sshd:latest   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp              ssh_ssh.2.f7d3cxvlar1rob0m6dg74ek7j
f6ba9f573b38        rastasheep/ubuntu-sshd:latest   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp              ssh_ssh.3.m7v3mmouhplbhj4imbts7alki
335f9fed407f        rastasheep/ubuntu-sshd:latest   "/usr/sbin/sshd -D"   About a minute ago   Up About a minute   22/tcp              ssh_ssh.1.vishpj0pwrnko5xr47kgxqv0r
$ docker inspect $(docker ps -f name=ssh | grep -v IMAGE | head -1 | cut -d" " -f1) | grep X_
                "X_NODE_HOSTNAME=Chromebox",
                "X_NODE_ID=j92dthzieshbvx2exi30hq2dy",
                "X_NODE_PLATFROM={x86_64 linux}",
                "X_NODE_PLATFROM_ARCHITECTURE=x86_64",
                "X_NODE_PLATFROM_OS=linux",
                "X_SERVICE_ID=7iim2c7je2wati3bwo4j6odcs",
                "X_SERVICE_LABELS=map[com.docker.stack.image:rastasheep/ubuntu-sshd:latest com.docker.stack.namespace:ssh]",
                "X_SERVICE_LABEL_CUSTOM=",
                "X_SERVICE_LABEL_STACK_IMAGE=rastasheep/ubuntu-sshd:latest",
                "X_SERVICE_LABEL_STACK_NAMESPACE=ssh",
                "X_SERVICE_NAMES=ssh_ssh",
                "X_TASK_ID=f7d3cxvlar1rob0m6dg74ek7j",
                "X_TASK_NAME=ssh_ssh.2.f7d3cxvlar1rob0m6dg74ek7j",
                "X_TASK_SLOT=2",
$ ssh root@localhost -p 2222 # password 'root'
root@localhost's password: 
root@myapp-2:~# ssh myapp-1
ssh: Could not resolve hostname myapp-1: No address associated with hostname
root@myapp-2:~# ssh ssh_ssh.1
ssh: Could not resolve hostname ssh_ssh.1: Name or service not known
root@myapp-2:~# ssh ssh_ssh.1.vishpj0pwrnko5xr47kgxqv0r
The authenticity of host 'ssh_ssh.1.vishpj0pwrnko5xr47kgxqv0r (10.0.2.29)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
Are you sure you want to continue connecting (yes/no)? ^C
root@myapp-2:~# exit
logout
Connection to localhost closed.

[antoineco]

@deftdawg No, I meant setting the hostname works. Here is a usage example: https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster#swarm-mode

I'm not sure if it works for everything, but it works with env vars.

[deftdawg]

So this works in Docker built from HEAD, the patch that @Penagwin referred to was merged into Docker with this commit (849af5e343b5a2ca691758a2b8518243968b3a00) on June 2nd 2019, so the next major release (20.x?) should work out of the box... Wonder how much longer we'll have to wait.

Here's the retest against docker head:

root@localhost's password: 
Last login: Thu Mar  5 16:30:18 2020 from 10.0.1.5
root@myapp-1:~# ssh myapp-2
The authenticity of host 'myapp-2 (10.0.1.4)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
Are you sure you want to continue connecting (yes/no)? ^C
root@myapp-1:~# ssh myapp-3
The authenticity of host 'myapp-3 (10.0.1.5)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
Are you sure you want to continue connecting (yes/no)? ^C
root@myapp-1:~# 

[ssikyou]

So is there any workaround to achieve the same goal about hostname resolving before the new release launch?

[diego2glez]

So is there any workaround to achieve the same goal about hostname resolving before the new release launch?

I'm struggling with this. Did you find any workaround?

[TuzelKO]

Hello! Did you find a solution? A feature to predefine container names in a cluster would be very useful.

I would like to see the network environment in the csynk2 service stack. For example: the csync2 stack consists of one declared service (storage-replicator) with 5 replicas, as a result, I would like to be able to access neighboring replicas inside the service at a convenient address {{service_name}}.{{replica_id}} (storage_replicator.1, storage_replicator.2, storage_replicator.3, etc.).

This feature in swarm mode will significantly reduce the size of yml files and make them more unified and simple.

[JKJameson]

It works for me in Debian with official packages from Docker for quite some time now.

[TuzelKO]

@JKJameson how it works? I can't find this in documentation...

[JKJameson]

@TuzelKO In a swarm, with this line in the docker-compose.yml for each service.

hostname: "{{.Service.Name}}.{{.Task.Slot}}"

[TuzelKO]

@JKJameson wow... Thanks!

[nexovec]

@JKJameson Please is there any further documentation on this and also the tasks.dockerservice dns entries?