Switch to using hyphens as a separator in hostnames

[kung-foo]

The hostname format used by fig creates names that are not strictly valid.

Current pseudo-code:

name = '_'.join([project, container, instance_num])

This generates names like cluster_hadoop_1. Underscores are not valid (though in practice most components are tolerant of this).

Valid names should match [a-zA-Z0-9\-]+.

I came across this error when trying to test out some hadoop/hdfs containers and hadoop bailed with an exception saying that hdfs://flume_hadoop_1/ was not valid URI (even though it was in /etc/hosts and ping flume_hadoop_1 worked just fine).

See: http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names

Changing this to dashes is easy, but it would break existing configurations that depend on hard coded container names.

[kung-foo]

also see: moby/moby#5418

[danburkert]

This is causing pain for me as well, in pretty much the exact same scenario (the hostname being parsed by Java's standard library URI parser). Is this a fig issue, or should it be fixed in docker?

[aanand]

Frustratingly, underscores are invalid in hostnames, and dashes are invalid in shell variable names. So if you can't name a Docker link without either breaking one of those or smushing everything together (e.g. hadoop1). Not ideal.

However, looks like moby/moby#6270 might fix this by sanitising environment variable names, in which case Fig can switch to dashes and everything will hopefully work:

web:
  links:
    - hadoop

$ fig run web cat /etc/hosts
...
172.17.0.42 hadoop-1
...

$ fig run web env
...
HADOOP_1_PORT=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP_ADDR=172.17.0.42
...

[aanand]

#349 is also relevant.

[nubs]

Technically speaking, I don't believe that hyphens in shell variable names are strictly forbidden. You just can't access them in the normal way:

$ env foo-bar=baz bash
$ printenv foo-bar
baz

[bfirsh]

👍 switching to hyphens

[zeevallin]

What about writing both hyphenated and underscored to the hosts file for now?

[aanand]

@zeeraw that would be out of fig's scope - a job for docker itself.

[bfirsh]

We can do this now that moby/moby#5418 has been fixed, right?

[jamshid]

Ugh java's URI is so annoying. The single-value constructor does not throw an exception for underscores in the hostname.

// Error: java.net.URISyntaxException: Illegal character in hostname at index 10: http://foo_bar:9200/path?query=1
URI u1 = new URI("http", null, "foo_bar", 9200, "/path", "query=1", null);

// works fine:
URI u2 = new URI("http://foo_bar:9200/path?query=1");

[thaJeztah]

Will this also be used for the container names? If so, I think moby/moby#8961 should also be taken into account.

[dnephin]

We could add aliases for dashes when we implement #2312, keep both underscores and dashes for a release or two, then drop support for underscores in hostnames.

[susu]

Same here, cannot run spark cluster with docker-compose. Showstopper
for docker-compose, falling back to "manual docker".

[aanand]

@susu Why is this a showstopper? The problem of underscores in hostnames is easily worked around, either with link aliases or by changing service names.

[susu]

Nope, because spark somehow extracts the underscored hostname of docker
container (projectname_container_1.default_network), and try to use it
(which is an invalid hostname in URI). Anyway, falling back to version 1
"solved" the issue.
On Jul 13, 2016 6:52 PM, "Aanand Prasad" notifications@github.com wrote:

@susu https://github.com/susu Why is this a showstopper? The problem of
underscores in hostnames is easily worked around, either with link aliases
or by changing service names.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#229 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AA5B7o9ixPlutkGcoBwiBnp2N5EcqBszks5qVRfQgaJpZM4B9pLe
.

[aanand]

spark somehow extracts the underscored hostname of docker container (projectname_container_1.default_network), and try to use it

However it's doing that, that's the wrong thing to do. Is this an image you're using from the Hub, or did you build it yourself?

Switching back to version 1 is not a future-proof solution.

[gordontyler]

@susu The workaround I used for this was to create a network so I could control the name:

docker network create spark

Use that as an external network in my (version 2) docker-compose.yml file:

networks:
  spark:
    external: true

Connect each service to that network:

services:
  spark-master:
    image: spark
    command: org.apache.spark.deploy.master.Master --host spark-master
    networks:
      - spark

And finally override the container name and hostname in the spark worker service:

  spark-worker:
    image: spark
    container_name: "spark-worker"
    hostname: "spark-worker"
    command: org.apache.spark.deploy.worker.Worker --host spark-worker spark://spark-master:7077
    networks:
      - spark

[gordontyler]

P.S. I determined that the spark worker is using reverse DNS lookup on its IP address to get the DNS name to register with the master.

[susu]

Thanks @gordontyler ! I didn't know I can override container and hostname! Already migrated to version 2 :)

@aanand I know version 1 is not future-proof, however at the moment I'm only creating a PoC with spark, so clean, production-grade solution is not a must-have :) But anyway, using @gordontyler 's solution, I can go forward with version 2! Thank you guys!

[laugimethods]

FYI, that issue has also been raised in Docker Forum as Underscore in domain names.

In particular, I explain there where & why Spark (from version 1.6.x) doesn't allow underscores in hostnames.

// We identify hosts on which the block is cached with this prefix. Because this prefix contains
// underscores, which are not legal characters in hostnames, there should be no potential for
// confusion. See RFC 952 and RFC 1123 for information about the format of hostnames.

[wrmay]

I am having this problem as well, related to Hazelcast. I can work around it using fixed container names without underscores but it precludes the use of the docker-compose scaling mechanism since the system assigned name is mystack_myservice_n.

[aki-k]

389 directory server's admin console, 389-console, is not able to handle the underscores in container DNS names, but adding a network alias hostname with no underscores resolved that. I wasn't even able to complete the directory server configuration until adding the hostname aliases with no underscores.

I had to also use "StrictHostCheck = false" in the initial-setup.inf file and "/usr/sbin/setup-ds-admin.pl --silent --file=initial-setup.inf General.StrictHostCheck=false" when installing the directory server binaries, admin server binaries and the initial directory configuration. This was because Docker DNS is a can of worms regarding reverse DNS entries.

[TrentonAdams]

Is this being done any time soon? This makes docker-compose unusable with many languages, which is really unfortunate.

[michaelsstuff]

How is this still an issue after 6 years? This is a basic DNS violation.
RFC 1034 and 1035 ->

<subdomain> ::= <label> | <subdomain> "." <label>

<label> ::= <letter> [ [ <ldh-str> ] <let-dig> ]

<ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str>

<let-dig-hyp> ::= <let-dig> | "-"

<let-dig> ::= <letter> | <digit>

<letter> ::= any one of the 52 alphabetic characters A through Z in
upper case and a through z in lower case

<digit> ::= any one of the ten digits 0 through 9

This makes compose absolutely useless for more "strict" or rather correct tools.
There is also no reasonable way to overwrite this. Als if i spawn more than 1 container of a service, the alias and container names no longer work.

[mike-marcacci]

Reading through this thread, there don't appear to be any real arguments against the proposed change. (Perhaps I missed them?)

Are there any philosophical or practical reasons that a PR to address this would not be merged, or is this simply a matter of nobody putting in the work yet?

[noahkawasakigoogle]

+1

[rolve]

This is a problem with Tomcat (and, by extension, with many Spring Boot applications) too: https://stackoverflow.com/q/53504857/1374678

[libricoleur]

Just got bit by this myself. Connecting two containers from different services should have been a breeze, instead it has been a day-long nightmare. Please fix this. It has been open for almost seven years.

[lump]

I discovered that we can put dots in service aliases! That means that I can define names how I want them: in FQDN format and no underscores. I have scripting that standardizes our docker-compose.yml files and automatically defines services under the RFC6762 .private domain as <service-name>.<stack-name>.<network-name>.private, e.g. mariadb.db.swarm.private. This also makes it easier to filter swarm service name queries on upstream nameservers when the service isn't up yet, and many recursive nameservers won't recurse for that domain. Docker can keep its wonky underscores and unqualified names because it doesn't matter to me anymore. 😃

[Dr-Irv]

Just came across this problem with spring cloud config and docker stack deploy on docker 18.09, the error is the following:

2018-11-25 12:37:21.566  INFO 1 --- [nio-8090-exec-2] o.apache.coyote.http11.Http11Processor   : The host [tskur-ci_tskur-svc-config:8090] is not valid
 Note: further occurrences of request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
	at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]

Is anyone able to point me to a workaround?

Rename your containers to change the underscores to hyphens.

IMHO - the documentation should indicate that container names that include characters that are not valid for hostnames might break, dependent on the underlying server URL parser (e.g. Spring, tomcat)

[ndeloof]

eventually closing this issue, by Compose V2 hostname will be set using hyphens, unless --compatibility flag is set

[jamshid]

Whoa this is a big change to spring on users after it wasn't enabled in the last RC (it's the default even for existing projects). Is there a way to make this setting global e.g. via a config file, rather than updating all my scripts?

[flowl]

unknown flag: --compatibility

Use this env variable instead: COMPOSE_COMPATIBILITY=true

[ndeloof]

→ docker compose --help
Options:
      --compatibility              Run compose in backward compatibility mode

[flowl]

@ndeloof something is messed with the command line.

This command works:

docker compose build --pull --force-rm

This doesn't:

docker compose build --pull --force-rm --compatibility
unknown flag: --compatibility

This doesn't either:

docker compose --pull --force-rm --compatibilty build
unknown flag: --pull

Version on MacOS:

 ~ % docker --version
Docker version 20.10.8, build 3967b7d

Edit:

Command is ment to be like this I guess:

docker compose --compatibility build --pull --force-rm

It takes some getting used to :-S

[ndeloof]

"compatibility" applies to the top-level "compose" verb:

docker compose --compatibilty (other flags and commands) ...

[Issykul]

MinIO, which is a S3-compatible storage solution, doesn't support underscores in given hostnames anymore. Which means all my stacks that are deployed in my docker swarm with docker stack deploy -c docker-compose.yml stackname are not usable anymore.

As far as I know compose verison 3 uses underscores like this: stackname_containername by default. Is there any way to change the underscore myself in some setting? Or do i have to use a newer compose version?

[aki-k]

@Issykul This is how I solved the problem in Docker compose file:

    hostname: user-001-login.login

    networks:
      login:
        aliases:
          - user-001-login.login
          - user-001-login

[lump]

Docker's default DNS service naming is a little silly because it does not take advantage of standard DNS hierarchy conventions, which could still be useful.

I like to ignore Docker's automatic service names and define my own instead, just as @aki-k suggested. The only difference is that I like to make mine fully qualified. Doing it this way allows me to use filter the top-level domain and avoid hammering my upstream nameservers if a service isn't up yet. I could filter all unqualified names, but I don't like that. I like to name things [service].[stack].[network].private, for example:

    networks:
      swarm:
        aliases:
        - prometheus.monitor.swarm.private