Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to using hyphens as a separator in hostnames #229

Closed
kung-foo opened this issue May 23, 2014 · 64 comments · Fixed by #8655
Closed

Switch to using hyphens as a separator in hostnames #229

kung-foo opened this issue May 23, 2014 · 64 comments · Fixed by #8655

Comments

@kung-foo
Copy link

@kung-foo kung-foo commented May 23, 2014

The hostname format used by fig creates names that are not strictly valid.

Current pseudo-code:

name = '_'.join([project, container, instance_num])

This generates names like cluster_hadoop_1. Underscores are not valid (though in practice most components are tolerant of this).

Valid names should match [a-zA-Z0-9\-]+.

I came across this error when trying to test out some hadoop/hdfs containers and hadoop bailed with an exception saying that hdfs://flume_hadoop_1/ was not valid URI (even though it was in /etc/hosts and ping flume_hadoop_1 worked just fine).

See: http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names

Changing this to dashes is easy, but it would break existing configurations that depend on hard coded container names.

@kung-foo
Copy link
Author

@kung-foo kung-foo commented May 24, 2014

also see: moby/moby#5418

@danburkert
Copy link

@danburkert danburkert commented Jul 25, 2014

This is causing pain for me as well, in pretty much the exact same scenario (the hostname being parsed by Java's standard library URI parser). Is this a fig issue, or should it be fixed in docker?

@aanand
Copy link

@aanand aanand commented Jul 25, 2014

Frustratingly, underscores are invalid in hostnames, and dashes are invalid in shell variable names. So if you can't name a Docker link without either breaking one of those or smushing everything together (e.g. hadoop1). Not ideal.

However, looks like moby/moby#6270 might fix this by sanitising environment variable names, in which case Fig can switch to dashes and everything will hopefully work:

web:
  links:
    - hadoop
$ fig run web cat /etc/hosts
...
172.17.0.42 hadoop-1
...

$ fig run web env
...
HADOOP_1_PORT=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP_ADDR=172.17.0.42
...

@aanand
Copy link

@aanand aanand commented Jul 25, 2014

#349 is also relevant.

@nubs
Copy link

@nubs nubs commented Jul 26, 2014

Technically speaking, I don't believe that hyphens in shell variable names are strictly forbidden. You just can't access them in the normal way:

$ env foo-bar=baz bash
$ printenv foo-bar
baz

@bfirsh
Copy link

@bfirsh bfirsh commented Jul 28, 2014

👍 switching to hyphens

@bfirsh bfirsh changed the title invalid hostnames Switch to using hyphens as a separator in hostnames Jul 28, 2014
@zeeraw
Copy link

@zeeraw zeeraw commented Aug 12, 2014

What about writing both hyphenated and underscored to the hosts file for now?

@aanand
Copy link

@aanand aanand commented Aug 12, 2014

@zeeraw that would be out of fig's scope - a job for docker itself.

@bfirsh
Copy link

@bfirsh bfirsh commented Dec 5, 2014

We can do this now that moby/moby#5418 has been fixed, right?

@jamshid
Copy link

@jamshid jamshid commented Dec 18, 2014

Ugh java's URI is so annoying. The single-value constructor does not throw an exception for underscores in the hostname.

// Error: java.net.URISyntaxException: Illegal character in hostname at index 10: http://foo_bar:9200/path?query=1
URI u1 = new URI("http", null, "foo_bar", 9200, "/path", "query=1", null);

// works fine:
URI u2 = new URI("http://foo_bar:9200/path?query=1");

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Dec 18, 2014

Will this also be used for the container names? If so, I think moby/moby#8961 should also be taken into account.

@dnephin
Copy link

@dnephin dnephin commented Nov 13, 2015

We could add aliases for dashes when we implement #2312, keep both underscores and dashes for a release or two, then drop support for underscores in hostnames.

@susu
Copy link

@susu susu commented Jul 13, 2016

Same here, cannot run spark cluster with docker-compose. Showstopper
for docker-compose, falling back to "manual docker".

@aanand
Copy link

@aanand aanand commented Jul 13, 2016

@susu Why is this a showstopper? The problem of underscores in hostnames is easily worked around, either with link aliases or by changing service names.

@susu
Copy link

@susu susu commented Jul 13, 2016

Nope, because spark somehow extracts the underscored hostname of docker
container (projectname_container_1.default_network), and try to use it
(which is an invalid hostname in URI). Anyway, falling back to version 1
"solved" the issue.
On Jul 13, 2016 6:52 PM, "Aanand Prasad" notifications@github.com wrote:

@susu https://github.com/susu Why is this a showstopper? The problem of
underscores in hostnames is easily worked around, either with link aliases
or by changing service names.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#229 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AA5B7o9ixPlutkGcoBwiBnp2N5EcqBszks5qVRfQgaJpZM4B9pLe
.

@aanand
Copy link

@aanand aanand commented Jul 13, 2016

spark somehow extracts the underscored hostname of docker container (projectname_container_1.default_network), and try to use it

However it's doing that, that's the wrong thing to do. Is this an image you're using from the Hub, or did you build it yourself?

Switching back to version 1 is not a future-proof solution.

@gordontyler
Copy link

@gordontyler gordontyler commented Jul 14, 2016

@susu The workaround I used for this was to create a network so I could control the name:

docker network create spark

Use that as an external network in my (version 2) docker-compose.yml file:

networks:
  spark:
    external: true

Connect each service to that network:

services:
  spark-master:
    image: spark
    command: org.apache.spark.deploy.master.Master --host spark-master
    networks:
      - spark

And finally override the container name and hostname in the spark worker service:

  spark-worker:
    image: spark
    container_name: "spark-worker"
    hostname: "spark-worker"
    command: org.apache.spark.deploy.worker.Worker --host spark-worker spark://spark-master:7077
    networks:
      - spark

@gordontyler
Copy link

@gordontyler gordontyler commented Jul 14, 2016

P.S. I determined that the spark worker is using reverse DNS lookup on its IP address to get the DNS name to register with the master.

@susu
Copy link

@susu susu commented Jul 14, 2016

Thanks @gordontyler ! I didn't know I can override container and hostname! Already migrated to version 2 :)

@aanand I know version 1 is not future-proof, however at the moment I'm only creating a PoC with spark, so clean, production-grade solution is not a must-have :) But anyway, using @gordontyler 's solution, I can go forward with version 2! Thank you guys!

@laugimethods
Copy link

@laugimethods laugimethods commented Jul 21, 2016

FYI, that issue has also been raised in Docker Forum as Underscore in domain names.

In particular, I explain there where & why Spark (from version 1.6.x) doesn't allow underscores in hostnames.

// We identify hosts on which the block is cached with this prefix. Because this prefix contains
// underscores, which are not legal characters in hostnames, there should be no potential for
// confusion. See RFC 952 and RFC 1123 for information about the format of hostnames.

@teutat3s
Copy link

@teutat3s teutat3s commented Oct 12, 2019

Not stale at all, I'd say this is still unresolved? Consul DNS also doesn't like underscores in hostnames and complains... Is it possible to make this more flexible? Like letting the user change the seperator?

@stale
Copy link

@stale stale bot commented Oct 12, 2019

This issue has been automatically marked as not stale anymore due to the recent activity.

@stale stale bot removed the stale label Oct 12, 2019
@suthernfriend
Copy link

@suthernfriend suthernfriend commented Oct 23, 2019

just ran into a problem with logstash related to this.

can somebody at docker HQ at least make a decision?
shouldn't docker-compose comply with the relevant RFCs?

@wrmay
Copy link

@wrmay wrmay commented Nov 25, 2019

I am having this problem as well, related to Hazelcast. I can work around it using fixed container names without underscores but it precludes the use of the docker-compose scaling mechanism since the system assigned name is mystack_myservice_n.

@aki-k
Copy link

@aki-k aki-k commented Jan 25, 2020

389 directory server's admin console, 389-console, is not able to handle the underscores in container DNS names, but adding a network alias hostname with no underscores resolved that. I wasn't even able to complete the directory server configuration until adding the hostname aliases with no underscores.

I had to also use "StrictHostCheck = false" in the initial-setup.inf file and "/usr/sbin/setup-ds-admin.pl --silent --file=initial-setup.inf General.StrictHostCheck=false" when installing the directory server binaries, admin server binaries and the initial directory configuration. This was because Docker DNS is a can of worms regarding reverse DNS entries.

@TrentonAdams
Copy link

@TrentonAdams TrentonAdams commented Apr 9, 2020

Is this being done any time soon? This makes docker-compose unusable with many languages, which is really unfortunate.

@michaelsstuff
Copy link

@michaelsstuff michaelsstuff commented Jul 16, 2020

How is this still an issue after 6 years? This is a basic DNS violation.
RFC 1034 and 1035 ->

<subdomain> ::= <label> | <subdomain> "." <label>

<label> ::= <letter> [ [ <ldh-str> ] <let-dig> ]

<ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str>

<let-dig-hyp> ::= <let-dig> | "-"

<let-dig> ::= <letter> | <digit>

<letter> ::= any one of the 52 alphabetic characters A through Z in
upper case and a through z in lower case

<digit> ::= any one of the ten digits 0 through 9

This makes compose absolutely useless for more "strict" or rather correct tools.
There is also no reasonable way to overwrite this. Als if i spawn more than 1 container of a service, the alias and container names no longer work.

@mike-marcacci
Copy link

@mike-marcacci mike-marcacci commented Oct 30, 2020

Reading through this thread, there don't appear to be any real arguments against the proposed change. (Perhaps I missed them?)

Are there any philosophical or practical reasons that a PR to address this would not be merged, or is this simply a matter of nobody putting in the work yet?

@noahkawasakigoogle
Copy link

@noahkawasakigoogle noahkawasakigoogle commented Feb 10, 2021

+1

@rolve
Copy link

@rolve rolve commented Feb 25, 2021

This is a problem with Tomcat (and, by extension, with many Spring Boot applications) too: https://stackoverflow.com/q/53504857/1374678

@libricoleur
Copy link

@libricoleur libricoleur commented Apr 29, 2021

Just got bit by this myself. Connecting two containers from different services should have been a breeze, instead it has been a day-long nightmare. Please fix this. It has been open for almost seven years.

@lump
Copy link

@lump lump commented Jul 13, 2021

I discovered that we can put dots in service aliases! That means that I can define names how I want them: in FQDN format and no underscores. I have scripting that standardizes our docker-compose.yml files and automatically defines services under the RFC6762 .private domain as <service-name>.<stack-name>.<network-name>.private, e.g. mariadb.db.swarm.private. This also makes it easier to filter swarm service name queries on upstream nameservers when the service isn't up yet, and many recursive nameservers won't recurse for that domain. Docker can keep its wonky underscores and unqualified names because it doesn't matter to me anymore. 😃

@Dr-Irv
Copy link

@Dr-Irv Dr-Irv commented Sep 21, 2021

Just came across this problem with spring cloud config and docker stack deploy on docker 18.09, the error is the following:

2018-11-25 12:37:21.566  INFO 1 --- [nio-8090-exec-2] o.apache.coyote.http11.Http11Processor   : The host [tskur-ci_tskur-svc-config:8090] is not valid
 Note: further occurrences of request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
	at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]

Is anyone able to point me to a workaround?

Rename your containers to change the underscores to hyphens.

IMHO - the documentation should indicate that container names that include characters that are not valid for hostnames might break, dependent on the underlying server URL parser (e.g. Spring, tomcat)

@ndeloof
Copy link
Collaborator

@ndeloof ndeloof commented Sep 23, 2021

eventually closing this issue, by Compose V2 hostname will be set using hyphens, unless --compatibility flag is set

@ndeloof ndeloof closed this as completed Sep 23, 2021
@jamshid
Copy link

@jamshid jamshid commented Oct 1, 2021

Whoa this is a big change to spring on users after it wasn't enabled in the last RC (it's the default even for existing projects). Is there a way to make this setting global e.g. via a config file, rather than updating all my scripts?

@flowl
Copy link

@flowl flowl commented Oct 21, 2021

unknown flag: --compatibility

Use this env variable instead: COMPOSE_COMPATIBILITY=true

@ndeloof
Copy link
Collaborator

@ndeloof ndeloof commented Oct 21, 2021

→ docker compose --help
Options:
      --compatibility              Run compose in backward compatibility mode

@flowl
Copy link

@flowl flowl commented Oct 21, 2021

@ndeloof something is messed with the command line.

This command works:

docker compose build --pull --force-rm

This doesn't:

docker compose build --pull --force-rm --compatibility
unknown flag: --compatibility

This doesn't either:

docker compose --pull --force-rm --compatibilty build
unknown flag: --pull

Version on MacOS:

 ~ % docker --version
Docker version 20.10.8, build 3967b7d

Edit:

Command is ment to be like this I guess:

docker compose --compatibility build --pull --force-rm

It takes some getting used to :-S

@ndeloof
Copy link
Collaborator

@ndeloof ndeloof commented Oct 21, 2021

"compatibility" applies to the top-level "compose" verb:

docker compose --compatibilty (other flags and commands) ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.