DNS resolution in a Java Docker container fails

[JacekCala]

Netty version: 4.1.12.Final
Docker image: openjdk:8-jre

Context:
Running DNS resolution in a Java docker container fails. DNS resolution should complete successfully and return the IP address of the sibling docker container. Instead it throws the following exception:

Exception in thread "main" java.util.concurrent.ExecutionException: java.net.UnknownHostException: failed to resolve 'recomp-hdb' after 3 queries
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
    at TestNetty.main(TestNetty.java:15)
Caused by: java.net.UnknownHostException: failed to resolve 'recomp-hdb' after 3 queries
    at io.netty.resolver.dns.DnsNameResolverContext.finishResolve(DnsNameResolverContext.java:688)
    at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:620)
    at io.netty.resolver.dns.DnsNameResolverContext$3.operationComplete(DnsNameResolverContext.java:313)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
    ...

DNS resolution with plain Java code works as expected and returns the IP address.

The Netty-based code that throws the exception:

import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
import io.netty.resolver.dns.DnsNameResolverBuilder;
import io.netty.resolver.dns.DnsNameResolver;
import io.netty.resolver.dns.DnsServerAddresses;
public class TestNetty {
  private static NioEventLoopGroup bossGroup = new NioEventLoopGroup();
  
  public static void main(String[] args) throws Exception {
    DnsNameResolver res = new DnsNameResolverBuilder(bossGroup.next())
        .channelType(NioDatagramChannel.class)
        .build();

    System.out.println(res.resolve("recomp-hdb").await().get());
  }
}

The plain Java code which works fine:

import java.net.InetAddress;
public class Test {
    public static void main(String args[]) throws Exception {
            InetAddress address = InetAddress.getByName("recomp-hdb");
            System.out.println(address.getHostAddress());
    }
}

Steps to reproduce:

1. Create a docker compose file that defines two docker containers, one of which is based on openjdk image.
2. Build the code shown above, changing the name of the sibling container according to your docker compose.
3. Copy in the code to the java container and run.
4. The plain java code will show the correct response. The netty-based code will throw the exception as above.

[Scottmitch]

Create a docker compose file that defines two docker containers, one of which is based on openjdk image.

Can you describe in detail how this is done and what OS is running in the container? Assume we are not familiar with docker or your setup.

• Is this a unix based host? If so what is the contents of /etc/resolv.conf and /etc/resolver?
• Can you include a Wireshark dump of the DNS lookup done by the call to InetAddress.getByName and the call to res.resolve?

[JacekCala]

Hi,

Sorry for the delay. A minimal docker-compose.yml might be:

version: "3"
services:
  netty-host:
    image: openjdk:8-jre
    command: /bin/bash
    stdin_open: true
  recomp-hdb:
    image: openjdk:8-jre
    command: /bin/bash
    stdin_open: true

Given that, you can start the composition like:

> docker-compose u -d

Use option -d to run containers in background, so you have free console at hand.
Then, build the examples I provided. For netty it's best to use maven, I guess, and build a fat jar.

Copy the compiled code to your started docker containers. First check container names/ids:

> docker ps

Then copy the code:

> docker cp <CLASS_NAME>.class <PREFIX>_netty-host_1:./

Once the code is copied, run bash in the container (or attach to the netty-host container):

> docker exec -it <PREFIX>_netty-host_1 /bin/bash

And, finally, try:

 root# ping recomp-hdb

and

root# java -cp . Test

and

root# java -jar <YOUR_NETTY_FATJAR> 

The first two work fine, whereas the last throws the exception as above.

Hope it helps.

[Scottmitch]

I wasn't able to reproduce ... please advise. I'm running Docker on MacOS 10.12.5.

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.repro</groupId>
  <artifactId>netty-dns-repro</artifactId>
  <version>0.1</version>
  <packaging>jar</packaging>

  <properties>
    <netty.version>4.1.12.Final</netty.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-resolver-dns</artifactId>
      <version>${netty.version}</version>
    </dependency>
  </dependencies>
  
  <build>
    <plugins>
      <plugin>
        <inherited>true</inherited>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
          <optimize>true</optimize>
          <debug>false</debug>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.repro.DnsRepro</mainClass>
            </manifest>
          </archive>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id> <!-- this is used for inheritance merges -->
            <phase>package</phase> <!-- bind to the packaging phase -->
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

DnsRepro.java

package com.repro;

import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
import io.netty.resolver.dns.DnsNameResolver;
import io.netty.resolver.dns.DnsNameResolverBuilder;

import java.net.InetAddress;

public class DnsRepro {
    private static NioEventLoopGroup bossGroup = new NioEventLoopGroup();

    public static void main(String[] args) throws Exception {
        DnsNameResolver res = new DnsNameResolverBuilder(bossGroup.next())
                .channelType(NioDatagramChannel.class)
                .build();
        System.out.println("netty-dns: " +res.resolve("recomp-hdb").await().get());

        InetAddress address = InetAddress.getByName("recomp-hdb");
        System.out.println("java-dns:" + address.getHostAddress());

        bossGroup.shutdownGracefully();
    }
}

stdout

root@326481c34ae5:/# java -jar netty-dns-repro-0.1-jar-with-dependencies.jar
netty-dns: recomp-hdb/172.18.0.3
java-dns:172.18.0.3
root@326481c34ae5:/# exit

[JacekCala]

That's strange... I've copied and pasted your pom and code, built the jar, copied it to a newly started docker composition and it crashed as previously:

root@0f7ab08fc39b:/# ping recomp-hdb
PING recomp-hdb (172.20.0.3): 56 data bytes
64 bytes from 172.20.0.3: icmp_seq=0 ttl=64 time=0.096 ms
64 bytes from 172.20.0.3: icmp_seq=1 ttl=64 time=0.090 ms
^C--- recomp-hdb ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.090/0.093/0.096/0.000 ms
root@0f7ab08fc39b:/# java -jar netty-dns-repro-0.1-jar-with-dependencies.jar
Exception in thread "main" java.util.concurrent.ExecutionException: java.net.UnknownHostException: failed to resolve 'recomp-hdb' after 3 queries
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
    at com.repro.DnsRepro.main(DnsRepro.java:17)
Caused by: java.net.UnknownHostException: failed to resolve 'recomp-hdb' after 3 queries
    at io.netty.resolver.dns.DnsNameResolverContext.finishResolve(DnsNameResolverContext.java:688)
    at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:620)
    ...

I'm running the latest Docker and docker-compose on Ubuntu 16.04 which is a VirtualBox VM on my Windows 10 host:

$ uname -a
Linux recomp-1604 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ docker --version
Docker version 17.03.1-ce, build c6d412e

$ docker-compose --version
docker-compose version 1.13.0, build 1719ceb

[Scottmitch]

I was able to reproduce the issue on a Linux VM inside a docker container. See PR #6885 for a fix.

[JacekCala]

Thanks, I passed the details to the Vertx team. Cheers, Jacek