Buildah intermediate container(Id) is not resolvable

[rhatdan]

Discussed in #4424

^{Originally posted by SaurabhAhuja1983 November 18, 2022}

Docker is assigning ipaddress to each intermediate container and containerId is resolvable. But buildah intermediate containerId is not resolvable...

I used following Dockerfile
FROM <REPO-TAKE-ANY-JAVA-IMAGE>/java_jdk_8:latest
COPY MyClass.java .

RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass

cat MyClass.java

import java.net.InetAddress;
import java.net.UnknownHostException;

public class MyClass {
    public static void main(String args[]) {
        try {
            InetAddress ia = InetAddress.getLocalHost();
            String str = ia.getHostAddress();
            String HOSTNAME = InetAddress.getLocalHost().getHostName();
            System.out.println("hostname: " + HOSTNAME);
         } catch (final UnknownHostException e) {
            throw new IllegalStateException("cannot get host name", e);
         }
    }
}

See the following output
`buildah build --no-cache .
STEP 1/4: FROM <.............................>/java_jdk_8:latest
STEP 2/4: RUN which java
/usr/bin/java
STEP 3/4: COPY MyClass.java .
STEP 4/4: RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass
---
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.20.203.214	agent-79ccf9cbc-cvstd
10.20.203.214	host.containers.internal
Hostname: 43d4f3951829
---
Exception in thread "main" java.lang.IllegalStateException: cannot get host name
	at MyClass.main(MyClass.java:12)
Caused by: java.net.UnknownHostException: 43d4f3951829: 43d4f3951829: Name or service not known
	at java.net.InetAddress.getLocalHost(InetAddress.java:1418)
	at MyClass.main(MyClass.java:7)
Caused by: java.net.UnknownHostException: 43d4f3951829: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:866)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1288)
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:815)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1413)
	... 1 more
subprocess exited with status 1
subprocess exited with status 1
error building at STEP "RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass": exit status 1
`

However the same code works fine with docker build as i could see the intermediate containerId to IP mapping in /etc/hosts.

Not sure if buildah intermediate container is not assigned an ipaddress but it's blocking our java projects from getting build as we are transitioning from docker to buildah for building images.

[rhatdan]

@flouthoc PTAL

[flouthoc]

Hi @SaurabhAhuja1983 , I tried reproducing with the latest buildah I was unable to reproduce and java class was successfully able to resolve the hostname. Could you please try with latest buildah.

Ignore the hostname: command not found since my openjdk image did not had it but java class was able to do it.

[fl@fedora bin]$ ./buildah build --layers --no-cache -t test .
STEP 1/3: FROM openjdk:latest
STEP 2/3: COPY MyClass.java .
--> 3d18cc7dcbf
STEP 3/3: RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass
---
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1	localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.16	host.containers.internal
10.0.2.100	083ba9abf0aa 3d18cc7dcbf6-working-container
/bin/sh: hostname: command not found
Hostname: 
---
hostname: 083ba9abf0aa
COMMIT test
--> 0b556a7f11c
Successfully tagged localhost/test:latest
0b556a7f11cc670e6c40d6405ddcff0427287902adff2251e00122ea130bcf38
[fl@fedora bin]$ ./buildah build --no-cache -t test .
STEP 1/3: FROM openjdk:latest
STEP 2/3: COPY MyClass.java .
STEP 3/3: RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass
---
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1	localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.16	host.containers.internal
10.0.2.100	57de73bb13f4 openjdk-working-container
/bin/sh: hostname: command not found
Hostname: 
---
hostname: 57de73bb13f4
COMMIT test
--> b0971b4e1ff
Successfully tagged localhost/test:latest
b0971b4e1ff224bd5161cf3be6c582ad7421dbf7d6dd54261f7fc22823c4cb81

FROM openjdk:latest
COPY MyClass.java .
RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; javac MyClass.java ; /usr/bin/java -cp . MyClass

Please feel free to try it out and comment back if you think issue is with buildah and I will reopen this.

[SaurabhAhuja1983]

@flouthoc Thank you for reproducing the issue. But i still see the issue with latest buildah v 1.28.2
i added following as well 'cat /run/.containerenv'

STEP 1/3: FROM <Add Your path>/java_jdk_8:latest
STEP 2/3: COPY MyClass.java .
STEP 3/3: RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; cat /run/.containerenv ; javac MyClass.java ; /usr/bin/java -cp . MyClass
---
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.20.206.207	agent-74576f5997-fdbv5
10.20.206.207	host.containers.internal
Hostname: 86016af8d973
---

engine="buildah-1.28.2"
name="java_jdk_8-working-container"
id="86016af8d973451fd5f6612506a5e8408a1e8f2b926a505f1249f3053c198e7b"
image="<Add your image path>/java_jdk_8:latest"
imageid="fe28db6e98598be12e1fc74b7805dd7aa699e88b593c415257f91c19cda740db"
rootless=1
Exception in thread "main" java.lang.IllegalStateException: cannot get host name
	at MyClass.main(MyClass.java:12)
Caused by: java.net.UnknownHostException: 86016af8d973: 86016af8d973: Name or service not known
	at java.net.InetAddress.getLocalHost(InetAddress.java:1418)
	at MyClass.main(MyClass.java:7)
Caused by: java.net.UnknownHostException: 86016af8d973: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:866)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1288)
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:815)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1413)
	... 1 more
subprocess exited with status 1
subprocess exited with status 1
Error: building at STEP "RUN echo "---" ; cat /etc/hosts ; echo "Hostname: $(hostname)" ; echo "---"; cat /run/.containerenv ; javac MyClass.java ; /usr/bin/java -cp . MyClass": exit status 1

As you can see, i am using rootless buildah and not using slirp4netns

i can see in your output entry in /etc/hosts

10.0.2.100 57de73bb13f4 openjdk-working-container

Are you using slirp4netns ?
Are you using rootless ?

All i want is the ip -> containerId mapping in the /etc/hosts for rootless buildah.

Can you please share the output with loglevel trace on.
buildah --log-level trace bud -t test .

can you please reopen the issue.
I am looking at the run_linux.go and hosts.go in buildah and will be happy to help in resolving this issue.

[SaurabhAhuja1983]

@flouthoc not sure if you saw my comment. Pinging again.

[flouthoc]

@SaurabhAhuja1983 Thanks for reminding. I will try reproducing it again on a rootful env :)

[SaurabhAhuja1983]

Thank You.
rootless :)
I am deep diving into code today to try to find the problem and potential fix.

[SaurabhAhuja1983]

I created the following patch and it seemed to work [added ip-> hostname mapping]

diff --git a/run_common.go b/run_common.go
index 81fe4af7..2db2752c 100644
--- a/run_common.go
+++ b/run_common.go
@@ -117,7 +117,7 @@ func (b *Builder) addResolvConf(rdir string, chownOpts *idtools.IDPair, dnsServe
 }
 
 // generateHosts creates a containers hosts file
-func (b *Builder) generateHosts(rdir string, chownOpts *idtools.IDPair, imageRoot string) (string, error) {
+func (b *Builder) generateHosts(rdir string, chownOpts *idtools.IDPair, imageRoot, hostname string) (string, error) {
 	conf, err := config.Default()
 	if err != nil {
 		return "", err
@@ -134,6 +134,7 @@ func (b *Builder) generateHosts(rdir string, chownOpts *idtools.IDPair, imageRoo
 		ExtraHosts:               b.CommonBuildOpts.AddHost,
 		HostContainersInternalIP: etchosts.GetHostContainersInternalIP(conf, nil, nil),
 		TargetFile:               targetfile,
+		Hostname:                 hostname,
 	}); err != nil {
 		return "", err
 	}
diff --git a/run_linux.go b/run_linux.go
index 6370ba38..cfdff177 100644
--- a/run_linux.go
+++ b/run_linux.go
@@ -256,7 +256,7 @@ func (b *Builder) Run(command []string, options RunOptions) error {
 
 	hostFile := ""
 	if !options.NoHosts && !contains(volumes, config.DefaultHostsFile) && options.ConfigureNetwork != define.NetworkDisabled {
-		hostFile, err = b.generateHosts(path, rootIDPair, mountPoint)
+		hostFile, err = b.generateHosts(path, rootIDPair, mountPoint, spec.Hostname)
 		if err != nil {
 			return err
 		}
diff --git a/vendor/github.com/containers/common/libnetwork/etchosts/hosts.go b/vendor/github.com/containers/common/libnetwork/etchosts/hosts.go
index ce248a18..5b17c821 100644
--- a/vendor/github.com/containers/common/libnetwork/etchosts/hosts.go
+++ b/vendor/github.com/containers/common/libnetwork/etchosts/hosts.go
@@ -41,6 +41,8 @@ type Params struct {
 	HostContainersInternalIP string
 	// TargetFile where the hosts are written to.
 	TargetFile string
+	// Hostname of the working container
+	Hostname string
 }
 
 // New will create a new hosts file and write this to the target file.
@@ -120,6 +122,10 @@ func new(params *Params) error {
 	if params.HostContainersInternalIP != "" {
 		e := HostEntry{IP: params.HostContainersInternalIP, Names: []string{hostContainersInternal}}
 		containerIPs = append(containerIPs, e)
+		if params.Hostname != "" {
+			c := HostEntry{IP: params.HostContainersInternalIP, Names: []string{params.Hostname}}
+			containerIPs = append(containerIPs, c)
+		}
 	}
 	containerIPs = append(containerIPs, params.ContainerIPs...)
 

[SaurabhAhuja1983]

@flouthoc pinging again. Are you able to reproduce the issue? Can we please include the patch in buildah main repo.

[rhatdan]

Could you open a PR with a test?

[SaurabhAhuja1983]

Yes. Thank You. let me read the guidelines and procedure to contribute and open a PR.

[SaurabhAhuja1983]

@rhatdan can you please re-open this issue. Sorry for coming back after months. We have been dealing with other buildah issues. I am creating a PR to fix this, you can please take a look and review.