Device or resource busy - getaddrinfo when doing ruby http calls on ubuntu

[grosser]

Device or resource busy - getaddrinfo (Errno::EBUSY)

did not find anything on the web that would help ... something about ulimit might be setup wrong :/

require 'net/http'
Net::HTTP.get_response(URI('http://www.google.com'))

ulimit seems fine:
/proc/sys/fs/file-max -> 524288

FROM ruby as ruby

ADD https://github.com/coord-e/magicpak/releases/download/v1.3.0/magicpak-x86_64-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak && /usr/bin/magicpak -v "$(which ruby)" /magicpak

FROM scratch

COPY --from=ruby /magicpak /.
COPY --from=ruby /usr/local/lib/ruby /usr/local/lib/ruby

RUN ["ruby", "-r", "net/http", "-e", "Net::HTTP.get_response(URI('http://www.google.com'))"]

[grosser]

I got something working here:

docker run --platform linux/amd64 -it --rm unasuke/distroless-ruby ruby -r net/http -e "puts Net::HTTP.get('example.com', '/index.html')"

... so just need more digging
then turned out the issue was

FROM scratch

vs

FROM gcr.io/distroless/base-debian10

[grosser]

so kind of solved ... but would still be interesting to know why this happens ... but low priority

[coord-e]

Listed necessary runtime dependency using --dynamic:
https://gist.github.com/coord-e/933b458e1b8e9ef76d5465075ac10a9a

$ cat dynamic.log | grep 'emit:' | grep -v /usr/local/lib/ruby
 INFO action: emit: creating destination dir as it does not exist dest=/magicpak
 INFO emit: copy from=/usr/lib/locale/C.UTF-8/LC_CTYPE target=/magicpak/usr/lib/locale/C.UTF-8/LC_CTYPE
 INFO emit: copy from=/etc/nsswitch.conf target=/magicpak/etc/nsswitch.conf
 INFO emit: link link=/usr/share/zoneinfo/Etc/UTC target=/magicpak/etc/localtime
 INFO emit: copy from=/usr/share/zoneinfo/Etc/UTC target=/magicpak/usr/share/zoneinfo/Etc/UTC
 INFO emit: link link=/lib/x86_64-linux-gnu/libnss_files-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libnss_files.so.2
 INFO emit: copy from=/lib/x86_64-linux-gnu/libnss_files-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libnss_files-2.31.so
 INFO emit: copy from=/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache target=/magicpak/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
 INFO emit: copy from=/etc/host.conf target=/magicpak/etc/host.conf
 INFO emit: link link=/lib/x86_64-linux-gnu/librt-2.31.so target=/magicpak/lib/x86_64-linux-gnu/librt.so.1
 INFO emit: copy from=/lib/x86_64-linux-gnu/librt-2.31.so target=/magicpak/lib/x86_64-linux-gnu/librt-2.31.so
 INFO emit: link link=/usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1 target=/magicpak/usr/lib/x86_64-linux-gnu/libgmp.so.10
 INFO emit: copy from=/usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1 target=/magicpak/usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1
 INFO emit: link link=/lib/x86_64-linux-gnu/ld-2.31.so target=/magicpak/lib64/ld-linux-x86-64.so.2
 INFO emit: copy from=/lib/x86_64-linux-gnu/ld-2.31.so target=/magicpak/lib/x86_64-linux-gnu/ld-2.31.so
 INFO emit: link link=/lib/x86_64-linux-gnu/libpthread-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libpthread.so.0
 INFO emit: copy from=/lib/x86_64-linux-gnu/libpthread-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libpthread-2.31.so
 INFO emit: copy from=/etc/gai.conf target=/magicpak/etc/gai.conf
 INFO emit: link link=/lib/x86_64-linux-gnu/libnss_dns-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libnss_dns.so.2
 INFO emit: copy from=/lib/x86_64-linux-gnu/libnss_dns-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libnss_dns-2.31.so
 INFO emit: copy from=/etc/ld.so.cache target=/magicpak/etc/ld.so.cache
 INFO emit: copy from=/etc/hosts target=/magicpak/etc/hosts
 INFO emit: copy from=/proc/self/maps target=/magicpak/proc/self/maps
 INFO emit: link link=/lib/x86_64-linux-gnu/libdl-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libdl.so.2
 INFO emit: copy from=/lib/x86_64-linux-gnu/libdl-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libdl-2.31.so
 INFO emit: link link=/lib/x86_64-linux-gnu/libm-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libm.so.6
 INFO emit: copy from=/lib/x86_64-linux-gnu/libm-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libm-2.31.so
 INFO emit: copy from=/etc/resolv.conf target=/magicpak/etc/resolv.conf
 INFO emit: link link=/usr/local/lib/libruby.so.3.1.2 target=/magicpak/usr/local/lib/libruby.so.3.1
 INFO emit: copy from=/usr/local/lib/libruby.so.3.1.2 target=/magicpak/usr/local/lib/libruby.so.3.1.2
 INFO emit: link link=/lib/x86_64-linux-gnu/libz.so.1.2.11 target=/magicpak/lib/x86_64-linux-gnu/libz.so.1
 INFO emit: copy from=/lib/x86_64-linux-gnu/libz.so.1.2.11 target=/magicpak/lib/x86_64-linux-gnu/libz.so.1.2.11
 INFO emit: copy from=/usr/local/bin/ruby target=/magicpak/usr/local/bin/ruby
 INFO emit: link link=/lib/x86_64-linux-gnu/libc-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libc.so.6
 INFO emit: copy from=/lib/x86_64-linux-gnu/libc-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libc-2.31.so
 INFO emit: link link=/lib/x86_64-linux-gnu/libcrypt.so.1.1.0 target=/magicpak/lib/x86_64-linux-gnu/libcrypt.so.1
 INFO emit: copy from=/lib/x86_64-linux-gnu/libcrypt.so.1.1.0 target=/magicpak/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
 INFO emit: link link=/lib/x86_64-linux-gnu/libresolv-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libresolv.so.2
 INFO emit: copy from=/lib/x86_64-linux-gnu/libresolv-2.31.so target=/magicpak/lib/x86_64-linux-gnu/libresolv-2.31.so

Then I can get it working by manually including minimal set of files from the list above (but still needs a patch)
https://gist.github.com/coord-e/ee16d2c30a0670ff4e9849364378770c

FROM gcr.io/distroless/base-debian10 works because it already contains the necessary files shown above.

$ docker run --rm gcr.io/distroless/base-debian10:debug -c 'ls /lib/x86_64-linux-gnu/' | grep -e libnss_dns -e libresolv
libnss_dns-2.28.so
libnss_dns.so.2
libresolv-2.28.so
libresolv.so.2

[coord-e]

libnss_dns and libresolv are usually distributed together with glibc, and getaddrinfo(3) in glibc loads these libs at runtime (probably by dlopen(3))

$ cat test.c
#include <sys/types.h>
#include <stddef.h>
#include <sys/socket.h>
#include <netdb.h>

int main(int argc, char** argv) {
  struct addrinfo *result;
  if (getaddrinfo("google.com", NULL, NULL, &result) != 0) {
    return 1;
  }
  freeaddrinfo(result);
}

$ gcc test.c -o test
$ ldd test
        linux-vdso.so.1 (0x00007ffda1525000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe3c56d5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe3c58b9000)

$ strace ./test 2>&1 | grep -e libresolv -e libnss_dns
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3

So the existence of these libs seems to be an internal convention of glibc and I think it's difficult to detect them statically...

[coord-e]

actually every libnss_* are potentially loaded by nss (see nsswtich.conf(5)) and libresolv is just linked to libnss_dns.

$ ldd /lib/x86_64-linux-gnu/libnss_dns.so.2
        linux-vdso.so.1 (0x00007ffcc3d38000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fb725109000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb724f34000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb725135000)

So the solution is:

• implement a variant of --include which searches for statically linked dependencies
• document to use the flag to include libnss_* libraries for applications which may use NSS

[grosser]

awesome 🙇‍♂️

[grosser]

is there some ugly list of COPY instructions I can use in the meantime ?

[coord-e]

COPY --from=ruby /lib/x86_64-linux-gnu/libnss_dns.so.2 /lib/x86_64-linux-gnu/libnss_dns.so.2
COPY --from=ruby /lib/x86_64-linux-gnu/libresolv.so.2 /lib/x86_64-linux-gnu/libresolv.so.2

would be sufficient. like this: https://gist.github.com/coord-e/03e5b183e7317ab6502b01bff68f67e6

[coord-e]

v1.3.2 is out and now you can use additional --include as noted in README to address the problem.

FROM ruby as ruby

ADD https://github.com/coord-e/magicpak/releases/download/v1.3.2/magicpak-x86_64-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak && /usr/bin/magicpak -v "$(which ruby)" /magicpak --include '/lib/x86_64-linux-gnu/libnss_*'

FROM scratch

COPY --from=ruby /magicpak /.
COPY --from=ruby /usr/local/lib/ruby /usr/local/lib/ruby

RUN ["ruby", "-r", "net/http", "-e", "Net::HTTP.get_response(URI('http://www.google.com'))"]

[grosser]

awesome, thx!

[grosser]

so I can confirm that "Device or resource busy" is gone ...
but now I'm getting "Name or service not known" 🤦

docker run --rm --entrypoint '' <IMAGE> ruby -rnet/http -e "puts Net::HTTP.get_response(URI('http://www.google.com'))"
/usr/lib/ruby/3.1.0/socket.rb:227:in `getaddrinfo': Failed to open TCP connection to www.google.com:80 (getaddrinfo: Name or service not known) (SocketError)

... but it works fine with

docker run --rm --entrypoint '' ruby ruby -rnet/http -e "puts Net::HTTP.get_response(URI('http://www.google.com'))"

I'm using the new

RUN /usr/bin/magicpak --verbose --include '/lib/aarch64-linux-gnu/libnss_*' $(which ruby) /magicpak

• already checked /etc/resolv.conf is the same in both containers
• working: already tried --include '/lib/${ARCH}-linux-gnu/*'
• working: FROM <same as original image> instead of scratch
  ... so there is 1 more file I need

[grosser]

finally :D

RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" $(which ruby) /magicpak

... this might be good for the readme since it's not obvious :D

[grosser]

and from some reason "/lib/${ARCH}-linux-gnu/{libnss_,libresolv}*" does not work, so the expansion logic could use a bugfix too

[coord-e]

hmm...

1. can you confirm that you are using v1.3.2 of magicpak?
2. can you attach the output of docker run --rm --entrypoint '' <original base image> ldd /lib/x86_64-linux-gnu/libnss_dns.so.2 ?

"/lib/${ARCH}-linux-gnu/{libnss_,libresolv}*" does not work

--include accepts glob patterns, and the glob pattern is different from brace expansion in shells (the {a,b} syntax), so this is an expected behavior.

[grosser]

it works fine with

RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" $(which ruby) /magicpak

re glob: I though that would be the same library to do that, if it's easy to do I'd recommend adding but no big deal if it's not