curl and dns #16

kforner · 2016-03-30T16:40:00Z

Trying to investigate dns related issues, I made a curl sapp.

I can not pinpoint what's needed to make it work, but here's my findings:

Name: curl
Exec: /usr/bin/curl

does not work:
./curl.sapp google.fr
curl: (6) Could not resolve host: google.fr

Name: curl
Exec: /usr/bin/curl
%files:
/lib/x86_64-linux-gnu/

this works !

But from here, I used singularity strace to get the exact list of used libs, I pasted them in the %files section, but never managed to get it to work:

singularity strace 2>curl.err curl google.fr
parsing it gives me the following list of files:

/lib/i386-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libcom_err.so.2
/lib/x86_64-linux-gnu/libcrypt.so.1
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libgcrypt.so.11
/lib/x86_64-linux-gnu/libgpg-error.so.0
/lib/x86_64-linux-gnu/libkeyutils.so.1
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libresolv.so.2
/lib/x86_64-linux-gnu/libssl.so.1.0.0
/lib/x86_64-linux-gnu/libz.so.1
/usr/lib/x86_64-linux-gnu/libasn1.so.8
/usr/lib/x86_64-linux-gnu/libcurl.so.4
/usr/lib/x86_64-linux-gnu/libffi.so.6
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
/usr/lib/x86_64-linux-gnu/libgssapi.so.3
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
/usr/lib/x86_64-linux-gnu/libhcrypto.so.4
/usr/lib/x86_64-linux-gnu/libheimbase.so.1
/usr/lib/x86_64-linux-gnu/libheimntlm.so.0
/usr/lib/x86_64-linux-gnu/libhx509.so.5
/usr/lib/x86_64-linux-gnu/libidn.so.11
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3
/usr/lib/x86_64-linux-gnu/libkrb5.so.26
/usr/lib/x86_64-linux-gnu/libkrb5.so.3
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0
/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0
/usr/lib/x86_64-linux-gnu/libroken.so.18
/usr/lib/x86_64-linux-gnu/librtmp.so.0
/usr/lib/x86_64-linux-gnu/libsasl2.so.2
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
/usr/lib/x86_64-linux-gnu/libtasn1.so.6
/usr/lib/x86_64-linux-gnu/libwind.so.0

What could be the problem ?

Thanks.

The text was updated successfully, but these errors were encountered:

truatpasteurdotfr · 2016-03-30T18:49:35Z

reducing the scope: the minimal ping.sspec shows the same issue (CentOS-6 x86_64)

[tru@visu1 singularity]$ cat ping.sspec 
Name: ping
Exec: /bin/ping
[tru@visu1 singularity]$ singularity build ping.sspec 
...
[tru@visu1 singularity]$ ./ping.sapp -c 1 google.com
ping: unknown host google.com
[tru@visu1 singularity]$ ping -c 1 google.com
PING google.com (172.217.19.142) 56(84) bytes of data.
64 bytes from par03s12-in-f14.1e100.net (172.217.19.142): icmp_seq=1 ttl=53 time=2.53 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.533/2.533/2.533/0.000 ms

gmkurtzer · 2016-03-30T19:03:46Z

So, I'm able to build a curl sapp and run it without issue:

`[gmk@centos7-x64 demo]$ cat example.sspec
Name: curl
Exec: /bin/curl
[gmk@centos7-x64 demo]$ singularity --quiet build example.sspec
Running test...
Hello from within the container... (no test code defined)
WROTE: curl.sapp
[gmk@centos7-x64 demo]$ ./curl.sapp google.com

<TITLE>301 Moved</TITLE>

301 Moved

The document has moved
here.

[gmk@centos7-x64 demo]$ `

Ping is an interesting example, because technically to send an ICMP you need to special system privileges. This is what I'm getting on Centos 7 x64:

$ ./ping.sapp -c 1 google.com ping: icmp open socket: Operation not permitted [gmk@centos7-x64 demo]$

So, I'm guessing something is different on my system because it is resolving properly. DNS resolution usually happens by libc dlopen'ing name resolution libraries. Do this please:

$ singularity specgen curl google.com

and then paste the resulting curl.sspec (if both of you can do that, it would be great). Singularity always needs to included these libraries into a container.

Thanks!

truatpasteurdotfr · 2016-03-30T19:10:17Z

[tru@visu1 singularity]$ \rm -rf ~/.singularity-cache/
[tru@visu1 singularity]$ singularity list
CONTAINER NAME       UUID                                          SIZE SUMMARY
[tru@visu1 singularity]$  singularity specgen curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.fr/?gfe_rd=cr&amp;ei=HiT8VuLZOcHI8gfj0K-ABQ">here</A>.
</BODY></HTML>
WROTE: curl.sspec
[tru@visu1 singularity]$ singularity build curl.sspec 
...
WROTE: curl.sapp
Cleaning up temporary files...
[tru@visu1 singularity]$ ./curl.sapp google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.fr/?gfe_rd=cr&amp;ei=XiT8VqWmOcHI8gfj0K-ABQ">here</A>.
</BODY></HTML>

so it works with the "specgen"erated file :)

truatpasteurdotfr · 2016-03-30T19:11:30Z

attaching the generated file on CentOS-6
curl.sspec.txt

gmkurtzer · 2016-03-30T19:18:56Z

I betcha it is the libnss3 and libnssutil3. I don't think I'm adding those automatically... That should catch it for Centos6. ;)

Thanks Tru!

gmkurtzer · 2016-03-30T19:26:07Z

If you retest from trunk, it should now grab those two libraries when building a SAPP. Hopefully that will fix it for both cases! Karl, can you retest current master?

Thanks!

truatpasteurdotfr · 2016-03-30T19:40:39Z

:(

[tru@visu1 singularity]$ singularity   build curl.sspec 2>&1 |grep nss
Installing file: /lib64/libnss_files.so.2
Installing file: /lib64/libnss_dns.so.2
Installing file: /lib64/libnss3.so
Installing file: /usr/lib64/libnssutil3.so
Installing file: /lib64/libnssutil3.so
Installing file: /usr/lib64/libnss3.so

but

[tru@visu1 singularity]$ ./curl.sapp google.com
curl: (6) Couldn't resolve host 'google.com'

gmkurtzer · 2016-03-30T19:42:39Z

Hi Tru,

Can you run another test for me? Please capture all of the out from this command in /tmp/debug, and send me that file please:

$ singularity strace curl google.com >/tmp/debug 2>&1

Thanks!

kforner · 2016-03-31T08:31:07Z

my curl.sspec:

curl.sspec.txt

I tried the ping sapp: same problem:

./ping.sapp google.fr
ping: unknown host google.fr

the curl strace:
curl.strace.txt

gmkurtzer · 2016-03-31T13:48:20Z

Hi Karl, I'm no at computer yet (so I'm not sure what the attachments are) but can you also do the strace against curl? Ping is an odditity because it always requires additional permission to run. Up until just recently ping was setuid root. Now it generally uses Linux capabilities to achieve similar results which is why I'm not counting on ping to ever work properly in a singularity container.

Probably the smarter thing would be for singularity to check on files with non standard permissions and error if one is found.

Thanks again!

kforner · 2016-03-31T13:53:00Z

can you also do the strace against curl?
look at the previous post !

gmkurtzer · 2016-03-31T14:06:36Z

Oh, Tru's response? I wasn't making the assumption that you are running on the same host OS. If you are that will indeed work fine and will use what he provided.

Will investigate more soon as I'm at a real keyboard.

Thanks!

gmkurtzer · 2016-03-31T22:45:32Z

Heya Karl,

Sorry for the confusion about not seeing your attachments. Looking at them now, and I'm noticing that libnss_myhostname.so.* was not installed into your container. I am debugging why that might have happened as that file should always be installed (as of several days ago!).

Thanks!

gmkurtzer · 2016-03-31T23:02:03Z

ok, I think Karl's issue is now fixed now in master.

The problem was that I made an assumption that the libraries I am statically including (e.g. libnss_myhostname.so.) exist right off of the /lib directories. But on Debian like systems they are actually 2 levels deep. Please test. :)

Tru, can you also test against the git master?

Thanks!

kforner · 2016-04-01T11:51:18Z

Still does not work for me.
I do not know if it is relevant, but for nslookup.spec, I had to include
/usr/lib/x86_64-linux-gnu/openssl-1.0.0

gmkurtzer · 2016-04-01T13:53:49Z

Can you try that for curl too, and then if it works send me an strace of it?

Can you remind me, when you used specgen to create the spec for curl, did it work? If so can you show me that spec?

Thanks!

kforner · 2016-04-01T14:10:23Z

Can you try that for curl too,
It did not work for curl.

Can you remind me, when you used specgen to create the spec for curl, did it work?

nope

same spec as above

gmkurtzer · 2016-04-01T14:23:52Z

What distribution are you using?

Can you send me your curl SAPP? (Either attach to the ticket or email direct to me please).

Thanks!

kforner · 2016-04-01T14:30:30Z

ubuntu 14.04

Can you send me your curl SAPP? (Either attach to the ticket or email direct to me please).
which curl sapp ? and which email ? the berkeley one ?

gmkurtzer · 2016-04-01T14:50:09Z

I have it running on another Debian/Ubuntu based distribution (Lubuntu) without a problem.

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but I am creating a generic /etc/nsswitch.conf. I wonder if that is the problem... Might be worth copying your hosts's /etc/nsswitch.conf into ~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for testing.

kforner · 2016-04-01T14:59:49Z

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but I am creating a generic >/etc/nsswitch.conf. I wonder if that is the problem... Might be worth copying your hosts's >/etc/nsswitch.conf into ~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for testing.

I tried copying both the nsswitch.conf then the resolv.conf without success.

But remember, if I add /lib/x86_64-linux-gnu/ in %files it works.
Could it be related to pam, sssd, etc... ?

kforner · 2016-04-01T15:00:07Z

On Fri, Apr 1, 2016 at 4:50 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

I have it running on another Debian/Ubuntu based distribution (Lubuntu)
without a problem.

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but
I am creating a generic /etc/nsswitch.conf. I wonder if that is the
problem... Might be worth copying your hosts's /etc/nsswitch.conf into
~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for
testing.

in any case, I think that files explicitly specified in the %files should
override your templates.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/gmkurtzer/singularity/issues/16#issuecomment-204425317

gmkurtzer · 2016-04-01T15:47:12Z

But remember, if I add /lib/x86_64-linux-gnu/ in %files it works. Could it be related to pam, sssd, etc... ?

Yes, we are missing some library that is getting dlopen'ed probably by libc itself would be my guess. But why we aren't able to detect it is worrysome. I went over your specgen output and your strace output of library open(), and there were no differences.

I didn't get the SAPP file though, did you send it? Can you also send me the SAPP file where it worked when you included all of /lib/x86_64-linux-gnu/? It will be bigger, so maybe a dropbox or other URL link if it doesn't fit in the ticket?

Thanks

gmkurtzer · 2016-04-01T15:49:38Z

in any case, I think that files explicitly specified in the %files should override your templates.

Oh, this is easy enough to do, but.... It does inherently break portability and possibly expose hosts (targets) to people who are distributing the SAPP file. Let me think about that.

kforner · 2016-04-04T09:22:48Z

I didn't get the SAPP file though, did you send it? Can you also send me the SAPP file where it worked when you included all of /lib/x86_64-linux-gnu/?

Here they are:

Oh, this is easy enough to do, but.... It does inherently break portability and possibly expose hosts >(targets) to people who are distributing the SAPP file. Let me think about that.

My point is that if you explicitly add them to %files, then you (should) know what you're doing.
You could maintain a list of sensitive files, and emit warnings if they are in %files...

gmkurtzer · 2016-04-04T14:46:11Z

I'm not sure if this is related or not, but both are picking up a library in a non-standard location:

/home/karl/bin/libnss_dns.so.2

Can you confirm that /bin/ldd is finding that, and that it is necessary?

Thanks!

kforner · 2016-04-04T14:55:40Z

there is no /home/karl/bin/libnss_dns.so.2

but there is a /home/karl/bin/libstderred.so which is loaded via
LD_PRELOAD=/home/karl/bin/libstderred.so
Could it be related ?

On Mon, Apr 4, 2016 at 4:46 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

I'm not sure if this is related or not, but both are picking up a library
in a non-standard location:

/home/karl/bin/libnss_dns.so.2

Can you confirm that /bin/ldd is finding that, and that it is necessary?

Thanks!

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/gmkurtzer/singularity/issues/16#issuecomment-205328751

gmkurtzer · 2016-04-04T14:57:15Z

No, I don't think that is related. Is your LD_LIBRARY_PATH set?

I found another thing very interesting:

The container that works has multiple copies of libnss_files and libnss_dns and they aren't the same.

$ md5sum lib64/libnss_files.so.2 lib/x86_64-linux-gnu/libnss_files.so.2
5b3b1eb6d2be03550623921a55c1b41f lib64/libnss_files.so.2
67f25cd8773539eda122b6b44c117048 lib/x86_64-linux-gnu/libnss_files.so.2
$ md5sum lib64/libnss_dns.so.2 lib/x86_64-linux-gnu/libnss_dns.so.2
36286bd509a1cf64ad086e3ca33255df lib64/libnss_dns.so.2
4c611c519cbd06d59ec35f8d210befdb lib/x86_64-linux-gnu/libnss_dns.so.2

To that end I just changed some of the code in master, hoping to fix that. Can you try again?

Thanks!

kforner · 2016-04-04T15:02:51Z

It works !!!

On Mon, Apr 4, 2016 at 4:57 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

No, I don't think that is related. Is your LD_LIBRARY_PATH set?

I found another thing very interesting:

The container that works has multiple copies of libnss_files and
libnss_dns and they aren't the same.

$ md5sum lib64/libnss_files.so.2 lib/x86_64-linux-gnu/libnss_files.so.2
5b3b1eb6d2be03550623921a55c1b41f lib64/libnss_files.so.2
67f25cd8773539eda122b6b44c117048 lib/x86_64-linux-gnu/libnss_files.so.2
$ md5sum lib64/libnss_dns.so.2 lib/x86_64-linux-gnu/libnss_dns.so.2
36286bd509a1cf64ad086e3ca33255df lib64/libnss_dns.so.2
4c611c519cbd06d59ec35f8d210befdb lib/x86_64-linux-gnu/libnss_dns.so.2

To that end I just changed some of the code in master, hoping to fix that.
Can you try again?

Thanks!

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/gmkurtzer/singularity/issues/16#issuecomment-205335807

gmkurtzer · 2016-04-04T15:04:49Z

Fantastic! Thank you so much for your patience while I wrap my head around it. I also wrote a note in the TODO to think about how to handle the issue with sensitive files in %files and how to include or overwrite them.

BTW, It was very cool getting your SAPPs and being able to replicate your exact problem. lol

kforner · 2016-04-04T15:06:15Z

BTW, It was very cool getting your SAPPs and being able to replicate your exact problem.
yes we'll know for the next time ;)

thanks for the fix.

Null-terminator for buf in src/lib/image/ext3/init.c

Remove old-style ISSUE_TEMPLATE.md

kforner closed this as completed Apr 4, 2016

GodloveD pushed a commit that referenced this issue Apr 27, 2018

Merge pull request #16 from sylabs/fix-issue-11

d4e0c34

Null-terminator for buf in src/lib/image/ext3/init.c

robbinslewis mentioned this issue Nov 2, 2020

rhel6 kernel panic #5681

Closed

pedroalvesbatista referenced this issue in pedroalvesbatista/singularity Jun 8, 2021

Merge pull request #16 from dtrudg/legacy-issue-tpl

b9f5717

Remove old-style ISSUE_TEMPLATE.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curl and dns #16

curl and dns #16

kforner commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

kforner commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

kforner commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016

curl and dns #16

curl and dns #16

Comments

kforner commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

301 Moved

truatpasteurdotfr commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

truatpasteurdotfr commented Mar 30, 2016

gmkurtzer commented Mar 30, 2016

kforner commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

kforner commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

gmkurtzer commented Mar 31, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 1, 2016

kforner commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

gmkurtzer commented Apr 1, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016

gmkurtzer commented Apr 4, 2016

kforner commented Apr 4, 2016