2 issues while trying to build pydmesg

[remyd1]

Hi,

I tried to create a sapp of a python script ( https://gist.githubusercontent.com/dopuskh3/9908ac6b31133eb54331/raw/74beaa1f2c64e09b6152b1ac8849af172c12b6e1/pydmesg ), but I had some issues :

/opt/singularity/singularity-1.0/bin/singularity specgen /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg
...
WROTE: pydmesg.sspec

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity build pydmesg.sspec
....
Running post scriptlet
Defining container shell environment
Building singularity main runscript
Building singularity test runscript
Building singularity exec runscript
Running test...
Including specfile
Writing metadata
Creating output SAPP container
WROTE: pydmesg.sapp
Cleaning up temporary files...

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity install pydmesg.sapp
Installing: pydmesg (5681a8f5-31f9-49b5-986e-61f06dac48a0)
Done...

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity check pydmesg.sapp
ERROR: Singularity container is not installed: pydmesg.sapp

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity install pydmesg.sapp
Singularity container is already installed: pydmesg (5681a8f5-31f9-49b5-986e-61f06dac48a0)

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity run pydmesg.sapp
/.singularity.exec: 2: exec: /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg: not found

remy@atlas:/tmp$ grep /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg pydmesg.sspec 
exec /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg "$@"
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg

remy@atlas:/tmp$/opt/singularity/singularity-1.0/bin/singularity strace pydmesg.sapp
ERROR: Singularity container is not installed: pydmesg.sapp

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity list |grep 5681a8f5-31f9-49b5-986e-61f06dac48a0
pydmesg              5681a8f5-31f9-49b5-986e-61f06dac48a0           33M Generated by 'singularity specgen'

remy@atlas:/tmp$ ./pydmesg.sapp 
-ne \e[0;31mERROR: File checksums not found for this SAPP
\e[0;39m

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/sapprun  pydmesg.sapp 
-ne \e[0;31mERROR: SAPP directory not found
\e[0;39m
-ne \e[0;31mNot launching SAPP container
\e[0;39m

I tried also with relative path, same error.

After editing sspec file, it seems it missed /var/log/dmesg and /proc/uptime. I added it and reproduce the creation of the sapp, but obviously, I still get that error.

I also tried with another "pydmesg" piece of code but I had the same issue.

BTW, I wonder how singularity analyze python code (import stuff, open, or when we append a path to PYTHONPATH (e.g. : sys.path.append('/path/to/whatever')...), or if it just add the whole PYTHONPATH directory to sapp file.

Regards,
Remy

[gmkurtzer]

Hi,

I can write a more detailed response when I get to my computer keyboard (phone isn't great for that) but quickly something that I noticed should help... Referencing a SAPP file by file name doesn't work for singularity commands (except run and install). For example:

$ singularity check foo.sapp

Will fail because the container name is not foo.sapp. Chances are it is just "foo" but that may not be guaranteed (eg someone renaming a sapp file). The output of "singularity list" will show you the installed container name. Use that name (or UUID) to address an installed container.

Now the other issues are packaging issues from what I can tell. Singularity will try and resolve dependencies as best it can but in many cases it is a trial and error of playing with the %files section. I will look when I get to keyboard and see if I see anything else that might be helpful.

Let me know if you get any further!

Greg

[remyd1]

Hi Greg,

With sapp UUID (the 'check' works; "Réussi" is french, it means succeeded) :

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity run 5681a8f5-31f9-49b5-986e-61f06dac48a0
/.singularity.exec: 2: exec: /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg: not found

remy@atlas:/tmp$ /opt/singularity/singularity-1.0/bin/singularity strace 5681a8f5-31f9-49b5-986e-61f06dac48a0
...
close(4)                                = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x7f7615382460, ~[RTMIN RT_1], SA_RESTORER, 0x7f7614dbcd40}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7f7614dbcd40}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7f7614dbcd40}, NULL, 8) = 0
read(10, "#!/bin/sh\nexec /media/remy/DATAP"..., 8192) = 77
execve("/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg", ["/media/remy/DATAPART1/codes/9908"...], [/* 83 vars */]) = -1 ENOENT (No such file or directory)
write(2, "/.singularity.exec: 2: exec: ", 29/.singularity.exec: 2: exec: ) = 29
write(2, "/media/remy/DATAPART1/codes/9908"..., 67/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg: not found) = 67
write(2, "\n", 1
)                       = 1
exit_group(127)                         = ?
+++ exited with 127 +++

/opt/singularity/singularity-1.0/bin/singularity check pydmesg
./.singularity.env: Réussi
./lib/x86_64-linux-gnu/libresolv.so.2: Réussi
./lib/x86_64-linux-gnu/librt.so.1: Réussi
./lib/x86_64-linux-gnu/libz.so.1: Réussi
./lib/x86_64-linux-gnu/libnss_files.so.2: Réussi
./lib/x86_64-linux-gnu/libnss_dns.so.2: Réussi
...
./usr/bin/strace: Réussi
./bin/sh: Réussi
./var/log/dmesg: Réussi


remy@atlas:/tmp$ grep -B1 /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg pydmesg.sspec 
%runscript
exec /media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg "$@"
--
%files
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg
--
/lib/x86_64-linux-gnu/libz.so.1
/media/remy/DATAPART1/codes/9908ac6b31133eb54331/pydmesg

edit : but as you can see, the container still does not work

Best,

Remy

[gmkurtzer]

Heya Remy,

Again, you don't need to reference the installed container via it's UUID, but that is an option. Notice container name when you do "singularity list", it does not include the ".sapp" suffix. If you include the suffix, it won't recognize the container name.

I was able to get your example working but it took both running the latest master branch (as I just committed several fixes) and some additions to the specfile. Specifically there were two dependencies that could not be automatically probed: "string-escape" and the shell command "/usr/bin/dmesg". Once I added those, it worked just fine:

`
[gmk@centos7-x64 demo]$ cat example.sspec
Name: pydmesg
Exec: ./pydmesg.py

%files
/usr/lib64/python2.7/encodings/string_escape.py
/usr/bin/dmesg
[gmk@centos7-x64 demo]$ singularity --quiet build example.sspec
Running test...
Hello from within the container... (no test code defined)
WROTE: pydmesg.sapp
[gmk@centos7-x64 demo]$ ./pydmesg.sapp | tail -n 5
[2016-04-25 08:34:48] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[2016-04-25 09:02:48] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[2016-04-25 09:03:46] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[2016-04-25 09:06:43] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[2016-04-25 09:07:25] SELinux: initialized (dev proc, type proc), uses genfs_contexts
`

[remyd1]

Thanks Greg !

I will try that asap !

Best
Remy

[remyd1]

Ok, it works.
Thanks !