More instructive error messages on singularity verify error

[vsoch]

It's going to be common for a user to try to verify an image that hasn't been signed, or was generated with a (pre 3.0) version of Singularity, or some weird combination of those things. Here I'll walk through a set of common scenarios that have unclear messages for what is going on:

Case 1. Pull and verify using Singularity 2.6.0

Version is 2.6.0-HEAD.579c415

$ singularity pull docker://ubuntu

If I try to verify it, I get incorrect usage (which goes against the docs. Note that the 2.6 docs don't seem to have this (so I'm following 3.0). Here is what I see:

USAGE: singularity [...] verify <data object id/groupid> <container path>

I think probably some version of the command was available, but docs never written for it? A good resolution here would be to provide a simple (parallel) page for 2.6 for how to use the older version. The link should be here.

Case 2. Pull with 2.6.0, verify with 3.0

This is a more common use case - we have a container that was previously pulled with 2.6, and we want to verify with 3.0. Now I get an actual error message (and the usage described in the docs is technically correct). Note that I'm running this via a dockerized 3.0 - it shouldn't have issue because we aren't doing anything that needs privileged (correct me if I am wrong here). I mostly don't want to install Singularity on my host because many users are still using 2.6.

$ docker run -v $PWD:/tmp singularityware/singularity:3.0 verify /tmp/ubuntu.simg
Verifying image: /tmp/ubuntu.simg
WARNING: Authentication token file not found : Only pulls of public images will succeed
ERROR:   verification failed: failed to load SIF container file: invalid SIF file: Magic |sqs�
                                                                                              M�
| want |SIF_MAGIC|

That's pretty fugly! Could the error give something more meaningful, like "This image is not a valid SIF file, meaning it was not generated with Singularity 3.0 or greater. This action "verify" is not available"

Case 3. Pull with 3.0, verify with 3.0

This one should be error free! Let's pull first

$ docker run -v $PWD:/tmp singularityware/singularity:3.0 pull --name /tmp/ubuntu-pull.simg docker://ubuntu
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f
 30.62 MiB / 30.62 MiB  6s
Copying blob sha256:da1315cffa03c17988ae5c66f56d5f50517652a622afc1611a8bdd6c00b1fde3
 847 B / 847 B  0s
Copying blob sha256:fa83472a3562898caaf8d77542181a473a84039376f2ba56254619d9317ba00d
 556 B / 556 B  0s
Copying blob sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a
 162 B / 162 B  0s
Copying config sha256:dec80b78b0e6d45cbe2e098c747d28dc7c4ca39cab2608fe22790e68098ae7c4
 2.42 KiB / 2.42 KiB  0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: /tmp/ubuntu-pull.simg

Houston, we have container! Let's try verify

$ docker run -v $PWD:/tmp singularityware/singularity:3.0 verify /tmp/ubuntu-pull.simg
Verifying image: /tmp/ubuntu-pull.simg
WARNING: Authentication token file not found : Only pulls of public images will succeed
ERROR:   verification failed: error while searching for signature blocks: no signatures found for system partition

This message is very good and meaningful, bravo! 🎊

So I think the thing to address are the first two cases when the user is running into issues with conflicting versions. Thanks for fixing this up!

[yhcote]

For Case 1, I am unaware that you can type the command 'verify' in 2.X series of Singularity. Not sure it makes sense. I will defer that to either @jms or @dave. For Case 2 and 3. I will take a look at making those reports more user friendly. Thanks, - Yannick

…

On Fri, Dec 7, 2018 at 2:42 PM Vanessa Sochat ***@***.***> wrote: It's going to be common for a user to try to verify an image that hasn't been signed, or was generated with a (pre 3.0) version of Singularity, or some weird combination of those things. Here I'll walk through a set of common scenarios that have unclear messages for what is going on: Case 1. Pull and verify using Singularity 2.6.0 Version is 2.6.0-HEAD.579c415 $ singularity pull docker://ubuntu If I try to verify it, I get incorrect usage (which goes against the docs <https://www.sylabs.io/guides/3.0/user-guide/signNverify.html>. Note that the 2.6 docs don't seem to have this (so I'm following 3.0). Here is what I see: USAGE: singularity [...] verify <data object id/groupid> <container path> I think probably some version of the command was available, but docs never written for it? A good resolution here would be to provide a simple (parallel) page for 2.6 for how to use the older version. The link should be here <https://www.sylabs.io/guides/2.6/user-guide/signNverify.html>. Case 2. Pull with 2.6.0, verify with 3.0 This is a more common use case - we have a container that was previously pulled with 2.6, and we want to verify with 3.0. Now I get an actual error message (and the usage described in the docs is technically correct). Note that I'm running this via a dockerized 3.0 - it shouldn't have issue because we aren't doing anything that needs privileged (correct me if I am wrong here). I mostly don't want to install Singularity on my host because many users are still using 2.6. $ docker run -v $PWD:/tmp singularityware/singularity:3.0 verify /tmp/ubuntu.simg Verifying image: /tmp/ubuntu.simg WARNING: Authentication token file not found : Only pulls of public images will succeed ERROR: verification failed: failed to load SIF container file: invalid SIF file: Magic |sqs� M� | want |SIF_MAGIC| That's pretty fugly! Could the error give something more meaningful, like "This image is not a valid SIF file, meaning it was not generated with Singularity 3.0 or greater. This action "verify" is not available" Case 3. Pull with 3.0, verify with 3.0 This one should be error free! Let's pull first $ docker run -v $PWD:/tmp singularityware/singularity:3.0 pull --name /tmp/ubuntu-pull.simg docker://ubuntu WARNING: Authentication token file not found : Only pulls of public images will succeed INFO: Starting build... Getting image source signatures Copying blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f 30.62 MiB / 30.62 MiB 6s Copying blob sha256:da1315cffa03c17988ae5c66f56d5f50517652a622afc1611a8bdd6c00b1fde3 847 B / 847 B 0s Copying blob sha256:fa83472a3562898caaf8d77542181a473a84039376f2ba56254619d9317ba00d 556 B / 556 B 0s Copying blob sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a 162 B / 162 B 0s Copying config sha256:dec80b78b0e6d45cbe2e098c747d28dc7c4ca39cab2608fe22790e68098ae7c4 2.42 KiB / 2.42 KiB 0s Writing manifest to image destination Storing signatures INFO: Creating SIF file... INFO: Build complete: /tmp/ubuntu-pull.simg Houston, we have container! Let's try verify $ docker run -v $PWD:/tmp singularityware/singularity:3.0 verify /tmp/ubuntu-pull.simg Verifying image: /tmp/ubuntu-pull.simg WARNING: Authentication token file not found : Only pulls of public images will succeed ERROR: verification failed: error while searching for signature blocks: no signatures found for system partition This message is very good and meaningful, bravo! 🎊 So I think the thing to address are the first two cases when the user is running into issues with conflicting versions. Thanks for fixing this up! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <sylabs/singularity#2435>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM7p4z1r24BoywvGAJXmANlhzITRZrvTks5u2u7ZgaJpZM4ZJQiR> .

[vsoch]

Here is an example of verify with 2.6:

$ singularity --version
2.6.0-HEAD.579c415
$ singularity verify
USAGE: singularity [...] verify <data object id/groupid> <container path>

[yhcote]

Right, I don't know why/how the command was backported to 2.X. It cannot work with the 2.X series of singularity. I am probably out of the loop with something. Hopefully someone else can shed some light on this.

…

On Sun, Dec 9, 2018 at 12:54 PM Vanessa Sochat ***@***.***> wrote: Here is an example of verify with 2.6: $ singularity --version 2.6.0-HEAD.579c415 $ singularity verify USAGE: singularity [...] verify <data object id/groupid> <container path> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <sylabs/singularity#2435 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM7p40Drswx_UG3kJ4HPf9vCmMvLDl_Eks5u3XiIgaJpZM4ZJQiR> .

[vsoch]

It's okay, we can blame the Github ghosties. They are always up to no good, especially since it was recently Halloween time! 🎃

[GodloveD]

I'm unable to replicate that @vsoch.

[vagrant@localhost singularity]$ singularity --version
2.6.0-HEAD.579c415
[vagrant@localhost singularity]$ singularity verify
ERROR: Unknown command 'verify'

Ghosties. Clearly.

[vsoch]

Weird! Is it possible that there is some crisscrossing of singularity versions? Since it’s doing this weird behavior on my machine, can we do some kind of trace to see what’s happening?

We should at least try to crash the ghostie party.

[GodloveD]

Dunno. Do you have 2 versions of Singularity installed on your machine. Can you replicate if you spin up a fresh VM and install there?

[vsoch]

Here's the relevant code! There was indeed a verify.exec, and verify.info with this version of singularity:

...
open("/usr/local/libexec/singularity/cli/verify.exec", O_RDONLY) = 3
ioctl(3, TCGETS, 0x7ffd75be5770)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "#!/bin/bash\n#\n# Copyright (c) 20"..., 80) = 80
lseek(3, 0, SEEK_SET)                   = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024*1024}) = 0
fcntl(255, F_GETFD)                     = -1 EBADF (Bad file descriptor)
dup2(3, 255)                            = 255
close(3)                                = 0
fcntl(255, F_SETFD, FD_CLOEXEC)         = 0
fcntl(255, F_GETFL)                     = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(255, {st_mode=S_IFREG|0755, st_size=2789, ...}) = 0
lseek(255, 0, SEEK_CUR)                 = 0
brk(0x1af0000)                          = 0x1af0000
brk(0x1af1000)                          = 0x1af1000
read(255, "#!/bin/bash\n#\n# Copyright (c) 20"..., 2789) = 2789
stat("/usr/local/libexec/singularity/functions", {st_mode=S_IFREG|0755, st_size=17310, ...}) = 0
open("/usr/local/libexec/singularity/functions", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=17310, ...}) = 0
brk(0x1af9000)                          = 0x1af9000
read(3, "#!/bin/bash\n#\n# Copyright (c) 20"..., 17310) = 17310
close(3)                                = 0
brk(0x1afa000)                          = 0x1afa000
brk(0x1afb000)                          = 0x1afb000
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
brk(0x1afc000)                          = 0x1afc000
...
brk(0x1b49000)                          = 0x1b49000
stat("/usr/local/libexec/singularity/cli/verify.info", {st_mode=S_IFREG|0644, st_size=733, ...}) = 0
open("/usr/local/libexec/singularity/cli/verify.info", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=733, ...}) = 0
read(3, "NAME=\"verify\"\nSECTION=\"managemen"..., 733) = 733
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
write(1, "USAGE: singularity [...] verify "..., 74USAGE: singularity [...] verify <data object id/groupid> <container path>
) = 74
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Here are the files:

$ ls /usr/local/libexec/singularity/cli/verify*
/usr/local/libexec/singularity/cli/verify.exec  /usr/local/libexec/singularity/cli/verify.info

I did a quick Google search for the file, and it does appear that it existed at some point!
https://git.biohpc.swmed.edu/biohpc/singularity/commit/f645833eb11c845cbc32e1c18229353186cc38fa?expanded=1&w=1#0e6513b9094af510cd32ca30c526ea060ea18193
Likely the commit was cherry picked / cleaned up, or other, but I happen to still have the clone, for one reason or another.

[GodloveD]

OK. Seems like this was some kind of a weird snafu. I'm closing. 😄

[GodloveD]

Oh whoops. There is more to this. re-opening. Don't mind me.

[hugobuddel]

First time singularity user here, following the quick start guide:

$ singularity pull library://sylabsed/linux/alpine

INFO:    Downloading library image
WARNING: unable to verify container: error while searching for signature blocks: no signatures found for system partition
This image is not signed, and thus its contents cannot be verified.
Do you with to proceed? [N/y] 

Can I trust this image? It seems that sylabsed is the official repository, so it should be trustworthy. But the container is not signed, and the default is to not proceed. So maybe the repository is compromised?

Probably everything is fine, but this being my first ever singularity command after singularity version, it is not clear what to expect. (Version is 3.2.1-1.el7.)

[GodloveD]

@hugobuddel yeah everything is likely OK. That warning is just telling you that the container has not been signed. Does not indicate that the library has been compromised or anything. Just means that Library admins did not sign the container after creation. Maybe it was just created from DockerHub and we did not feel confident signing it? Unsure.

[carterpeel]

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks,
Carter

[stale]

This issue has been automatically marked as stale because it has not had activity in over 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[pedroalvesbatista]

@vsoch @GodloveD Still getting this problem ? Have this been solved already ? If yes, what work around have you followed and applied ?

We're looking into the issue carefully, soon will bring to community and discuss ways to better solve as well address this. Thankyou for keeping the interest in the subject.

[vsoch]

I think it’s probably unlikely that someone is still using 2.x AND using the verify command, so it’s probably safe to close this.

[kmuriki]

Pending issues from the old repo copied to the new repo (apptainer/apptainer#1390) and cleaned from the old, retired repo.