Adding podman as container base

[vsoch]

This is a first shot at adding podman, and re-organizing the library so that most interactions are now part of the container class (as they differ between podman and singularity). This also includes commits from #367 so the PR looks larger than it really is. This needs to be tested on a cluster, and I'm also not sure about podman being available in GitHub actions but I think it might be ? actions/runner-images#320

[marcodelapierre]

All right, some tests with add/podman (latest commit as of 2 June 9PM GMT+8).
Using a Ubuntu 18.04 VM and Podman 1.6.2 (super super old ..).

First comment:

$ shpc install biocontainers/samtools:v1.9-4-deb_cv1
Trying to pull docker.io/biocontainers/samtools:v1.9-4-deb_cv1...
Getting image source signatures
Copying blob f464fdba1cc8 done
Copying blob 732b6ad56c57 done
Copying blob 1badce2e48be done
Copying blob 478cd0aa93c0 done
Copying blob e8e87313e9cb done
Copying blob 94d6a239eb0e done
Copying blob a955c5be8430 done
Copying config f210eb625b done
Writing manifest to image destination
Storing signatures
f210eb625ba612c85b7e721490130b2afead736d604032b7597e47758c3c6c4d
Module biocontainers-samtools:v1.9-4-deb_cv1 was created.

Similar to the behaviour for Singularity, shall the details of the pull output be hidden by default? (is there a verbose/debug option for install?)

[marcodelapierre]

Also, the directory structure for the containers is not required by podman:

$ ls -d containers/biocontainers/samtools/v1.9-4-deb_cv1/
containers/biocontainers/samtools/v1.9-4-deb_cv1/

So, it should be probably ignored by install/uninstall and other commands? (then use only podman images to ping availability -- haven't looked at how it's done right now )

[marcodelapierre]

This is a bit awful, but it's not a problem due to SHPC:

Creating host files with e.g. a ubuntu container works:

$ podman run --rm -it -v /data docker.io/ubuntu:18.04 bash
root@415152566d2d:/# touch /data/zio3

Some containers define a different user (I guess this is the problem), and then file creation does not work:

podman run -u $(id -u):$(id -g) --rm -it --env-file /home/ubuntu/singularity-hpc/modules/biocontainers/samtools/v1.9-4-deb_cv1/99-shpc.sh -v /data docker.io/biocontainers/samtools:v1.9-4-deb_cv1 bash
biodocker@b49e91c572cd:/data$ touch /data/zio3
touch: cannot touch '/data/zio3': Permission denied

Even if I force set the same user as host with -u

[marcodelapierre]

Then, one difference around default work directory in the container:

• Singularity uses the host PWD
• Podman does not, so to make the behaviour consistent one would need to ensure that PWD is bind mounted, and made the active directory with -w, so something like: -v $(pwd) -w $(pwd)

I am wondering whether such syntax should be added to all aliases.

Otherwise this won't work (here I have edited the modulefile manually):

ubuntu@u18:~/singularity-hpc$ cat py.py 
print('ciao')
$ type python
python is a function
python () 
{ 
    podman ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -it --rm --env-file /home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/99-shpc.sh -v /data -v $(pwd) -w $(pwd) --entrypoint /usr/local/bin/python docker.io/library/python:3.9.2-slim $@
}
ubuntu@u18:~/singularity-hpc$ python py.py 
ciao

[marcodelapierre]

Having issues executing simple commands such as echo Hello:
(here using python:3.9.2-slim)

$ python-exec echo ciao
Error: unable to pull ciao: image name provided is a short name and no search registries are defined in the registries config file.

$ python-exec echo Hello World
Error: unable to pull Hello: error getting default registries to try: repository name must be lowercase

I had a look at the modulefile, there were 3 missing spaces in the definition of -exec; the right one looks like:

set_shell_function("python-exec", execCmd .. " --entrypoint /bin/bash " .. containerPath .. "  $@",  execCmd .. " --entrypoint /bin/bash " .. containerPath .. " $*")

Even with this edit, still having issues:

$ python-exec echo ciao
/bin/echo: /bin/echo: cannot execute binary file
ERRO[0000] Error removing container d763f87f57b766e555b52a13c635e65c9808978a676ff9f9fdef0db8cd08990c: container d763f87f57b766e555b52a13c635e65c9808978a676ff9f9fdef0db8cd08990c does not exist in database: no such container 

ubuntu@u18:~/singularity-hpc$ python-exec echo Hello
/bin/echo: /bin/echo: cannot execute binary file

I think this is related to the use of entrypoint, but don't have more insights right now

[marcodelapierre]

Last one for this set of tests, seems like when using podman as runtime the version tag gets lost in the alias names:
(same branch/commit with singularity creates the right alias names)

$ module help python/3.9.2-slim

-------------------------------------------------------------- Module Specific Help for "python/3.9.2-slim/module" ---------------------------------------------------------------
This module is a podman container wrapper for python v3.9.2-slim
An interpreted, high-level and general-purpose programming language.

Container:

 - docker.io/library/python:3.9.2-slim

Commands include:

 - python-run:
       podman run --rm -it --env-file /home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/99-shpc.sh -v /data <container>
 - python-shell:
       podman run --rm -it --env-file /home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/99-shpc.sh -v /data <container>
 - python-exec:
       podman run -it --rm --entrypoint /bin/bash --env-file /home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/99-shpc.sh -v /data <container> "$@"
 - python-inspect:
       podman inspect <container>

 - python:
       podman run --rm -it --entrypoint /usr/local/bin/python --env-file /home/ubuntu/singularity-hpc/modules/python/3.9.2-slim/99-shpc.sh -v /data <container> ""


For each of the above, you can export:

 - PODMAN_OPTS: to define custom options for podman
 - PODMAN_COMMAND_OPTS: to define custom options for the command

[vsoch]

Responding to comments!

Similar to the behaviour for Singularity, shall the details of the pull output be hidden by default? (is there a verbose/debug option for install?)

The reason singularity is hidden by the nice command is that I use the spython client under the hood - and this will handle showing any issue if it arises. Podman seems to have podman-py, but it didn't work for me to pull, and it also seems geared toward interaction via a socket to do things like list local manifests. I chose to not add it and do simple subprocess commands instead, and show the output to the user so if something goes wrong it's easy to see. Ideally podman would have a python client that just works out of the box to better handle the UI, and that's something I could contribute if the need arises and there is heavy use here. But for now I think it's probably okay just to show the pull.

So, it should be probably ignored by install/uninstall and other commands? (then use only podman images to ping availability -- haven't looked at how it's done right now )

This is a good point! I think (since we derive installed from modules) we can just clean up this directory after an install if it's not in use.

I'll need to take a look at the python container again - I also had trouble with those commands (I think I got a few working but it looks like not all!)

For the alias names I was debating this - should they really have the version? E.g., why would someone load two python executables of different versions? I was thinking of removing that from the Singularity file so the aliases would look more like normal commands. What do you think?

[vsoch]

Ah I see, the issue is that /bin/echo works best as the entrypoint:

$ podman run -it --rm --entrypoint /bin/echo docker.io/library/python:3.9.5-alpine hello      
hello

It looks like this works running on the host:

podman run -it --rm --entrypoint /bin/sh docker.io/library/python:3.9.5-alpine -c “echo hello”     
hello

But when I add that logic to the module file (the update here) it prints an empty line. Do you know why maybe the args aren't being passed in the modulefile?

[marcodelapierre]

Oh, noticed a typo that needs fixing in the modulefile module help bit, as I was playing with this:
it shows the exec command be like: singularity exec -s /bin/sh, but in reality there's no -s option, and in fact this wouldn't work with exec

[vsoch]

okay! It possibly can't be perfect (there are always bugs) but I am happy enough with these changes (and they are a great improvement on the current main) to call this PR finished and merge and release. After this I can start working on Shifter, Docker, and other container technologies.

[marcodelapierre]

This is so exciting!! :-)