lxplus apptainer can't switch between registries

[matthewfeickert]

It seems on lxplus9 that apptainer is able to run a container image from Docker Hub at first just fine, but if the Singularity/Apptainer variables are set to be able to acces private CERN GitLab container registries then this causes apptainer to fail when trying to interact with DockerHub images.

Example:

$ ssh lxplus9.cern.ch
[feickert@lxplus916 ~]$ export APPTAINER_CACHEDIR="/tmp/${USER}/singularity"
[feickert@lxplus916 ~]$ export SINGULARITY_CACHEDIR="${APPTAINER_CACHEDIR}"
[feickert@lxplus916 ~]$ mkdir -p "${APPTAINER_CACHEDIR}"
[feickert@lxplus916 ~]$ apptainer exec -C docker://eschanet/docker_pyhf:v0.2 bash
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
INFO:    Creating SIF file...

Apptainer>
Apptainer> pyhf --version
pyhf, version 0.5.3
Apptainer> exit
exit
[feickert@lxplus916 ~]$
[feickert@lxplus916 ~]$ export APPTAINER_DOCKER_USERNAME=#secret
[feickert@lxplus916 ~]$ export APPTAINER_DOCKER_PASSWORD=#secret
[feickert@lxplus916 ~]$ apptainer exec -C docker://eschanet/docker_pyhf:v0.2 bash
FATAL:   Unable to handle docker://eschanet/docker_pyhf:v0.2 uri: failed to get checksum for docker://eschanet/docker_pyhf:v0.2: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password
[feickert@lxplus916 ~]$ unset APPTAINER_DOCKER_USERNAME
[feickert@lxplus916 ~]$ unset APPTAINER_DOCKER_PASSWORD
[feickert@lxplus916 ~]$ apptainer exec -C docker://eschanet/docker_pyhf:v0.2 bash
INFO:    Using cached SIF image
Apptainer> exit
exit
[feickert@lxplus916 ~]$ 

So I'm not sure how to work around this for people that want to test on lxplus but then use images from both private CERN GitLab container registries as well as public Docker Hub.

This will affect https://gitlab.cern.ch/recast-atlas/susy/ana-susy-2019-08.

[matthewfeickert]

We should instead have people switch to logging into apptainer with

$ apptainer remote login --username "${APPTAINER_DOCKER_USERNAME}" --password ${APPTAINER_DOCKER_PASSWORD}" docker://gitlab-registry.cern.ch
$ unset APPTAINER_DOCKER_USERNAME
$ unset APPTAINER_DOCKER_PASSWORD

[matthewfeickert]

Resolved by https://gitlab.cern.ch/recast-atlas/docs/-/merge_requests/63