2023-01-03 main branch e2e-failures RHEL 8 AMD64

[dtrudg]

--- FAIL: TestE2E (649.64s)
    --- FAIL: TestE2E/PAR (0.00s)
        --- FAIL: TestE2E/PAR/DOCKER (0.00s)
            --- FAIL: TestE2E/PAR/DOCKER/ordered (248.07s)
                --- FAIL: TestE2E/PAR/DOCKER/ordered/user (14.96s)
                    --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_df_build (1.65s)
                    --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif (0.70s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIUser (0.10s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIFakeroot (0.08s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIRoot (0.06s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIUser (0.09s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIFakeroot (0.08s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIRoot (0.05s)
    --- FAIL: TestE2E/SEQ (265.67s)
        --- FAIL: TestE2E/SEQ/BUILD (43.55s)
            --- FAIL: TestE2E/SEQ/BUILD/dockerfile (40.09s)
                --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root (17.07s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none (1.37s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none/build (1.18s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none/act (0.19s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong (1.35s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong/build (1.17s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong/act (0.19s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right (1.38s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right/build (1.19s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right/act (0.19s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right (1.37s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right/build (1.17s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right/act (0.20s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_none (1.35s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_none/build (1.18s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong (1.36s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong/build (1.17s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right (1.37s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right/build (1.19s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right/act (0.18s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right (1.36s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right/build (1.18s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right/act (0.18s)

[dtrudg]

Possibly a crun (version specific) related incompatibility with buildkit? Will try using runc instead, to verify.

"/usr/local/bin/singularity build --oci /tmp/dockerfile-build-USER-410657212/rootbuilt-docker-user.oci.sif ../test/defs/Docker
file.customuser" output:
        INFO:    Did not find usable system buildkitd daemon. Starting built-in singularity-buildkitd.
        INFO:    singularity-buildkitd: running server on /run/singularity-buildkitd/singularity-buildkitd-157013.sock
        time="2024-01-03T12:56:12Z" level=warning msg="no trace recorder found, skipping"
        INFO:    Terminating singularity-buildkitd (PID 157028)
        FATAL:   while building from dockerfile: failed to solve: process "/bin/sh -c addgroup -g 2000 -S testgroup && adduser -u 2000
 -S testuser -G testgroup" did not complete successfully: /usr/bin/crun did not terminate successfully: exit status 1: cannot open dir
ectory `/run/crun/902q5y5hkaw7b3nbl464k3fse`: No such file or directory
        #1 [internal] load build definition from Dockerfile.customuser
        #1 transferring dockerfile:
        #1 transferring dockerfile: 230B done
        #1 DONE 0.0s
        
        #2 [internal] load metadata for docker.io/library/alpine:latest
        #2 DONE 0.3s
        
        #3 [internal] load .dockerignore
        #3 transferring context: 2B done
        #3 DONE 0.0s
        
        #4 [1/2] FROM docker.io/library/alpine:latest@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
        #4 resolve docker.io/library/alpine:latest@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 done
        #4 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.1s done
        #4 DONE 0.1s
        
        #4 [1/2] FROM docker.io/library/alpine:latest@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
        #4 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0.1s done
        #4 DONE 0.1s
        
        #5 [2/2] RUN addgroup -g 2000 -S testgroup && adduser -u 2000 -S testuser -G testgroup
        #5 ERROR: process "/bin/sh -c addgroup -g 2000 -S testgroup && adduser -u 2000 -S testuser -G testgroup" did not complete succ
essfully: /usr/bin/crun did not terminate successfully: exit status 1: cannot open directory `/run/crun/902q5y5hkaw7b3nbl464k3fse`: No
 such file or directory

[dtrudg]

The failures on RHEL8 AMD64 are all covered by the bug #2484