singularity build --fakeroot fails with free(): invalid pointer while spawning RPC server

[haferman]

I only experience this problem when building from the tarball. The RPM works fine. I also randomly decided to build 3.8.4 and get the same error. I suspect something in my build environment, but I can't pin it down.

Version of Singularity

singularity-ce version 4.1.2

Describe the bug
ANY singularity build --fakeroot fails with:

free(): invalid pointer

but only from my local build. The RPM install works as expected. If I remove --fakeroot things work as expected. I have done singularity config fakeroot --add <user>.

To Reproduce

singularity build --fakeroot lolcow.sif docker://sylabsio/lolcow

Expected behavior
I expect the SIF to build.

OS / Linux Distribution
Which Linux distribution are you using?

Red Hat Enterprise Linux release 8.9 (Ootpa)

Installation Method
source, via:

./mconfig --prefix=/usr/local/singularity-ce-4.1.2

Note: using Go 1.22.2
No errors or warnings during build

Additional context

singularity -ddd build --fakeroot lolcow.sif docker://sylabsio/lolcow

relevant output:

DEBUG   [U=0,P=2]          apply_privileges()            Set user ID to 0
DEBUG   [U=0,P=2]          set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=2]          init()                        Spawn RPC server
free(): invalid pointer
VERBOSE [U=0,P=1]          wait_child()                  rpc server interrupted by signal number 6
DEBUG   [U=502,P=816682]   startup()                     fakeroot runtime engine selected
VERBOSE [U=502,P=816682]   startup()                     Execute master process
DEBUG   [U=502,P=816682]   func1()                       Waiting for container process 816697
DEBUG   [U=502,P=816682]   func1()                       Wait for process 816697 complete with status 34304, error <nil>
DEBUG   [U=502,P=816682]   Master()                      Child exited with exit status 134

[dtrudg]

This is potentially related to a known issue with an interaction between Go 1.22, older glibc versions, and the approach that Singularity and other container runtimes use.

The RPM packages are built with Go 1.21 for this reason, and there is a note on the 4.1.3 release...

Note - compilation with Go 1.22 currently causes an issue when using the PID namespace on distributions using older versions of glibc. We recommend using Go 1.21 at this time.

If possible, could you provide any output in dmesg that is associated with the error?

[haferman]

Nothing at all in dmesg associated with this error. But, I will rebuild with Go 1.21 and report back, thanks for the tip.

[haferman]

Go 1.21 build works as expected! Thank you.