Pipeline fails creating container with strange bind argument

[paulmenzel]

Dear Nextflow folks,

A user is trying to run the Nextflow based pipeline SIMPLI, but it fails despite it used to work several months ago. We updated glibc from 2.33 to 2.36 in the meantime.

$ singularity --version
singularity version 3.8.3+231-g9dceb4240
$ prun openjdk-17.0.2.8-0 javac -version
javac 17.0.2-internal
$ ./launch.sh -v
nextflow version 23.02.1-edge.5846
$ /lib64/ld-linux-x86-64.so.2 --version
ld.so (GNU libc) stable release version 2.36.
[…]
$ ls -l /lib64/ld-linux-x86-64.so.2
-rwxr-xr-x 1 root root 1581104 Sep  6 12:37 /lib64/ld-linux-x86-64.so.2

Now executing the test pipeline

./launch.sh run https://github.com/ciccalab/SIMPLI -profile test

it fails with the error below:

[…]
Error executing process > 'convert_raw_data:convert_raw_data_to_tiffs (1)'

Caused by:
  Process `convert_raw_data:convert_raw_data_to_tiffs (1)` terminated with an error exit status (255)

Command executed:

  python3.8 /opt/Tiff_extracter.py \
      CRC_1 \
      'ROI007_R-B02_x40560_850-y12265_1200-p0014-200Hz-p07dB-s0136' \
      crc_1.txt \
      single \
      ./ \
      channel_metadata.csv \
      CRC_1-raw_tiff_metadata.csv > extract_log.txt 2>&1

Command exit status:
  255

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  FATAL:   while extracting /dev/shm/nextflow/singularity/containers/simpli_imaging-default-simpli_imctools_stardist-test.img: root filesystem extraction failed: extract command failed: WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
  WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
  WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
  WARNING: Skipping mount /dev/shm/pmenzel/local/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
  WARNING: Skipping mount /dev/shm/pmenzel/local/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
  WARNING: Skipping mount /dev/shm/pmenzel/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  FATAL:   container creation failed: mount /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2->/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 error: while mounting /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2: mount source /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 doesn't exist
  : exit status 255

Work dir:
  /dev/shm/nextflow/work/cd/47549d50e9d2b8963296d34d6e9dfe

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

The second fatal log line looks suspicious:

FATAL:   container creation failed: mount /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2->/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 error: while mounting /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2: mount source /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

I didn’t find a log file, where the executed Singularity command is logged, so with strace -e execve -s4096 I found:

/scratch/local2/nf/nf.50261:execve("/dev/shm/pmenzel/local/bin/singularity", ["/dev/shm/pmenzel/local/bin/singularity", "-q", "exec", "--no-home", "--no-nv", "--no-rocm", "-C", "--no-init", "--writable", "-B", "/tmp/rootfs-1859789068:/image", "-B", "/usr/bin/unsquashfs:/usr/bin/unsquashfs:ro", "-B", "/lib/libpthread.so.0:/lib/libpthread.so.0:ro", "-B", "/lib/libm.so.6:/lib/libm.so.6:ro", "-B", "/lib/libz.so.1:/lib/libz.so.1:ro", "-B", "/usr/lib/liblzma.so.5:/usr/lib/liblzma.so.5:ro", "-B", "/usr/lib/liblzo2.so.2:/usr/lib/liblzo2.so.2:ro", "-B", "/usr/lib/liblz4.so.1:/usr/lib/liblz4.so.1:ro", "-B", "/lib/libc.so.6:/lib/libc.so.6:ro", "-B", "/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2:ro", "/tmp/rootfs-1859789068/tmp-rootfs-3269417444", "/usr/bin/unsquashfs", "-user-xattrs", "-r", "-d", "/image/root", "/image/archive-1745146396", "^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\\x2f]).*$"], 0xc0005bbfd0 /* 1 var */) = 0

The => in the bind/mount line looks suspicious. Is that expected?

Kind regards,

Paul

[pditommaso]

Please provide descriptive title

[pditommaso]

Not sure how much we can help. This an problem with the singularity execution

container creation failed: mount /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2->/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 error: while mounting /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2: mount source /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 doesn't exist

[paulmenzel]

Yes, that is the line I singled out too.

Nextflow created that Singularity command though (“the -B arguments”) though, didn’t it? How do the => get in there, and how can I find out, what command was actually executed? Is there a verbose mode?

[paulmenzel]

Any hints on how to debug Nextflow?

[pditommaso]

look into the .command.run script created by the nextflow. You can debug running bash .command.run as any Bash script

[paulmenzel]

Thank you. I always overlooked those, as I did not pass -a to ls. I also added set -x at the top of the script, and indeed the Singularity command below fails.

$ SINGULARITYENV_TMPDIR=/scratch/local singularity exec -B /scratch/local/nextflow/work -B /home/pmenzel/.nextflow/assets/ciccalab/SIMPLI/test/metadata --bind /home/pmenzel/.nextflow/assets/ciccalab/SIMPLI/scripts:/opt,/scratch/local/nextflow/:/data /scratch/local/nextflow/singularity/containers/simpli_imaging-default-simpli_imctools_stardist-test.img /bin/bash -ue /scratch/local/nextflow/work/a6/f755fea36cf58fd221fa59ff835ac4/.command.sh
INFO:    Converting SIF file to temporary sandbox...
FATAL:   while extracting /scratch/local/nextflow/singularity/containers/simpli_imaging-default-simpli_imctools_stardist-test.img: root filesystem extraction failed: extract command failed: WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
WARNING: Skipping mount /scratch/local/pmenzel/local/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
WARNING: Skipping mount /scratch/local/pmenzel/local/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
WARNING: Skipping mount /scratch/local/pmenzel/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container                                          FATAL:   container creation failed: mount /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2->/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 error: while mounting /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2: mount source /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 doesn't exist
: exit status 255

Moving to the Singularity successor Apptainer fixed it for me.

[paulmenzel]

Moving to the Singularity successor Apptainer fixed it for me.