WARNING: underlay of /usr/share/zoneinfo/Etc/UTC required more than 50 (57) bind mounts

[paulmenzel]

Running SIMPLI’s test profile fails with the error below in our environment:

$ javac -version
javac 17.0.2-internal
$ singularity version
3.8.3+231-g9dceb4240
$ git log --oneline --no-decorate -1
7236f830a Fix Orphaned processes when using Singularity
$ ./launch.sh -v
nextflow version 23.02.1-edge.5846
$ ./launch.sh run https://github.com/ciccalab/SIMPLI -profile test
N E X T F L O W  ~  version 23.02.1-edge
Launching `https://github.com/ciccalab/SIMPLI` [high_mcnulty] DSL2 - revision: 09e3bd432a [master]
executor >  local (2)
[37/ac837e] process > convert_raw_data:convert_raw_data_to_tiffs (2)                           [  0%] 0 of 2
[-        ] process > convert_raw_data:collect_raw_tiff_metadata                               -
[-        ] process > normalize_images:normalize_tiffs                                         -
[-        ] process > normalize_images:collect_normalized_tiff_metadata                        -
[-        ] process > preprocess_images:image_preprocessing                                    -
[-        ] process > preprocess_images:process_preprocessed_metadata                          -
[-        ] process > measure_areas:measure_positive_areas                                     -
[-        ] process > cp_segment_cells:cp_cell_segmentation                                    -
[-        ] process > cp_segment_cells:collect_single_cell_data                                -
[-        ] process > sd_segment_cells:sd_cell_segmentation                                    -
[-        ] process > sd_segment_cells:collect_single_cell_data                                -
[-        ] process > identify_cell_types_mask:cell_type_identification_mask                   -
[-        ] process > threshold_expression:threshold_cells                                     -
[-        ] process > cluster_cells:cell_clustering                                            -
[-        ] process > cluster_cells:collect_clustering_data                                    -
[-        ] process > analyse_homotypic_interactions:homotypic_interaction_analysis            -
[-        ] process > analyse_homotypic_interactions:collect_homotypic_interactions            -
[-        ] process > calculate_heterotypic_distances:get_heterotypic_distances                -
[-        ] process > calculate_heterotypic_distances:collect_heterotypic_distances            -
[-        ] process > permute_heterotypic_interactions:permute_heterotypic_distances           -
[-        ] process > visualize_areas:area_visualization                                       -
[-        ] process > visualize_cell_types:cell_type_visualization                             -
[-        ] process > visualize_cell_clusters:cell_cluster_visualization                       -
[-        ] process > visualize_cell_thresholds:cell_threshold_visualization                   -
[-        ] process > visualize_homotypic_interactions:homotypic_interaction_visualization     -
[-        ] process > visualize_heterotypic_interactions:heterotypic_interaction_visualization -
[-        ] process > visualize_permuted_interactions:permuted_interaction_visualization       -


Error executing process > 'convert_raw_data:convert_raw_data_to_tiffs (2)'

Caused by:
  Process `convert_raw_data:convert_raw_data_to_tiffs (2)` terminated with an error exit status (1)

Command executed:

  python3.8 /opt/Tiff_extracter.py \
      CRC_2 \
      'ROI004_R-A02_x13400_800-y12700_1250-p0014-200Hz-p07dB-s0137' \
executor >  local (2)
[cc/e822d9] process > convert_raw_data:convert_raw_data_to_tiffs (1)                           [100%] 1 of 1, failed: 1
[-        ] process > convert_raw_data:collect_raw_tiff_metadata                               -
[-        ] process > normalize_images:normalize_tiffs                                         -
[-        ] process > normalize_images:collect_normalized_tiff_metadata                        -
[-        ] process > preprocess_images:image_preprocessing                                    -
[-        ] process > preprocess_images:process_preprocessed_metadata                          -
[-        ] process > measure_areas:measure_positive_areas                                     -
[-        ] process > cp_segment_cells:cp_cell_segmentation                                    -
[-        ] process > cp_segment_cells:collect_single_cell_data                                -
[-        ] process > sd_segment_cells:sd_cell_segmentation                                    -
[-        ] process > sd_segment_cells:collect_single_cell_data                                -
[-        ] process > identify_cell_types_mask:cell_type_identification_mask                   -
[-        ] process > threshold_expression:threshold_cells                                     -
[-        ] process > cluster_cells:cell_clustering                                            -
[-        ] process > cluster_cells:collect_clustering_data                                    -
[-        ] process > analyse_homotypic_interactions:homotypic_interaction_analysis            -
[-        ] process > analyse_homotypic_interactions:collect_homotypic_interactions            -
[-        ] process > calculate_heterotypic_distances:get_heterotypic_distances                -
[-        ] process > calculate_heterotypic_distances:collect_heterotypic_distances            -
[-        ] process > permute_heterotypic_interactions:permute_heterotypic_distances           -
[-        ] process > visualize_areas:area_visualization                                       -
[-        ] process > visualize_cell_types:cell_type_visualization                             -
[-        ] process > visualize_cell_clusters:cell_cluster_visualization                       -
[-        ] process > visualize_cell_thresholds:cell_threshold_visualization                   -
[-        ] process > visualize_homotypic_interactions:homotypic_interaction_visualization     -
[-        ] process > visualize_heterotypic_interactions:heterotypic_interaction_visualization -
[-        ] process > visualize_permuted_interactions:permuted_interaction_visualization       -
Error executing process > 'convert_raw_data:convert_raw_data_to_tiffs (2)'

Caused by:
  Process `convert_raw_data:convert_raw_data_to_tiffs (2)` terminated with an error exit status (1)

Command executed:

  python3.8 /opt/Tiff_extracter.py \
      CRC_2 \
      'ROI004_R-A02_x13400_800-y12700_1250-p0014-200Hz-p07dB-s0137' \
      crc_2.txt \
      single \
      ./ \
      channel_metadata.csv \
      CRC_2-raw_tiff_metadata.csv > extract_log.txt 2>&1

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: underlay of /usr/share/zoneinfo/Etc/UTC required more than 50 (57) bind mounts
  INFO:    Cleaning up image...

Work dir:
  /dev/shm/nextflow/work/37/ac837e73cad46da68cc57e2a64ec93

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

No idea, if the Singularity warning is a red herring, and something else causes this.

Is there a way to run that single Singularity “step” directly?

[MicheleBortol]

Hi @paulmenzel,

Sorry for the late reply, but I switched jobs, and I am longer working on mantaining/developing SIMPLI.
I think the singularity warning could be a good indication of what's going on, but we would need more info.
What are the contents of /dev/shm/nextflow/work/37/ac837e73cad46da68cc57e2a64ec93 ?
There should be a log file `` with a more descriptive error message. I expect a "File not found" type of error .
The folder should also contain a symlink to a .mcd file with the raw IMC data. If the symlink is broken, then the problem is with the path in the raw IMC metadata file. This is quite unlikely since you are running the example workflow. If the symlink is OK, then it means that singularity can't bind the folder, which could point back to the warning you see.
In the same folder you can also find the `.command.sh` and `.command.run` files which contain the exact commands ran by nextflow in this step.
Hope this helps, if you could post the contents of the `/dev/shm/nextflow/work/37/ac837e73cad46da68cc57e2a64ec93 ` folder and the log file, then I could have a better look into this.
Thank you for bringing up this issue.
Best wishes

Michele

[MicheleBortol]

Solved by switching from singularity to apptainer according to this discussion in the nextflow repository.
For the switch to apptainer from singularity see this issue: #8 (comment).