Container hash mismatch at `NFCORE_CHIPSEQ:CHIPSEQ:PHANTOMPEAKQUALTOOLS (1)` step

[alextruong]

I'm running this pipeline in an HPC environment, but I'm getting an error where the container effectively can't be found.

The error appears to be looking for

community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-4a-4a1cddfad5b503ee347cc5de17d172e1876c547fca00aa844559c9e764fb400f-data.img

but the container that is present is

community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-45-4569ff9993578b8402d00230ab9dd75ce6e63529731eb24f21579845e6bd5cdb-data.img

This results in the step (and the whole workflow) failing. Any ideas?

In our HPC environment, the nf-core containers are located centrally, but manually downloading the containers locally via nf-core download pulls down an exactly matching container list, so I'm unable to find a container with the hash that matches the error. I tracked the reference down to

chipseq/modules/nf-core/phantompeakqualtools/main.nf

Line 8 in 76e2382

'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/4a/4a1cddfad5b503ee347cc5de17d172e1876c547fca00aa844559c9e764fb400f/data' :

, but there's nothing in the file's history that matches the hash I've downloaded.

$ nextflow run nf-core/chipseq -r 2.1.0 -profile singularity --input samples_GSM325933.csv --outdir results_GSM325933 --fasta chm13v2.0.fa --gtf chm13v2.0_RefSeq_Liftoff_v5.2.gtf --igenomes_ignore --genome null --aligner bowtie2 --macs_gsize 3117292070 --keep_multi_map --save_unaligned --narrow_peak --skip_preseq

Nextflow 24.10.3 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.4

Launching `https://github.com/nf-core/chipseq` [cheeky_kirch] DSL2 - revision: 76e2382b6d [2.1.0]

WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/chipseq v2.1.0-g76e2382
------------------------------------------------------
Core Nextflow options
  revision       : 2.1.0
  runName        : cheeky_kirch
  containerEngine: singularity
  launchDir      : /n/scratch/users/a/at237/kornelia
  workDir        : /n/scratch/users/a/at237/kornelia/work
  projectDir     : /home/at237/.nextflow/assets/nf-core/chipseq
  userName       : at237
  profile        : singularity
  configFiles    :

Input/output options
  input          : samples_GSM325933.csv
  outdir         : results_GSM325933

Reference genome options
  genome         : null
  fasta          : chm13v2.0.fa
  gtf            : chm13v2.0_RefSeq_Liftoff_v5.2.gtf
  macs_gsize     : 3117292070
  igenomes_ignore: true

Alignment options
  aligner        : bowtie2
  keep_multi_map : true
  save_unaligned : true

Peak calling options
  narrow_peak    : true

Process skipping options
  skip_preseq    : true

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/chipseq for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.3240506

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/chipseq/blob/master/CITATIONS.md
------------------------------------------------------
executor >  local (71)
[f1/2b2bdc] NFCORE_CHIPSEQ:PREPARE_GENOME:GTF2BED (chm13v2.0_RefSeq_Liftoff_v5.2.gtf)                            [100%] 1 of 1 ✔
[c4/f3b6f5] NFCORE_CHIPSEQ:PREPARE_GENOME:CUSTOM_GETCHROMSIZES (chm13v2.0.fa)                                    [100%] 1 of 1 ✔
[7f/492eff] NFCORE_CHIPSEQ:PREPARE_GENOME:GENOME_BLACKLIST_REGIONS (chm13v2.0.fa.sizes)                          [100%] 1 of 1 ✔
[1a/5d4f5c] NFCORE_CHIPSEQ:PREPARE_GENOME:BOWTIE2_BUILD (chm13v2.0.fa)                                           [100%] 1 of 1 ✔
[53/7c2254] NFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samples_GSM325933.csv)                         [100%] 1 of 1 ✔
[e9/b65e15] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:FASTQC (Input_REP5_T1)                       [100%] 6 of 6 ✔
[e3/a3b565] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (Input_REP4_T1)                   [100%] 6 of 6 ✔
[ad/3e37bc] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_ALIGN_BOWTIE2:BOWTIE2_ALIGN (Input_REP4_T1)                             [100%] 6 of 6 ✔
executor >  local (71)
[f1/2b2bdc] NFCORE_CHIPSEQ:PREPARE_GENOME:GTF2BED (chm13v2.0_RefSeq_Liftoff_v5.2.gtf)                            [100%] 1 of 1 ✔
[c4/f3b6f5] NFCORE_CHIPSEQ:PREPARE_GENOME:CUSTOM_GETCHROMSIZES (chm13v2.0.fa)                                    [100%] 1 of 1 ✔
[7f/492eff] NFCORE_CHIPSEQ:PREPARE_GENOME:GENOME_BLACKLIST_REGIONS (chm13v2.0.fa.sizes)                          [100%] 1 of 1 ✔
[1a/5d4f5c] NFCORE_CHIPSEQ:PREPARE_GENOME:BOWTIE2_BUILD (chm13v2.0.fa)                                           [100%] 1 of 1 ✔
[53/7c2254] NFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samples_GSM325933.csv)                         [100%] 1 of 1 ✔
[e9/b65e15] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:FASTQC (Input_REP5_T1)                       [100%] 6 of 6 ✔
[e3/a3b565] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (Input_REP4_T1)                   [100%] 6 of 6 ✔
[ad/3e37bc] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_ALIGN_BOWTIE2:BOWTIE2_ALIGN (Input_REP4_T1)                             [100%] 6 of 6 ✔
[30/739122] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_ALIGN_BOWTIE2:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (Input_REP4_T1)     [100%] 6 of 6 ✔
[d9/2509dc] NFCORE_CHIPSEQ:CHIPSEQ:FASTQ_ALIGN_BOWTIE2:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_INDEX (Input_REP4_T1)    [100%] 6 of 6 ✔
[ad/f4ce45] NFC…STQ_ALIGN_BOWTIE2:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_STATS (PolII_K562_REP1_T1) [100%] 6 of 6 ✔
[89/c5273e] NFC…_ALIGN_BOWTIE2:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_FLAGSTAT (PolII_K562_REP1_T1) [100%] 6 of 6 ✔
[8f/edbae5] NFC…_ALIGN_BOWTIE2:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS (PolII_K562_REP1_T1) [100%] 6 of 6 ✔
[98/fecf4a] NFCORE_CHIPSEQ:CHIPSEQ:PICARD_MERGESAMFILES (Input_REP4)                                             [100%] 6 of 6 ✔
[ef/fe53e0] NFCORE_CHIPSEQ:CHIPSEQ:BAM_MARKDUPLICATES_PICARD:PICARD_MARKDUPLICATES (Input_REP2)                  [ 66%] 4 of 6
[20/dfc110] NFCORE_CHIPSEQ:CHIPSEQ:BAM_MARKDUPLICATES_PICARD:SAMTOOLS_INDEX (Input_REP2)                         [100%] 4 of 4
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_MARKDUPLICATES_PICARD:BAM_STATS_SAMTOOLS:SAMTOOLS_STATS                   -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_MARKDUPLICATES_PICARD:BAM_STATS_SAMTOOLS:SAMTOOLS_FLAGSTAT                -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_MARKDUPLICATES_PICARD:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS                -
[9d/9e801f] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAMTOOLS_FILTER (Input_REP5)                              [ 25%] 1 of 4
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:SAMTOOLS_INDEX                                            [  0%] 0 of 1
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_STATS                         -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_FLAGSTAT                      -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS                      -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:SAMTOOLS_SORT                                             -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_REMOVE_ORPHANS                                        -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT                     -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_INDEX                    -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_STATS -
[-        ] NFC…CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_FLAGSTAT -
[-        ] NFC…CHIPSEQ:CHIPSEQ:BAM_FILTER_BAMTOOLS:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:PICARD_COLLECTMULTIPLEMETRICS                                                 [  0%] 0 of 1
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:PHANTOMPEAKQUALTOOLS                                                          -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:MULTIQC_CUSTOM_PHANTOMPEAKQUALTOOLS                                           -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_BEDGRAPH_BIGWIG_BEDTOOLS_UCSC:BEDTOOLS_GENOMECOV                          -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:BAM_BEDGRAPH_BIGWIG_BEDTOOLS_UCSC:UCSC_BEDGRAPHTOBIGWIG                       -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:DEEPTOOLS_COMPUTEMATRIX                                                       -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:DEEPTOOLS_PLOTPROFILE                                                         -
[-        ] NFCORE_CHIPSEQ:CHIPSEQ:DEEPTOOLS_PLOTHEATMAP                                                         -
Plus 14 more processes waiting for tasks…
ERROR ~ Error executing process > 'NFCORE_CHIPSEQ:CHIPSEQ:PHANTOMPEAKQUALTOOLS (1)'

Caused by:
  java.io.FileNotFoundException: /n/app/singularity/containers/nf-core/chipseq/2.1.0/.community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-4a-4a1cddfad5b503ee347cc5de17d172e1876c547fca00aa844559c9e764fb400f-data.img.lock (Permission denied)



 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-[nf-core/chipseq] Pipeline completed with errors-
WARN: Killing running tasks (3)

[alextruong]

Just wondering if anyone got a chance to look at this? Is there any way to download the desired container version here? I'm not sure how to proceed, and the pipeline execution is completely halted while this issue is present.

[alextruong]

I revisited this a bit more thoroughly, and actually found a reference to the container hash here:

chipseq/modules/local/multiqc_custom_phantompeakqualtools.nf

Line 5 in 76e2382

'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/45/4569ff9993578b8402d00230ab9dd75ce6e63529731eb24f21579845e6bd5cdb/data':

Perhaps this wasn't updated for version 2.1.0?

I found the commit for the 400f hash:

b2f68c7

Is this a case of having missed updating a reference in the config? I'll update the reference here to see if I can get the pipeline to run on my end.