LILO with SLURM

[MarieLataretu]

Hi there,

I'm trying to run LILO on a SLURM HPC and I'm not sure what the errors are related to.
Do you have an idea? It seems really environment depended, but maybe you stumbled across something similar.

Call:

snakemake -k -s [...]/tools/Lilo/LILO --configfile $CONFIG --profile [...]/tools/config-snippets/snake-cookies/slurm

Log:

[...]
MissingOutputException in line 84 of [...]/tools/Lilo/LILO:
Job Missing files after 30 seconds:
FAR95540_pass_unclassified_7f618209_73/split/amplicon51.fastq
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Job id: 133673 completed successfully, but some output files are missing. 133673
Trying to restart job 133673.
[...]
Error in rule assign:
    jobid: 133673
    output: FAR95540_pass_unclassified_7f618209_73/split/amplicon51.fastq
    shell:
        bedtools intersect -F 0.9 -wa -wb -bed -abam FAR95540_pass_unclassified_7f618209_73/alignments/reads_to_ref.bam -b amplicons.bed  | grep amplicon51 - | awk '{print $4}' - | seqtk subseq porechop/FAR95540_pass_unclassified_7f618209_73.fastq.gz - > FAR95540_pass_unclassified_7f618209_73/split/amplicon51.fastq
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: 210115

Error executing rule assign on cluster (jobid: 133673, external: 210115, jobscript: [...]/.snakemake/tmp.cssfeg5e/snakejob.assign.133673.sh). For error details see the cluster log and the log files of the involved rule(s).
[...]
Traceback (most recent call last):
  File "/scratch/lataretum/miniconda3/envs/LILO/lib/python3.8/site-packages/snakemake/__init__.py", line 701, in snakemake
    success = workflow.execute(
  File "/scratch/lataretum/miniconda3/envs/LILO/lib/python3.8/site-packages/snakemake/workflow.py", line 1077, in execute
    success = self.scheduler.schedule()
  File "/scratch/lataretum/miniconda3/envs/LILO/lib/python3.8/site-packages/snakemake/scheduler.py", line 441, in schedule
    self._error_jobs()
  File "/scratch/lataretum/miniconda3/envs/LILO/lib/python3.8/site-packages/snakemake/scheduler.py", line 557, in _error_jobs
    self._handle_error(job)
  File "/scratch/lataretum/miniconda3/envs/LILO/lib/python3.8/site-packages/snakemake/scheduler.py", line 615, in _handle_error
    self.running.remove(job)
KeyError: assign

I set --latency-wait 90 it again breaks after some time at a assign rule and a KeyError: read_select from the snakemake scheduler.

Let me know which input/config files might be interesting to solve this. :)

[amandawarr]

Hi Marie, would you be able to check how much coverage amplicon 51 has? If it has coverage below the threshold that would suggest that the pipeline is failing because the output file isn't being produced for that amplicon, however with "-k" snakemake should just carry on regardless. It would be useful to know if that is the source of the problem.

Are there any files that have been output in the split/ folder when it fails?

[MarieLataretu]

Hi :)

yes, in the split folder are several ampliconXX.fastq files, but the one snakemake looks for is indeed missing.

I noticed, that it's always about unclassified reads - I guess it's smarter to input only classified (barcoded) reads 😅 I'll keep you updated!

[amandawarr]

Could you have a try with the attached version of Lilo and let me know if it gets past that point?

LILO_issue_1.txt

It shouldn't be necessary, I have it running on very gappy SARS-CoV-2s right now, but in this version if there are no reads for the amplicon it creates an empty fastq file.

[MarieLataretu]

Yep, it was not the unclassified reads. I'll try the new version :)

Maybe you can also check on my primer input (ARTIC V4.1)
nCoV-2019.V4.1.primer.csv
nCoV-2019.V4.1.noalt.scheme.bed.txt

[amandawarr]

I have just tested your input files on my data and they work perfectly, so it isn't that.

Are you using the version of snakemake from the conda environment? The only reason I can think it wouldn't be working is snakemake is for some reason ignoring the -k telling it not to worry about low coverage amplicons that don't produce output. Is there anything in your --profile configuration for slurm that would overwrite that?

[MarieLataretu]

I updated snakemake at some point to 6.15.5 to see if it makes a difference.

In my config is keep-going true and it's also explicitly called with -k 🤔

[amandawarr]

Can you check what happens when you run the command that it says failed?

i.e.:

bedtools intersect -F 0.9 -wa -wb -bed -abam FAR95540_pass_unclassified_7f618209_73/alignments/reads_to_ref.bam -b amplicons.bed | grep amplicon51 - | awk '{print $4}' - | seqtk subseq porechop/FAR95540_pass_unclassified_7f618209_73.fastq.gz - > FAR95540_pass_unclassified_7f618209_73/split/amplicon51.fastq