Skip to content
This repository has been archived by the owner on Jan 31, 2024. It is now read-only.

can't create report binding error #12

Closed
AlcaArctica opened this issue Sep 14, 2023 · 14 comments
Closed

can't create report binding error #12

AlcaArctica opened this issue Sep 14, 2023 · 14 comments
Labels
bug Something isn't working

Comments

@AlcaArctica
Copy link

I am successfully running the test data set, but the pipeline stops at Error in rule run_report.
I am executing the command
culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT/data_test_config.yaml --singularity-args "--bind /lustre/projects/dazzlerAssembly"

This is the error I get

Error in rule run_report:
    jobid: 1
    input: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_time.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/dag.png, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/versions.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/all_stats_5PXoo1.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/QUAST/5PXoo1/icarus.html, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/QUAST_RESULTS/QUAST-version.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_busco.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/BUSCO-version.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/BLOBTOOLS/5PXoo1_CANU_PILON.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/BLOBTOOLS/5PXoo1_FLYE_PILON.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/ASSEMBLYTICS/5PXoo1__CANU__PILON.Assemblytics_structural_variants.summary, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/ASSEMBLYTICS/5PXoo1__FLYE__PILON.Assemblytics_structural_variants.summary, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/MERQURY/5PXoo1.qv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/MERQURY/5PXoo1.qv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/MERQURY/5PXoo1.completeness.stats, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/MERQURY/5PXoo1.completeness.stats, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_flagstats.csv
    output: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/CulebrONT_report.html
    log: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e (check log file(s) for error details)
    cluster_jobid: 48918253
Logfile /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o not found.

Error executing rule run_report on cluster (jobid: 1, external: 48918253, jobscript: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/.snakemake/tmp.03o82kqy/snakejob.run_report.1.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: ../.snakemake/log/2023-09-14T152430.940894.snakemake.log

Unfortunately, there is no hint as to what the problem may be and both /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o and /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e are empty.
There is also no more information found in the /lustre/projects/dazzlerAssembly/test_culebrONT/.snakemake/log/2023-09-14T152430.940894.snakemake.log file.

What could it be? I am guessing there might be a problem with required R packages or similar. But strictly this should not happen, as I am using the singularity container?

Do you have any ideas how I could debug this problem? I would really love to see the report!
@julieaorjuela
Copy link
Contributor

julieaorjuela commented Sep 15, 2023 •
edited
Loading

Hello,
Could you please send us your config.yaml and cluster_config.yaml ?
We are testing install in our cluster and it's working. Have you installed culebrONT in singularity mode?
Keep in touch 👍 !

@AlcaArctica
Copy link
Author

Archive.zip

Thank you for looking into this. Please finde the config.yaml, cluster_config.yaml and log file in the attached zip archive. I am curious what you will find!

@AlcaArctica
Copy link
Author

Here a small update: I have tried installing culebrONT in different conda envs, thinking perhaps the error has something to do with the requirements. But so far not luck. Here is the environment.yml file of one of the environments I have tried out:

name: culebrONT3
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - _r-mutex=1.0.0=anacondar_1
  - _sysroot_linux-64_curr_repodata_hack=3=haa98f57_10
  - atk-1.0=2.36.0=ha1a6a79_0
  - binutils_impl_linux-64=2.38=h2a08ee3_1
  - binutils_linux-64=2.38.0=hc2dff05_0
  - blas=1.0=openblas
  - boost-cpp=1.73.0=h7f8727e_12
  - bwidget=1.9.11=1
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.19.1=h5eee18b_0
  - ca-certificates=2023.08.22=h06a4308_0
  - cairo=1.16.0=hb05425b_5
  - curl=7.88.1=hdbd6064_2
  - expat=2.5.0=h6a678d5_0
  - font-ttf-dejavu-sans-mono=2.37=hd3eb1b0_0
  - font-ttf-inconsolata=2.001=hcb22688_0
  - font-ttf-source-code-pro=2.030=hd3eb1b0_0
  - font-ttf-ubuntu=0.83=h8b1ccd4_0
  - fontconfig=2.14.1=h4c34cd2_2
  - fonts-anaconda=1=h8fa9717_0
  - fonts-conda-ecosystem=1=hd3eb1b0_0
  - freetype=2.12.1=h4a9f257_0
  - fribidi=1.0.10=h7b6447c_0
  - gcc_impl_linux-64=11.2.0=h1234567_1
  - gcc_linux-64=11.2.0=h5c386dc_0
  - gdk-pixbuf=2.42.10=h5eee18b_0
  - gfortran_impl_linux-64=11.2.0=h1234567_1
  - gfortran_linux-64=11.2.0=hc2dff05_0
  - giflib=5.2.1=h5eee18b_3
  - glib=2.69.1=he621ea3_2
  - gobject-introspection=1.72.0=py311hbb6d50b_2
  - graphite2=1.3.14=h295c915_1
  - graphviz=2.50.0=h1b29801_1
  - gtk2=2.24.33=h73c1081_2
  - gts=0.7.6=hb67d8dd_3
  - gxx_impl_linux-64=11.2.0=h1234567_1
  - gxx_linux-64=11.2.0=hc2dff05_0
  - harfbuzz=4.3.0=hf52aaf7_1
  - icu=58.2=he6710b0_3
  - jpeg=9e=h5eee18b_1
  - kernel-headers_linux-64=3.10.0=h57e8cba_10
  - krb5=1.20.1=h143b758_1
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.38=h1181459_1
  - lerc=3.0=h295c915_0
  - libboost=1.73.0=h28710b8_12
  - libcurl=7.88.1=h251f7ec_2
  - libdeflate=1.17=h5eee18b_0
  - libedit=3.1.20221030=h5eee18b_0
  - libev=4.33=h7f8727e_1
  - libffi=3.4.4=h6a678d5_0
  - libgcc-devel_linux-64=11.2.0=h1234567_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgd=2.3.3=h695aa2c_1
  - libgfortran-ng=11.2.0=h00389a5_1
  - libgfortran5=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libiconv=1.16=h7f8727e_2
  - libnghttp2=1.52.0=h2d74bed_1
  - libopenblas=0.3.21=h043d6bf_0
  - libpng=1.6.39=h5eee18b_0
  - librsvg=2.54.4=h36cc946_3
  - libssh2=1.10.0=hdbd6064_2
  - libstdcxx-devel_linux-64=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libtiff=4.5.1=h6a678d5_0
  - libtool=2.4.6=h6a678d5_1009
  - libuuid=1.41.5=h5eee18b_0
  - libwebp=1.3.2=h11a3e52_0
  - libwebp-base=1.3.2=h5eee18b_0
  - libxcb=1.15=h7f8727e_0
  - libxml2=2.10.4=hcbfbd50_0
  - lz4-c=1.9.4=h6a678d5_0
  - make=4.2.1=h1bed415_1
  - ncurses=6.4=h6a678d5_0
  - ninja=1.10.2=h06a4308_5
  - ninja-base=1.10.2=hd09550d_5
  - nspr=4.35=h6a678d5_0
  - nss=3.89.1=h6a678d5_0
  - openjpeg=2.4.0=h3ad879b_0
  - openssl=3.0.10=h7f8727e_2
  - pango=1.50.7=h05da053_0
  - pcre=8.45=h295c915_0
  - pcre2=10.37=he7ceb23_1
  - pip=23.2.1=py311h06a4308_0
  - pixman=0.40.0=h7f8727e_1
  - poppler=22.12.0=h381b16e_0
  - poppler-data=0.4.11=h06a4308_1
  - python=3.11.5=h955ad1f_0
  - r-base=4.2.0=h1ae530e_0
  - readline=8.2=h5eee18b_0
  - setuptools=68.0.0=py311h06a4308_0
  - sqlite=3.41.2=h5eee18b_0
  - sysroot_linux-64=2.17=h57e8cba_10
  - tk=8.6.12=h1ccaba5_0
  - tktable=2.10=h14c3975_0
  - wheel=0.38.4=py311h06a4308_0
  - xz=5.4.2=h5eee18b_0
  - zlib=1.2.13=h5eee18b_0
  - zstd=1.5.5=hc292b87_0
  - pip:
      - appdirs==1.4.4
      - argparse==1.4.0
      - arrow==1.2.3
      - asttokens==2.4.0
      - attrs==23.1.0
      - backcall==0.2.0
      - binaryornot==0.4.4
      - biopython==1.81
      - certifi==2023.7.22
      - cffi==1.15.1
      - chardet==5.2.0
      - charset-normalizer==3.2.0
      - click==8.1.7
      - configargparse==1.7
      - connection-pool==0.0.3
      - contourpy==1.1.1
      - cookiecutter==2.3.0
      - culebront==2.2.0
      - cycler==0.11.0
      - cython==3.0.2
      - datrie==0.8.2
      - decorator==5.1.1
      - docutils==0.17.1
      - dpath==2.1.6
      - executing==1.2.0
      - fastjsonschema==2.18.0
      - fonttools==4.42.1
      - gitdb==4.0.10
      - gitpython==3.1.36
      - humanfriendly==10.0
      - idna==3.4
      - ipython==8.15.0
      - jedi==0.19.0
      - jinja2==3.1.2
      - jsonschema==4.19.0
      - jsonschema-specifications==2023.7.1
      - jupyter-core==5.3.1
      - kiwisolver==1.4.5
      - markdown-it-py==3.0.0
      - markupsafe==2.1.3
      - matplotlib==3.8.0
      - matplotlib-inline==0.1.6
      - mdurl==0.1.2
      - nbformat==5.9.2
      - numpy==1.26.0
      - packaging==23.1
      - pandas==2.1.0
      - parso==0.8.3
      - pexpect==4.8.0
      - pickleshare==0.7.5
      - pillow==10.0.1
      - plac==1.3.5
      - platformdirs==3.10.0
      - prompt-toolkit==3.0.39
      - psutil==5.9.5
      - ptyprocess==0.7.0
      - pulp==2.7.0
      - pure-eval==0.2.2
      - pycparser==2.21
      - pygments==2.16.1
      - pyparsing==3.1.1
      - pysam==0.21.0
      - python-dateutil==2.8.2
      - python-slugify==8.0.1
      - pytz==2023.3.post1
      - pyyaml==6.0.1
      - referencing==0.30.2
      - requests==2.31.0
      - reretry==0.11.8
      - rich==13.5.3
      - rpds-py==0.10.3
      - rpy2==3.5.14
      - six==1.16.0
      - smart-open==6.4.0
      - smmap==5.0.1
      - snakemake==7.32.4
      - stack-data==0.6.2
      - stopit==1.1.2
      - tabulate==0.9.0
      - text-unidecode==1.3
      - throttler==1.2.2
      - toposort==1.10
      - tqdm==4.66.1
      - traitlets==5.10.0
      - tzdata==2023.3
      - tzlocal==5.0.1
      - urllib3==2.0.4
      - wcwidth==0.2.6
      - wrapt==1.15.0
      - yte==1.5.1
prefix: /lustre/projects/dazzler/uelze/conda_envs/culebrONT3

I has graphviz=2.50.0, python=3.11.5 and r-base=4.2.0.

I usually create a conda environment first, activate it, install R, python and graphviz and then continue with the pip installation of culebrONT.

Besides the run_report, my culebrONT pipeline also fails at the steps run_racon_version and blobtools. Do any of these rules have anything in common? Its just a pity that snakemake does not provide any insight as to what the error might be. This may be related to the particular snakemake version (see similar case here snakemake/snakemake#1698). I also noticed that the jobs fail, before they are submitted to slurm. Not sure, how to proceed. Perhaps a successful installation on my system is not possible. =(

@julieaorjuela
Copy link
Contributor

Hello ! thanks for files
Maybe you can try to recovery report error by using the "singularity" version installation
try to put error and output in a directory by changing cluster_config.yaml

__default__:
    cpus-per-task: 1
    mem-per-cpu: 10G
    partition: PARTITION
    output: 'slurm_log/{log.output}_cluster'
    error: 'slurm_log/{log.error}_cluster'
    job-name: '{rule}.{wildcards}'

Could you confirm us you are installing culebront in a slurm cluster?
Thanks
Julie

@AlcaArctica
Copy link
Author

Yes, I want to use it on our institutional HPC cluster. I installed it with
culebrONT install_cluster --scheduler slurm --env singularity
Okay, I will change

output: '{log.output}_cluster'
error: '{log.error}_cluster'

to

output: 'slurm_log/{log.output}_cluster'
error: 'slurm_log/{log.error}_cluster'

But I am not sure, what this will achieve, when the log files are just empy, id doesnt matter where we save them?

@julieaorjuela
Copy link
Contributor

julieaorjuela commented Sep 19, 2023 •
edited
Loading

hello, the problem with report rule is because snakemake "script" key (into the report rule) sent stdout to the terminal and cannot be redirected to a log file.
We are trying to know if something has been written into the cluster log.... have you checked the sbatch log files? :S
And could you send us your sbatch script launching this culebront command line?
culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT/data_test_config.yaml --singularity-args "--bind /lustre/projects/dazzlerAssembly"

@sravel
Copy link
Member

sravel commented Sep 19, 2023

Hello,

Have you file call
/lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o_cluster
/lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e_cluster

Snakemake run Rmarkdown but can't redirect log on file. So normally this is write on terminal, on cluster this is write on slurm log.
We capture slurm log on the same folder of rule log and add prefix _cluster. of file.

for exemple I have error with pandoc:

Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=8041, mem_mib=7669, disk_mb=8041, disk_mib=7669
Singularity containers: ignored
Select jobs to execute...

[Wed Jun 14 12:07:42 2023]
Job 0: 
        print final CulebrONT_report ...
        
Reason: Missing output files: /shared/ifbstor1/projects/glp701/get2023/assemblies/fijiensis/FINAL_REPORT/CulebrONT_report.html

Rscript --vanilla -e 'rmarkdown::render("/shared/ifbstor1/projects/glp701/get2023/assemblies/fijiensis/.snakemake/scripts/tmpbh28e7gb.Report.Rmd", output_file
="/shared/ifbstor1/projects/glp701/get2023/assemblies/fijiensis/FINAL_REPORT/CulebrONT_report.html", quiet=TRUE, knit_root_dir = "/shared/ifbstor1/projects/gl
p701/get2023/assemblies/fijiensis", params = list(rmd="/shared/ifbstor1/projects/glp701/get2023/assemblies/fijiensis/.snakemake/scripts/tmpbh28e7gb.Report.Rmd
"))'
Activating environment modules: r
Erreur : pandoc version 1.12.3 or higher is required and was not found (see the help page ?rmarkdown::pandoc_available).
Exécution arrêtée
Not cleaning up /shared/ifbstor1/projects/glp701/get2023/assemblies/fijiensis/.snakemake/scripts/tmpbh28e7gb.Report.Rmd
[Wed Jun 14 12:07:47 2023]
Error in rule run_report:

@AlcaArctica
Copy link
Author

AlcaArctica commented Sep 19, 2023 •
edited
Loading

Not sure I understand. I need to wrap the culebrONT command in its own sbatch script? I previously just entered it on the command line. The pipeline itself then submits individual rules / jobs to slurm. Is that not correct?
Anyway, here I made on for you:
(/lustre/projects/dazzler/uelze/conda_envs/culebrONT3) [uelze@falcon1 test_culebrONT_slurm]$ cat slurm.sbatch

#!/bin/bash
#SBATCH -J test_culebrONT_slurm
#SBATCH -p gpu
#SBATCH -c 38 #cores
#SBATCH -n 1 #nodes
#SBATCH --time=2:00:00
#SBATCH -o myfile.out

cd /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/ 
culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/data_test_config.yaml  --singularity-args "--bind /lustre/projects/dazzlerAssembly"

@AlcaArctica
Copy link
Author

AlcaArctica commented Sep 19, 2023 •
edited
Loading

@julieaorjuela Alright, I am not understanding it fully, but now I have run the culebrONT run_cluster command inside my sbatch script I have found the following.

  1. It is much faster
  2. It still submits individual jobs to slurm
  3. It still fails at the racon step
    BUT
  4. I can now access some files, I previously couldnt, such as the 5PXoo1-RACON-VERSION.e_cluster, which can now be found in a new directory named slurm_log (this was what you intended by changing the two lines in the cluster_config.yaml, right?)
    The content of 5PXoo1-RACON-VERSION.e_cluster is:
Building DAG of jobs...
WorkflowError:
Failed to get singularity version:
WARNING: Could not lookup the current user's information: user: unknown userid 10819
FATAL:   Couldn't determine user account information: user: unknown userid 10819

The file 5PXoo1-RACON-VERSION.o_cluster is still empty.

PS: So I am guessing this is a problem with how I use singularity (similar problem here: apptainer/singularity#5962). I have not used singulariy before and I am new to this particular HPC. So I guess its not surprising I dont have a user id. I just dont understand why the other rules, which also depend on software inside the singularity container, such as the fly and canu assemblies finish just fine?

@sravel
Copy link
Member

sravel commented Sep 19, 2023

I am successfully running the test data set, but the pipeline stops at Error in rule run_report. I am executing the command culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT/data_test_config.yaml --singularity-args "--bind /lustre/projects/dazzlerAssembly"

This is the error I get

Error in rule run_report:
    jobid: 1
    input: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-CANU-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/DATA/5PXoo1-FLYE-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_time.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/dag.png, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/versions.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/all_stats_5PXoo1.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-CANU-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-PILON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-RACON.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/ASSEMBLIES/5PXoo1-FLYE-ASSEMBLY.fasta, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/QUAST/5PXoo1/icarus.html, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/AGGREGATED_QC/QUAST_RESULTS/QUAST-version.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_busco.csv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/BUSCO-version.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/BLOBTOOLS/5PXoo1_CANU_PILON.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/BLOBTOOLS/5PXoo1_FLYE_PILON.txt, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/ASSEMBLYTICS/5PXoo1__CANU__PILON.Assemblytics_structural_variants.summary, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/ASSEMBLYTICS/5PXoo1__FLYE__PILON.Assemblytics_structural_variants.summary, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/MERQURY/5PXoo1.qv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/MERQURY/5PXoo1.qv, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/MERQURY/5PXoo1.completeness.stats, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/FLYE/QUALITY/PILON/MERQURY/5PXoo1.completeness.stats, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/5PXoo1/REPORT/5PXoo1_flagstats.csv
    output: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/CulebrONT_report.html
    log: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o, /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e (check log file(s) for error details)
    cluster_jobid: 48918253
Logfile /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o not found.

Error executing rule run_report on cluster (jobid: 1, external: 48918253, jobscript: /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/.snakemake/tmp.03o82kqy/snakejob.run_report.1.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: ../.snakemake/log/2023-09-14T152430.940894.snakemake.log

Unfortunately, there is no hint as to what the problem may be and both /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o and /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e are empty. There is also no more information found in the /lustre/projects/dazzlerAssembly/test_culebrONT/.snakemake/log/2023-09-14T152430.940894.snakemake.log file.

What could it be? I am guessing there might be a problem with required R packages or similar. But strictly this should not happen, as I am using the singularity container?

Do you have any ideas how I could debug this problem? I would really love to see the report!

Can you send me the files:

  • /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.o_cluster
  • /lustre/projects/dazzlerAssembly/test_culebrONT/culebrONT_OUTPUT/FINAL_REPORT/LOGS/REPORT.e_cluster

@AlcaArctica
Copy link
Author

@sravel Thanks for chipping in! I do have the described file. However it only contains:

cat GRAPH.e
Building DAG of jobs...
Error: Directory cannot be locked. This usually means that another Snakemake instance is running on this directory. Another possibility is that a previous run exited unexpectedly.

I guess, it is because the pipeline already failed at the racon step and did not run all the way to the run_report rule.

@julieaorjuela
Copy link
Contributor

yes, in culebront command line you can also use snakemake parametters as --rerun-incomplete --nolock etc...

first, please launch culebront with unlock param :

culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/data_test_config.yaml  --unlock --singularity-args "--bind /lustre/projects/dazzlerAssembly

and add this params --rerun-incomplete --nolock to avoid this problem again such as:

culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/data_test_config.yaml   --rerun-incomplete --nolock  --singularity-args "--bind /lustre/projects/dazzlerAssembly

and relaunch again.

Julie

@julieaorjuela
Copy link
Contributor

and ... I wonder if report problem is not because yous singularity bind is strange
change this
--singularity-args "--bind /lustre/projects/dazzlerAssembly"
by
--singularity-args "--bind /lustre:/lustre"
Sorry for multiples answers !

@AlcaArctica
Copy link
Author

AlcaArctica commented Sep 20, 2023 •
edited
Loading

I am happy to report, that I got the report! :)
image
I used the following sbatch script:

#!/bin/bash
#SBATCH -J test_culebrONT_slurm
#SBATCH -p gpu
#SBATCH -c 38 #cores
#SBATCH -n 1 #nodes
#SBATCH --time=24:00:00
#SBATCH -o myfile.out

cd /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/ 
conda activate /lustre/projects/dazzler/uelze/conda_envs/culebrONT3
culebrONT run_cluster --config /lustre/projects/dazzlerAssembly/test_culebrONT_slurm/data_test_config.yaml --rerun-incomplete --nolock --singularity-args "--bind /lustre:/lustre"

In the end, I think the main culprit was that I did not define the singularity bind well. I hadn't used it before, and clearly specified it wrong (even though curiously it worked for some rules).
But for future problems its also really helpful to know how to produce the error messages in the slurm_log directory. Thanks @julieaorjuela for that tip!
The full pipeline run on the test data took about an hour on our cluster, with my slurm script and the default cluster config parameters. I only had to reduce the memory of canu from 25G to 20G.
So thank you so much for all your help. I also learned at lot. Next I will try to run your pipeline on some of our own data. I will keep you posted how that goes!

PS: One more simple question. I can display the report just fine on the cluster. However, to share it with my team I need to pull it onto another fileserver. I tried to copy the complete FINAL_REPORT directory, but then the connection between the blobtools and assemblytics plots breaks.
image
Which other files to I need to copy to keep the report intact? I do not want to copy the whole analyses directory if it can be helped, because of size restraints.
PPS: In the html code I can trace the images. I found for example, <p><img src="/lustre/projects/dazzlerAssembly/test_culebrONT_slurm/culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/CANU/QUALITY/PILON/BLOBTOOLS/blob.png" alt="thumbnail", so thats the reason why the blobtool plot is not displayed (its located in the culebrONT_OUTPUT/5PXoo1/ASSEMBLERS/ directory and not in the FINAL_REPORT directory. Maybe thats something that could be improved? It would be really nice to have the report self-contained in one directory.

@sravel sravel added the bug Something isn't working label Oct 24, 2023
@sravel sravel changed the title cant create report can't create report binding error Oct 24, 2023
@sravel sravel closed this as completed Oct 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@julieaorjuela @sravel @AlcaArctica