hecatomb combine not working

[gregorykrice]

Running "hecatomb combine --comb SRR4295172/hecatomb.out --comb SRR4295173/hecatomb.out --comb SRR4295174/hecatomb.out --comb SRR4295175/hecatomb.out --output combined_output"

[2024:03:07 10:57:15] Config file combined_output/hecatomb.config.yaml already exists. Using existing config file.
[2024:03:07 10:57:15] Updating config file with new values
[2024:03:07 10:57:15] Writing config file to combined_output/hecatomb.config.yaml
[2024:03:07 10:57:15] ------------------
[2024:03:07 10:57:15] | Runtime config |
[2024:03:07 10:57:15] ------------------

hecatomb:
addHost:
minViralAlnLen: 1000
args:
comb:
- SRR4295172/hecatomb.out
- SRR4295173/hecatomb.out
- SRR4295174/hecatomb.out
- SRR4295175/hecatomb.out
combineRuns:
- SRR4295172/hecatomb.out
- SRR4295173/hecatomb.out
- SRR4295174/hecatomb.out
- SRR4295175/hecatomb.out
conda_prefix: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/conda
configfile: combined_output/hecatomb.config.yaml
example_profile: false
log: combined_output/hecatomb.log
output: combined_output
profile: null
simulate: false
snake_args: []
system_config: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/config/config.yaml
system_workflow_profile: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/profile/default/config.yaml
threads: 12
use_conda: true
workflow_profile: combined_output/hecatomb.profile
assembly:
canu: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False
genomeSize=10M -nanopore
flye: -g 1g
megahit: --presets meta-large
metaflye: --meta -g 1g --nano-raw
mmseqs:
fast: -s 4
filtAA: --min-length 30 -e 1e-5
filtNT: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97
--alignment-mode 3
sensitive: --start-sens 2 --sens-steps 3 -s 7
taxIdIgnore: 0 1 2 10239 131567 12429 2759
resources:
big:
cpu: 12
mem: 128000
time: '24:00:00'
lrg:
cpu: 8
mem: 32000
time: 04:00:00
med:
cpu: 8
mem: 16000
time: 04:00:00
ram:
cpu: 2
mem: 16000
time: 04:00:00
sml:
cpu: 1
mem: 2000
time: 01:00:00
trimnami:
qc:
compression: 1
cutadapt: -m 40
fastp: --qualified_quality_phred 15 --length_required 90 --cut_tail --cut_tail_window_size
25 --cut_tail_mean_quality 15 --dedup --dup_calc_accuracy 4 --trim_poly_x --detect_adapter_for_pe
filtlong: --min_length 1000 --keep_percent 95
hostRemoveFlagstat: -f 4 -F 3584
minimapIndex: -I 8G
prinseq: -min_len 60 -min_qual_mean 25 -ns_max_n 1 -derep 1 -out_format 0 -trim_tail_left
5 -trim_tail_right 5 -ns_max_n 5 -trim_qual_type min -trim_qual_left 30 -trim_qual_right
30 -trim_qual_window 10

[2024:03:07 10:57:15] Config file combined_output/hecatomb.profile/config.yaml already exists. Using existing config file.
[2024:03:07 10:57:15] ---------------------
[2024:03:07 10:57:15] | Snakemake command |
[2024:03:07 10:57:15] ---------------------

snakemake -s /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk --configfile combined_output/hecatomb.config.yaml --cores 12 --use-conda --conda-prefix /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/conda --workflow-profile combined_output/hecatomb.profile
Using profile combined_output/hecatomb.profile and workflow specific profile combined_output/hecatomb.profile for setting default command line arguments.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/immutable.yaml is extended by additional config specified via the command line.
No/missing assembly files for SRR4295172/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295173/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295174/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295175/hecatomb.out, skipping assembly files.
TypeError in file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/rules/preflight/contig_mapping.smk, line 8:
unsupported operand type(s) for +: 'NoneType' and 'str'
File "/opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk", line 52, in
File "/opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/rules/preflight/contig_mapping.smk", line 8, in
[2024:03:07 10:57:15] ERROR: Snakemake failed

The comb directories have results/merged_assembly.fasta files, but it looks like combineOutputs.smk is expecting assembly.fasta files.

[beardymcjohnface]

This might be a simple fix. Can you please try and update the combined_output/hecatomb.config.yaml file? I think you just need to add a config setting:

hecatomb:
    args:
+        assembly: merged

If that works I'll push a fix for the next release. I also need to add build tests for this workflow at some point.

[gregorykrice]

[2024:03:13 11:51:59] Config file combined_output/hecatomb.config.yaml already exists. Using existing config file.
[2024:03:13 11:51:59] Updating config file with new values
[2024:03:13 11:51:59] Writing config file to combined_output/hecatomb.config.yaml
[2024:03:13 11:51:59] ------------------
[2024:03:13 11:51:59] | Runtime config |
[2024:03:13 11:51:59] ------------------

hecatomb:
addHost:
minViralAlnLen: 1000
args:
assembly: merged
comb:
- SRR4295172/hecatomb.out
- SRR4295173/hecatomb.out
- SRR4295174/hecatomb.out
- SRR4295175/hecatomb.out
combineRuns:
- SRR4295172/hecatomb.out
- SRR4295173/hecatomb.out
- SRR4295174/hecatomb.out
- SRR4295175/hecatomb.out
conda_prefix: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/conda
configfile: combined_output/hecatomb.config.yaml
example_profile: false
log: combined_output/hecatomb.log
output: combined_output
profile: null
simulate: false
snake_args: []
system_config: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/config/config.yaml
system_workflow_profile: /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/profile/default/config.yaml
threads: 12
use_conda: true
workflow_profile: combined_output/hecatomb.profile
assembly:
canu: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False
genomeSize=10M -nanopore
flye: -g 1g
megahit: --presets meta-large
metaflye: --meta -g 1g --nano-raw
mmseqs:
fast: -s 4
filtAA: --min-length 30 -e 1e-5
filtNT: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97
--alignment-mode 3
sensitive: --start-sens 2 --sens-steps 3 -s 7
taxIdIgnore: 0 1 2 10239 131567 12429 2759
resources:
big:
cpu: 12
mem: 128000
time: '24:00:00'
lrg:
cpu: 8
mem: 32000
time: 04:00:00
med:
cpu: 8
mem: 16000
time: 04:00:00
ram:
cpu: 2
mem: 16000
time: 04:00:00
sml:
cpu: 1
mem: 2000
time: 01:00:00
trimnami:
qc:
compression: 1
cutadapt: -m 40
fastp: --qualified_quality_phred 15 --length_required 90 --cut_tail --cut_tail_window_size
25 --cut_tail_mean_quality 15 --dedup --dup_calc_accuracy 4 --trim_poly_x --detect_adapter_for_pe
filtlong: --min_length 1000 --keep_percent 95
hostRemoveFlagstat: -f 4 -F 3584
minimapIndex: -I 8G
prinseq: -min_len 60 -min_qual_mean 25 -ns_max_n 1 -derep 1 -out_format 0 -trim_tail_left
5 -trim_tail_right 5 -ns_max_n 5 -trim_qual_type min -trim_qual_left 30 -trim_qual_right
30 -trim_qual_window 10

[2024:03:13 11:51:59] Config file combined_output/hecatomb.profile/config.yaml already exists. Using existing config file.
[2024:03:13 11:51:59] ---------------------
[2024:03:13 11:51:59] | Snakemake command |
[2024:03:13 11:51:59] ---------------------

snakemake -s /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk --configfile combined_output/hecatomb.config.yaml --cores 12 --use-conda --conda-prefix /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/conda --workflow-profile combined_output/hecatomb.profile
Using profile combined_output/hecatomb.profile and workflow specific profile combined_output/hecatomb.profile for setting default command line arguments.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Config file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/../config/immutable.yaml is extended by additional config specified via the command line.
No/missing assembly files for SRR4295172/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295173/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295174/hecatomb.out, skipping assembly files.
No/missing assembly files for SRR4295175/hecatomb.out, skipping assembly files.
KeyError in file /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk, line 162:
'resources'
File "/opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk", line 162, in
[2024:03:13 11:51:59] ERROR: Snakemake failed

awk 'NR>=161 && NR<=163 {print}' /opt/mambaforge/envs/hecatomb/lib/python3.12/site-packages/hecatomb/snakemake/workflow/combineOutputs.smk
threads:
config["resources"]["med"]["cpu"]
resources:

[beardymcjohnface]

I think there are a couple of things I need to fix. I'll see if I can get it done this week.