Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR ~ Error executing process > 'graph_align_reads (41)' #33

Open
yykaya opened this issue Jul 12, 2024 · 7 comments
Open

ERROR ~ Error executing process > 'graph_align_reads (41)' #33

yykaya opened this issue Jul 12, 2024 · 7 comments

Comments

@yykaya
Copy link

yykaya commented Jul 12, 2024 •
edited
Loading

Hi,

I'm attempting to run GraffiTE in GT-sv-GA mode, but I encounter an error during the execution of the graph_align_reads (41) process, as shown in the message below. (However, when I run this step manually using GraphAligner, it completes successfully but the job gets killed.) Do you have any idea what might be causing this issue?

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

executor > local (507)
[21/3da086] process > map_asm (1) [100%] 115 of 115 ✔
[cb/7c0a52] process > svim_asm (115) [100%] 115 of 115 ✔
[13/513669] process > survivor_merge [100%] 1 of 1 ✔
[b8/15025a] process > repeatmask_VCF (1) [100%] 1 of 1 ✔
[a8/d2e2df] process > tsd_prep (1) [100%] 1 of 1 ✔
[94/e3d127] process > tsd_search (148) [100%] 157 of 157 ✔
[60/be72ec] process > tsd_report (1) [100%] 1 of 1 ✔
[77/79f066] process > make_graph (1) [100%] 1 of 1 ✔
[1a/75fbbf] process > graph_align_reads (49) [100%] 115 of 115, failed: 89 ✘
[- ] process > vg_call -
[- ] process > merge_VCFs -
ERROR ~ Error executing process > 'graph_align_reads (41)'

Caused by:
Process graph_align_reads (41) terminated with an error exit status (134)

Command executed:

GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

vg pack -x index/index.vg -g X.gam -o X.pack -Q 0

Command exit status:
134

Command output:
GraphAligner bioconda 1.0.13-
Load graph from index/index.vg
Build minimizer seeder from the graph
Minimizer seeds, length 15, window size 20, density 10
Seed cluster size 1
Alignment bandwidth 10
Clip alignment ends with identity < 66%
X-drop DP score cutoff 14705
write alignments to X.gam
Align

Command error:
INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO: fuse2fs not found, will not be able to mount EXT3 filesystems
GraphAligner bioconda 1.0.13-
GraphAligner bioconda 1.0.13-
Load graph from index/index.vg
Build minimizer seeder from the graph
Minimizer seeds, length 15, window size 20, density 10
Seed cluster size 1
Alignment bandwidth 10
Clip alignment ends with identity < 66%
X-drop DP score cutoff 14705
write alignments to X.gam
Align
Signal 11. Read: 2a9030c3-0fd1-4219-8496-ad2d7f1fb33b runid=b8d4812b34a5a9fdf693319045b7ee7e26775c97 read=30068 ch=2021 start_time=2019-11-30T05:26:38Z flow_cell_id=PAD94007 protocol_group_id=191128-CLIMARES sample_id=MULTIPLEX6-2-M40-S
00-L14 barcode=barcode07. Seed: 34132+,3021,15,752
.command.sh: line 2: 16 Aborted GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

This is from .nextflow log file:

Jul-12 03:21:25.913 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (35); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/afee1f76707eac941eba261f63cc7d
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (35) failed
Jul-12 03:21:25.914 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 426; name: graph_align_reads (34); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE
xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656]
Jul-12 03:21:25.914 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (34); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (34) failed
Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 440; name: graph_align_reads (48); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE
xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a]
Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (48); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (48) failed
Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 396; name: graph_align_reads (4); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e]
Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (4); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (4) failed
Jul-12 03:21:25.916 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 434; name: graph_align_reads (42); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88]
Jul-12 03:21:25.916 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (42); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (42) failed
Jul-12 03:21:25.929 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 483; name: graph_align_reads (91); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24]
Jul-12 03:21:25.929 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (91); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (91) failed
Jul-12 03:21:25.953 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 405; name: graph_align_reads (13); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1]
Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (13); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (13) failed
Jul-12 03:21:25.954 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 462; name: graph_align_reads (70); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3]
Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (70); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (70) failed
Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 484; name: graph_align_reads (92); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8]
Jul-12 03:21:25.955 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (92); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (92) failed
Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 441; name: graph_align_reads (49); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c]
Jul-12 03:21:25.956 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=graph_align_reads (49); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c
error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (49) failed
Jul-12 03:21:26.093 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-12 03:21:26.103 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jul-12 03:21:26.210 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=418; failedCount=89; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=9d 21h 44m 9s; failedDuration=43d 16h 36m 38s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=115; peakCpus=115; peakMemory=480 GB; ]
Jul-12 03:21:26.517 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-12 03:21:26.571 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-12 03:21:26.571 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

The command I tried:

/u/ykaya/nextflow run https://github.com/cgroza/GraffiTE --genotype true \
   --assemblies assemblies.csv \
   --TE_library TE.nonredun.fa \
   --reference /ptmp/X.fasta \
   --graph_method graphaligner \
   --reads longreads.csv
@cgroza
Copy link
Owner

cgroza commented Jul 12, 2024 via email

@yykaya
Copy link
Author

yykaya commented Jul 14, 2024

I disagree that the issue is solely due to time limits. Despite attempting the task three times with different configurations and time limits, I consistently encounter the error "ERROR ~ Error executing process > 'graph_align_reads (105)'" for various accessions. My jobs run on HPC nodes equipped with GPUs, allowing for a 24-hour execution window. The tool typically runs for 18-19 hours before encountering these errors.

Here are the updated commands I'm using (currently running for 115 samples):

/u/ykaya/nextflow run https://github.com/cgroza/GraffiTE --genotype true \
   --assemblies assemblies.csv \
   --TE_library TE.nonredun.fa \
   --reference /ptmp/X.fasta \
   --graph_method graphaligner \
   --cores 4 \
   --repeatmasker_memory 18G \
   --graph_align_memory 18G \
   --vg_call_memory 18G \
   --make_graph_memory 18G \
   --map_longreads_memory 18G \
   --map_longreads_time 5h \
   --sniffles_time 8h \
   --graph_align_time 18h \
   --repeatmasker_time 5h \
   --min_mapq 5 \
   --longreads longreads.csv

@cgroza
Copy link
Owner

cgroza commented Jul 15, 2024 •
edited
Loading

I see this line in your logs coming from GraphAligner:

Signal 11. Read: 2a9030c3-0fd1-4219-8496-ad2d7f1fb33b runid=b8d4812b34a5a9fdf693319045b7ee7e26775c97 read=30068 ch=2021 start_time=2019-11-30T05:26:38Z flow_cell_id=PAD94007 protocol_group_id=191128-CLIMARES sample_id=MULTIPLEX6-2-M40-S
00-L14 barcode=barcode07. Seed: 34132+,3021,15,752

It indicates the name of the read that crashed GraphAligner.
This is either a bug in GraphAligner or the read crashes GraphAligner.
Is it possible that it's running out of memory?

@clemgoub
Copy link
Collaborator

clemgoub commented Jul 18, 2024 •
edited
Loading

I had similar issue in the past, and I would lean onto the possibility that this is a bug in GraphAligner. I would suggest to perhaps remove this read from the dataset (though another one may be problematic).

This error seem to be non-deterministic. What seemed to work in some case was changing the number of CPU. I had mentioned the problem here but did not get any answer. I will update my message there in case they can help.

Best,

Clément

@yykaya
Copy link
Author

yykaya commented Jul 19, 2024

Despite multiple attempts to run with GraphAligner, the issue persists.

I have now attempted the genotyping step using vg giraffie, but encountered the following error message. This issue appears to have been addressed in commit For reference, you can review the related issue

executor > local (621)
[d0/8f092f] process > map_longreads (76) [100%] 115 of 115 ✔
[fd/0cc43b] process > sniffles_sample_call (115) [100%] 115 of 115 ✔
[44/c4b4f6] process > sniffles_population_call (1) [100%] 1 of 1 ✔
[ba/97c7eb] process > map_asm (105) [100%] 115 of 115 ✔
[45/77378f] process > svim_asm (115) [100%] 115 of 115 ✔
[b4/e3ced0] process > survivor_merge [100%] 1 of 1 ✔
[2c/60a236] process > merge_svim_sniffles2 (1) [100%] 1 of 1 ✔
[91/1b394e] process > repeatmask_VCF (1) [100%] 1 of 1 ✔
[50/838199] process > tsd_prep (1) [100%] 1 of 1 ✔
[c5/ed6b2e] process > tsd_search (3) [100%] 154 of 154 ✔
[c7/9d28ed] process > tsd_report (1) [100%] 1 of 1 ✔
[d5/b8a4b2] process > make_graph (1) [100%] 1 of 1, failed: 1 ✘
[- ] process > graph_align_reads -
[- ] process > vg_call -
[- ] process > merge_VCFs -
ERROR ~ Error executing process > 'make_graph (1)'

Caused by:
Process make_graph (1) terminated with an error exit status (134)

Command executed:

bcftools sort -Oz -o sorted.vcf.gz pangenome.vcf
tabix sorted.vcf.gz
mkdir index

  vg autoindex --tmp-dir $PWD  -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta

vg snarls index/index.giraffe.gbz > index/index.pb

Command exit status:
134

Command output:
(empty)

Command error:
INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO: fuse2fs not found, will not be able to mount EXT3 filesystems
Writing to /tmp/bcftools.OAKXDo
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_ID"
Merging 1 temporary files
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE"
[W::bcf_hrec_check] Invalid tag name: "svim-asm_ID"
Cleaning
Done
[vg autoindex] Executing command: vg autoindex --tmp-dir /raven/ptmp/ykaya/path/work/d5/b8a4b27a8b0aa20f39868bb276d9a5 -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta
[IndexRegistry]: Checking for phasing in VCF(s).
[IndexRegistry]: Chunking inputs for parallelism.
index file X.fasta.fai not found, generating...
[IndexRegistry]: Chunking FASTA(s).
[IndexRegistry]: Chunking VCF(s).
[IndexRegistry]: Constructing VG graph from FASTA and VCF input.
warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr2; coercing to N.
warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr3; coercing to N.
warning:[vg::Constructor] Skipping duplicate variant with hash 7a4a5efd14440d32a6d1bb6f91a992cd6178fcf7 at Chr3:3110404
warning:[vg::Constructor] Skipping duplicate variant with hash d1c7edc8494207c7c32084674a18a25159078735 at Chr4:4150882
warning:[vg::Constructor] Skipping duplicate variant with hash e37e77706134d082ccb2d9eb8ac109fa35b4f3a3 at Chr2:4593323
warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr4; coercing to N.
vg: src/constructor.cpp:528: vg::ConstructedChunk vg::Constructor::construct_chunk(std::string, std::string, std::vectorvcflib::Variant, size_t) const: Assertion `!variant->isSymbolicSV()' failed.
ERROR: Signal 6 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.
Stack trace path: /tmp/vg_crash_9IYtPC/stacktrace.txt
Please include the stack trace file in your bug report!

any comments and suggestions are welcome!

@cgroza
Copy link
Owner

cgroza commented Jul 19, 2024 via email

@cgroza
Copy link
Owner

cgroza commented Sep 5, 2024

Also

std::vectorvcflib::Variant, size_t) const: Assertion `!variant->isSymbolicSV()' failed.

This tells me that there are SVs in the VCF that are not fully sequence resolved.
Could we see sample of these so we can filter them out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cgroza @clemgoub @yykaya