Identifying TIR uses only one CPU

[augustold]

Hi Shujun,
I am running EDTA.pl in a conda environment using --threads 30. The 'Identify LTR' step finished in less than one day and the 'Identify TIR' has been running for six days now. I've also noticed that this process is using only one CPU. Is it normal?

##### Extensive de-novo TE Annotator (EDTA) v1.7.9  ####
##### Shujun Ou (shujun.ou.1@gmail.com)             ####
########################################################



Mon Feb  3 19:37:57 -02 2020    Dependency checking:
                                All passed!

Mon Feb  3 19:38:41 -02 2020    Obtain raw TE libraries using various structure-based programs:
Mon Feb  3 19:38:41 -02 2020    EDTA_raw: Check dependencies, prepare working directories.

Mon Feb  3 19:38:53 -02 2020    Start to find LTR candidates.

Mon Feb  3 19:38:53 -02 2020    Identify LTR retrotransposon candidates from scratch.

Use of uninitialized value $chr_pre in hash element at /home/augustold/miniconda3/envs/EDTA/share/LTR_retriever/bin/call_seq_by_list.pl line 86.
Tue Feb  4 13:07:26 -02 2020    Finish finding LTR candidates.

Tue Feb  4 13:07:26 -02 2020    Start to find TIR candidates.

Tue Feb  4 13:07:26 -02 2020    Identify TIR candidates from scratch.

Species: others

Best wishes and thank you for providing this tool.

[oushujun]

You can check if it's producing new files in the TIR directory. Some of the processes are single threaded but most of those are quick. Shujun

…

On Mon, Feb 10, 2020, 12:53 PM augustold ***@***.***> wrote: Hi Shujun, I am running EDTA.pl in a conda environment using --threads 30. The 'Identify LTR' step finished in less than one day and the 'Identify TIR' has been running for six days now. I've also noticed that this process is using only one CPU. Is it normal? ##### Extensive de-novo TE Annotator (EDTA) v1.7.9 #### ##### Shujun Ou ***@***.***) #### ######################################################## Mon Feb 3 19:37:57 -02 2020 Dependency checking: All passed! Mon Feb 3 19:38:41 -02 2020 Obtain raw TE libraries using various structure-based programs: Mon Feb 3 19:38:41 -02 2020 EDTA_raw: Check dependencies, prepare working directories. Mon Feb 3 19:38:53 -02 2020 Start to find LTR candidates. Mon Feb 3 19:38:53 -02 2020 Identify LTR retrotransposon candidates from scratch. Use of uninitialized value $chr_pre in hash element at /home/augustold/miniconda3/envs/EDTA/share/LTR_retriever/bin/call_seq_by_list.pl line 86. Tue Feb 4 13:07:26 -02 2020 Finish finding LTR candidates. Tue Feb 4 13:07:26 -02 2020 Start to find TIR candidates. Tue Feb 4 13:07:26 -02 2020 Identify TIR candidates from scratch. Species: others Best wishes and thank you for providing this tool. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=ABNX4NEVW6B4UFM2CAGQVXLRCGPEHA5CNFSM4KSTTWAKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IMLA2OA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NDDFORU2LXDK7LZA33RCGPEHANCNFSM4KSTTWAA> .

[augustold]

Yes. There are files being produced in the folder '*.fa.mod.EDTA.raw/TIR/Module3_New/TIR-Learner'. I'll keep waiting.
Thank you for the quick reply.
Augusto

[oushujun]

Hi Augusto,

Just want to check back, did you finish the run successfully?

Thanks,
Shujun

[augustold]

Hi Shujun,
Unfortunately, I had to kill the process because after 30 days it didn't finish running. I'm dealing with a large polyploid genome and the assembly is highly fragmented. When I run EDTA.pl on a small slice of contigs, it works fine. What do you think about running the pipeline on subsets of contigs?

Thanks,
Augusto

[oushujun]

Hi Augusto,

Thanks for your feedback. Both genome size and fragment number are deterministic for the run time of TIR-Learner. I've run through the chromosomal-scaffold wheat genome and the TIR part took about 26 days (#61). I also tested on the apple genome which is not big but with 127k sequences and it took about 10 days with large memory (>128Gb and I allocate 1 TB on it). You may split the genome into different batches such as split by ploidy, and combine them afterward (also see #61 and other issues).

Best,
Shujun

[oushujun]

Close due to no activity. Please reopen if the issue persists.

[Dichopsis]

Hi @oushujun,
I have approximately the same problem, except that I have 0 process corresponding to the program. I don't have any error message either. Penaeus vanamei genome (1.6 Gb)

nohup singularity exec -B /mystore /biolo/EDTA/1.9.5/edta.sif EDTA.pl --genome GCF_003789085.1_ASM378908v1_genomic.fna --overwrite 1 --sensitive 1 --anno 1 --evaluate 1 --threads 48 > run_EDTA.out 2>&1 &

########################################################

Extensive de-novo TE Annotator (EDTA) v1.9.4

Shujun Ou (shujun.ou.1@gmail.com)

########################################################

Mon Mar 15 16:45:20 CET 2021 Dependency checking:
All passed!

Mon Mar 15 16:45:34 CET 2021 The longest sequence ID in the genome contains 136 characters, which is longer than the limit (15)
Trying to reformat seq IDs...
Attempt 1...
Mon Mar 15 16:45:42 CET 2021 Seq ID conversion successful!

Mon Mar 15 16:45:42 CET 2021 Obtain raw TE libraries using various structure-based programs:
Mon Mar 15 16:45:42 CET 2021 EDTA_raw: Check dependencies, prepare working directories.

Mon Mar 15 16:45:47 CET 2021 Start to find LTR candidates.

Mon Mar 15 16:45:47 CET 2021 Identify LTR retrotransposon candidates from scratch.

[oushujun]

Hi, Please try out the latest singularity image (1. 9. 6 if I remember correctly). Or use the conda version which is up to date. Best, Shujun

…

On Fri, Mar 19, 2021 at 4:42 PM Dichopsis ***@***.***> wrote: Hi @oushujun <https://github.com/oushujun>, I have approximately the same problem, except that I have 0 process corresponding to the program. I don't have any error message either. Penaeus vanamei genome (1.6 Gb) nohup singularity exec -B /mystore /biolo/EDTA/1.9.5/edta.sif EDTA.pl --genome GCF_003789085.1_ASM378908v1_genomic.fna --overwrite 1 --sensitive 1 --anno 1 --evaluate 1 --threads 48 > run_EDTA.out 2>&1 & ######################################################## Extensive de-novo TE Annotator (EDTA) v1.9.4 Shujun Ou ( ***@***.***) ######################################################## Mon Mar 15 16:45:20 CET 2021 Dependency checking: All passed! Mon Mar 15 16:45:34 CET 2021 The longest sequence ID in the genome contains 136 characters, which is longer than the limit (15) Trying to reformat seq IDs... Attempt 1... Mon Mar 15 16:45:42 CET 2021 Seq ID conversion successful! Mon Mar 15 16:45:42 CET 2021 Obtain raw TE libraries using various structure-based programs: Mon Mar 15 16:45:42 CET 2021 EDTA_raw: Check dependencies, prepare working directories. Mon Mar 15 16:45:47 CET 2021 Start to find LTR candidates. Mon Mar 15 16:45:47 CET 2021 Identify LTR retrotransposon candidates from scratch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NAZ37IJ25GM2JD7PGDTEME6RANCNFSM4KSTTWAA> .