MITE-Hunter produces no results #12

liu3zhenlab · 2019-08-16T13:05:34Z

Shujun,

Thank you for updating EDTA. I am using 1.3 on a maize genome and the MITE step took a long time (~11 days). The problem is that no MITE raw sequences were output after TIR and MITE runs. Now the running is at Helitron. I will update after the run is finished.

-Sanzhen

oushujun · 2019-08-16T16:23:29Z

Dear Sanzhen,

MITE-Hunter sometimes chokes and emits random errors. Normally a fresh rerun can solve such issues. You can just run the MITE module with EDTA_raw.pl under the same job directory.
Assume your genome file is genome.fa
Run:
rm -rf ./genome.fa.EDTA.raw/MITE
perl EDTA_raw.pl -genome genome.fa -type mite -t 10

Hope this helps!
Shujun

oushujun · 2019-08-16T21:42:01Z

I just pushed some new updates to EDTA, mainly to fix the TIR-Learner issue. Please reinstall EDTA and rerun it in the same work folder. Existing results will be reused so there is essentially no waste of time. Thank you for your patience and support!

liu3zhenlab · 2019-08-16T22:25:16Z

Shujun, thank you very much. This helps. I will update and test it.

QiushiLi · 2019-08-21T04:08:24Z

Dear Shujun,
In one of my fish cases, the MITE-Hunter choked on the blastall for more than 72h.

Here is the screen output

Sun Aug 18 03:38:01 MDT 2019 Dependency checking:
All passed!
Sun Aug 18 03:38:12 MDT 2019 Obtain raw TE libraries using various structure-based programs:
Sun Aug 18 03:38:12 MDT 2019 EDTA_raw: Check files and dependencies, prepare working directories.

Sun Aug 18 03:38:12 MDT 2019 Start to find LTR candidates.

Sun Aug 18 03:38:12 MDT 2019 Identify LTR retrotransposon candidates from scratch.

Sun Aug 18 04:53:51 MDT 2019 Finish finding LTR candidates.

Sun Aug 18 04:53:51 MDT 2019 Start to find TIR candidates.

Sun Aug 18 04:53:51 MDT 2019 Identify TIR candidates from scratch.

Species: others
Finish finding TIR candidates.

Sun Aug 18 08:17:50 MDT 2019 Start to find MITE candidates.

Sun Aug 18 08:17:50 MDT 2019 Identify MITE candidates from scratch.

The cmd hanging there:

blastall -i genome_raw_NC_200 -d genome_raw_NC_200 -e 1e-10 -p blastn -o genome_raw_NC_200.self -m 8 -v 60 -b 60 -a 24 -F F -G 4 -E 2 -q -3 -r 2

I tried to rerun this specific cmd, and the error reproduced.
Will see if a refresh rerun can solve the issue.

Thanks,
Qiushi

liu3zhenlab · 2019-08-21T12:59:25Z

My rerunning with the updated EDTA failed at the TIR step again. Will check in more details to see what was going on.

oushujun · 2019-08-23T17:20:18Z

Hi Sanzhen,

Sorry the TIR issue is not resolved. Please let me know the specific error message so that I can work on it.

Best,
Shujun

liu3zhenlab · 2019-08-23T17:56:23Z

Shujun, here is the error message:

Fri Aug 16 21:13:30 CDT 2019 Identify TIR candidates from scratch.
Species: Maize
cat: xxx/TIR/Module1/TIR-Learner/*-select.csv: No such file or directory
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /homes/liu3zhen/local/conda/EDTA2/EDTA/util/rename_tirlearner.pl line 18.
Warning: LOC list A.fasta.TIR.ext30.list is empty.
Warning: The TIR result file has 0 bp!

No output was generated for Module2 or Module3.

thanks,

-Sanzhen

oushujun · 2019-08-26T06:03:41Z

Hi @liu3zhenlab, @QiushiLi and @philippbayer,

I updated the TIR-Learner to v1.23 and hope to solve this issue. I also abandoned MITE-Hunter for its poor performance. Please update EDTA (git pull in /EDTA) and try again. Please let me know if it doesn't work on your side.

Best,
Shujun

philippbayer · 2019-08-26T07:28:06Z

Thanks for that!
So I'm still running the split-up EDTA_raw.pl with the four different types, after pulling the changes MITE just finishes immediately with the MITE-Hunter warning.
Should I rerun the TIR step separately since that's where the changes seem to have happened?

liu3zhenlab · 2019-08-26T13:06:46Z

Thanks a lot Shujun. MITE-Hunter is also slow. Glad you found an alternative solution.

oushujun · 2019-08-26T14:23:36Z

No you don't have to. The change is to copy over TIR results directly and pretend it's MITE result, which will be removed entirely in the next combination step. MITE-Hunter does not contribute a lot (usually just a couple sequences) so you won't loss too much of sensitivity.

…

On Mon, Aug 26, 2019, 2:28 AM Philipp Bayer ***@***.***> wrote: Thanks for that! So I'm still running the split-up EDTA_raw.pl with the four different types, after pulling the changes MITE just finishes immediately with the MITE-Hunter warning. Should I rerun the TIR step separately since that's where the changes seem to have happened? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=ABNX4NG7JOASUM5VKDHN7W3QGOAYNA5CNFSM4IMHJQHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5DQ3WY#issuecomment-524750299>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNX4NC5WZKRGOIZOR4TMMLQGOAYNANCNFSM4IMHJQHA> .

liu3zhenlab · 2019-08-26T17:41:56Z

Shujun, I tested the v1.5 with a small data set. It showed errors as:

########################################################

Extensive de-novo TE Annotator (EDTA) v1.5

Shujun Ou (shujun.ou.1@gmail.com)

########################################################

Mon Aug 26 12:33:52 CDT 2019 Dependency checking:
All passed!
Mon Aug 26 12:33:57 CDT 2019 Obtain raw TE libraries using various structure-based programs:
Mon Aug 26 12:33:57 CDT 2019 EDTA_raw: Check files and dependencies, prepare working directories.

Mon Aug 26 12:33:57 CDT 2019 Start to find LTR candidates.

Mon Aug 26 12:33:57 CDT 2019 Identify LTR retrotransposon candidates from scratch.

    Usage: perl cleanup.pl -f sample.fa [options] > sample.cln.fa 
Options:
	-misschar	n	Define the letter representing unknown sequences; case insensitive; default: n
	-Nscreen	[0|1]	Enable (1) or disable (0) the -nc parameter; default: 1
	-nc		[int]	Ambuguous sequence len cutoff; discard the entire sequence if > this number; default: 0
	-nr		[0-1]	Ambuguous sequence percentage cutoff; discard the entire sequence if > this number; default: 1
	-minlen		[int]	Minimum sequence length filter after clean up; default: 100 (bp)
	-cleanN		[0|1]	Retain (0) or remove (1) the -misschar taget in output sequence; default: 0
	-trf		[0|1]	Enable (1) or disable (0) tandem repeat finder (trf); default: 1
	-trf_path	path	Path to the trf program

cp: cannot stat ‘TF05-1v012.fasta.mod.retriever.scn.adj’: No such file or directory
cp: cannot stat ‘TF05-1v012.fasta.LTRlib.fa’: No such file or directory
cp: cannot stat ‘TF05-1v012.fasta.LTRlib.fa’: No such file or directory
Error: LTR results not found!

ERROR: Raw LTR results not found in TF05-1v012.fasta.EDTA.raw/TF05-1v012.fasta.LTR.raw.fa at /homes/liu3zhen/.conda/envs/EDTA3/EDTA/EDTA.pl line 176.

csxie-666 · 2019-08-27T11:18:11Z

Hi,shujun
Im using EDTA 1.5 on a nematode genome.The first running i got TIR errors(but i cant remmber error message....).Then I increase max user processes , TIR is running.
And i met same issue like QiushiLi, MITE's blastall produces no results. I change the version of blast from 2.2.6 to 2.2.2. MITE is running.

#################################################################
2019年 08月 25日星期日 17:07:57 CST EDTA_raw: Check files and dependencies, prepare working directories.

2019年 08月 25日星期日 17:07:57 CST Start to find LTR candidates.

2019年 08月 25日星期日 17:07:57 CST Identify LTR retrotransposon candidates from scratch.

2019年 08月 25日星期日 18:38:13 CST Finish finding LTR candidates.

2019年 08月 25日星期日 18:38:13 CST Start to find TIR candidates.

2019年 08月 25日星期日 18:38:13 CST Identify TIR candidates from scratch.

Species: others
Finish finding TIR candidates.

2019年 08月 25日星期日 20:55:41 CST Start to find MITE candidates.

2019年 08月 25日星期日 20:55:41 CST Identify MITE candidates from scratch.

2019年 08月 26日星期一 00:00:38 CST Finish finding MITE candidates.

2019年 08月 26日星期一 00:00:38 CST Start to find Helitron candidates.

2019年 08月 26日星期一 00:00:38 CST Identify Helitron candidates from scratch.

2019年 08月 26日星期一 02:04:48 CST Finish finding Helitron candidates.

2019年 08月 26日星期一 02:04:48 CST Execution of EDTA_raw.pl is finished!

#################################################################
success~

baozg · 2019-08-28T07:48:46Z

After changing blast version from 2.2.26 to 2.2.22, the MITE finally has its result. It worked for me.
Thanks! @csxie-666

liu3zhenlab · 2019-08-28T15:15:07Z

Seems we have a problem for numpy in our cluster. I changed to another server and now I can fully run EDTA :). Thanks Shujun.

oushujun · 2019-08-28T19:50:24Z

I will put the requirement for blast-legacy=2.2.22 in the next update. Thank you all for testing and reporting.

oushujun changed the title ~~MITE~~ MITE-Hunter produces no results Aug 16, 2019

oushujun added the bug Something isn't working label Aug 16, 2019

oushujun mentioned this issue Aug 26, 2019

EDTA v1.5 fail on a small dataset #16

Closed

oushujun closed this as completed Aug 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MITE-Hunter produces no results #12

MITE-Hunter produces no results #12

liu3zhenlab commented Aug 16, 2019

oushujun commented Aug 16, 2019

oushujun commented Aug 16, 2019

liu3zhenlab commented Aug 16, 2019

QiushiLi commented Aug 21, 2019

liu3zhenlab commented Aug 21, 2019

oushujun commented Aug 23, 2019

liu3zhenlab commented Aug 23, 2019

oushujun commented Aug 26, 2019 •

edited

philippbayer commented Aug 26, 2019

liu3zhenlab commented Aug 26, 2019

oushujun commented Aug 26, 2019 via email

liu3zhenlab commented Aug 26, 2019

csxie-666 commented Aug 27, 2019 •

edited

baozg commented Aug 28, 2019

liu3zhenlab commented Aug 28, 2019

oushujun commented Aug 28, 2019

MITE-Hunter produces no results #12

MITE-Hunter produces no results #12

Comments

liu3zhenlab commented Aug 16, 2019

oushujun commented Aug 16, 2019

oushujun commented Aug 16, 2019

liu3zhenlab commented Aug 16, 2019

QiushiLi commented Aug 21, 2019

liu3zhenlab commented Aug 21, 2019

oushujun commented Aug 23, 2019

liu3zhenlab commented Aug 23, 2019

oushujun commented Aug 26, 2019 • edited

philippbayer commented Aug 26, 2019

liu3zhenlab commented Aug 26, 2019

oushujun commented Aug 26, 2019 via email

liu3zhenlab commented Aug 26, 2019

Extensive de-novo TE Annotator (EDTA) v1.5

Shujun Ou (shujun.ou.1@gmail.com)

csxie-666 commented Aug 27, 2019 • edited

baozg commented Aug 28, 2019

liu3zhenlab commented Aug 28, 2019

oushujun commented Aug 28, 2019

oushujun commented Aug 26, 2019 •

edited

csxie-666 commented Aug 27, 2019 •

edited