Error in run annotate predict #517
Comments
|
oops, seems like this is just like #503. I update diamond. Will let you know if that solved my problem |
|
I updated diamond but I still get the same issue. Could you please give me advise on how to solve this issue? |
|
after upgrading diamond did you rebuild your diamond indexes - eg running |
|
Thanks for your response. I run predict again and looking at my system monitor it does look like all the memory is quickly used and then the error appears. What could I do to solve this issue o so much memory being used at this stage? |
|
It has to do with large contigs I think. So with perhaps the newest version I think the memory leak was fixed if you use some other parameters. I've never seen the error presumably because I've not used a large enough reference genome/contig. What versions of diamond have you tried? |
|
the new diamond was a minor fix only, so for me it worked with the same diamond DB. But I am using diamond in frameshift mode to avoid the segfault on large sequences (the memory leak was fixed but the segfault no yet or not that I know of. the frameshift mode is a workaround) nano $HOME/.conda/envs/funnotate181/lib/python3.7/site-packages/funannotate/aux_scripts/funannotate-p2g.py |
Are you using the latest release?
Hello, I amusing the latest release of funnanotate but keep getting this error when I run fun annotate predict (I already run the mask, and train steps of the pipeline)
base) digitalstorm@velox:/mnt/cb2d0a32-df3a-4e6f-8491-e153d9ae19f4/ana/holbrook_annotations$ funannotate predict -i hol_masked.fasta -s "Stristichus11_hol" -o Chromosomes_NEW_TRI/ --busco_db tetrapoda --organism other --repeats2evm --cpus 10 --max_intronlen 500000 --optimize_augustus
[08:32 AM]: OS: Ubuntu 20.04, 64 cores, ~ 132 GB RAM. Python: 3.7.6
[08:32 AM]: Running funannotate v1.8.1
[08:32 AM]: Found training files, will re-use these files:
--rna_bam Chromosomes_NEW_TRI/training/funannotate_train.coordSorted.bam
--pasa_gff Chromosomes_NEW_TRI/training/funannotate_train.pasa.gff3
--stringtie Chromosomes_NEW_TRI/training/funannotate_train.stringtie.gtf
--transcript_alignments Chromosomes_NEW_TRI/training/funannotate_train.transcripts.gff3
[08:32 AM]: Skipping CodingQuarry as --organism=other. Pass a weight larger than 0 to run CQ, ie --weights codingquarry:1
[08:32 AM]: Parsed training data, run ab-initio gene predictors as follows:
Program Training-Method
augustus pasa
genemark selftraining
glimmerhmm pasa
snap pasa
[08:33 AM]: Loading genome assembly and parsing soft-masked repetitive sequences
[08:34 AM]: Genome loaded: 11 scaffolds; 1,617,070,191 bp; 7.77% repeats masked
[08:34 AM]: Parsed 310,522 transcript alignments from: Chromosomes_NEW_TRI/training/funannotate_train.transcripts.gff3
[08:34 AM]: Creating transcript EVM alignments and Augustus transcripts hintsfile
[08:34 AM]: Existing RNA-seq BAM hints found: Chromosomes_NEW_TRI/predict_misc/hints.BAM.gff
[08:34 AM]: Mapping 550,947 proteins to genome using diamond and exonerate
[10:48 PM]: CMD ERROR: diamond blastx --threads 8 -q /mnt/cb2d0a32-df3a-4e6f-8491-e153d9ae19f4/ana/holbrook_annotations/Chromosomes_NEW_TRI/predict_misc/genome.softmasked.fa --db diamond -o diamond.matches.tab -e 1e-10 -k 0 --more-sensitive -f 6 sseqid slen sstart send qseqid qlen qstart qend pident length evalue score qcovhsp qframe
b"diamond v2.0.4.142 (C) Max Planck Society for the Advancement of Science\nDocumentation, support and updates available at http://www.diamondsearch.org\n\n#CPU threads: 8\nScoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)\nTemporary directory: \nOpening the database... [0.043s]\n#Target sequences to report alignments for: unlimited\nReference = diamond.dmnd\nSequences = 550947\nLetters = 202630500\nBlock size = 2000000000\nOpening the input file... [0.02s]\nOpening the output file... [0s]\nLoading query sequences... [12.157s]\nMasking queries... [49.876s]\nBuilding query seed set... [0s]\n\x1b[1;33mThe host system is detected to have 135 GB of RAM. It is recommended to use this parameter for better performance: -c1\n\x1b[0;39mAlgorithm: Double-indexed\nBuilding query histograms... [48.813s]\nAllocating buffers... [0s]\nLoading reference sequences... [0.537s]\nMasking reference... [1.171s]\nInitializing temporary storage... [0s]\nBuilding reference histograms... [4.333s]\nAllocating buffers... [0s]\nProcessing query block 1, reference block 1/1, shape 1/16, index chunk 1/4.\nBuilding reference seed array... [0.299s]\nBuilding query seed array... [3.421s]\nComputing hash join... [1.084s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.998s]\nProcessing query block 1, reference block 1/1, shape 1/16, index chunk 2/4.\nBuilding reference seed array... [0.395s]\nBuilding query seed array... [4.248s]\nComputing hash join... [0.815s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.781s]\nProcessing query block 1, reference block 1/1, shape 1/16, index chunk 3/4.\nBuilding reference seed array... [0.386s]\nBuilding query seed array... [4.595s]\nComputing hash join... [0.833s]\nBuilding seed filter... [0.062s]\nSearching alignments... [7.65s]\nProcessing query block 1, reference block 1/1, shape 1/16, index chunk 4/4.\nBuilding reference seed array... [0.288s]\nBuilding query seed array... [3.362s]\nComputing hash join... [0.777s]\nBuilding seed filter... [0.062s]\nSearching alignments... [7.922s]\nProcessing query block 1, reference block 1/1, shape 2/16, index chunk 1/4.\nBuilding reference seed array... [0.293s]\nBuilding query seed array... [3.486s]\nComputing hash join... [0.77s]\nBuilding seed filter... [0.071s]\nSearching alignments... [7.899s]\nProcessing query block 1, reference block 1/1, shape 2/16, index chunk 2/4.\nBuilding reference seed array... [0.336s]\nBuilding query seed array... [4.395s]\nComputing hash join... [0.774s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.776s]\nProcessing query block 1, reference block 1/1, shape 2/16, index chunk 3/4.\nBuilding reference seed array... [0.377s]\nBuilding query seed array... [4.538s]\nComputing hash join... [0.763s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.689s]\nProcessing query block 1, reference block 1/1, shape 2/16, index chunk 4/4.\nBuilding reference seed array... [0.314s]\nBuilding query seed array... [3.681s]\nComputing hash join... [0.768s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.807s]\nProcessing query block 1, reference block 1/1, shape 3/16, index chunk 1/4.\nBuilding reference seed array... [0.293s]\nBuilding query seed array... [3.496s]\nComputing hash join... [0.8s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.889s]\nProcessing query block 1, reference block 1/1, shape 3/16, index chunk 2/4.\nBuilding reference seed array... [0.344s]\nBuilding query seed array... [4.13s]\nComputing hash join... [0.787s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.937s]\nProcessing query block 1, reference block 1/1, shape 3/16, index chunk 3/4.\nBuilding reference seed array... [0.397s]\nBuilding query seed array... [4.327s]\nComputing hash join... [0.781s]\nBuilding seed filter... [0.062s]\nSearching alignments... [7.827s]\nProcessing query block 1, reference block 1/1, shape 3/16, index chunk 4/4.\nBuilding reference seed array... [0.308s]\nBuilding query seed array... [3.352s]\nComputing hash join... [0.768s]\nBuilding seed filter... [0.073s]\nSearching alignments... [7.87s]\nProcessing query block 1, reference block 1/1, shape 4/16, index chunk 1/4.\nBuilding reference seed array... [0.313s]\nBuilding query seed array... [3.412s]\nComputing hash join... [0.772s]\nBuilding seed filter... [0.063s]\nSearching alignments... [8.053s]\nProcessing query block 1, reference block 1/1, shape 4/16, index chunk 2/4.\nBuilding reference seed array... [0.363s]\nBuilding query seed array... [4.374s]\nComputing hash join... [0.818s]\nBuilding seed filter... [0.062s]\nSearching alignments... [7.874s]\nProcessing query block 1, reference block 1/1, shape 4/16, index chunk 3/4.\nBuilding reference seed array... [0.383s]\nBuilding query seed array... [4.508s]\nComputing hash join... [0.83s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.715s]\nProcessing query block 1, reference block 1/1, shape 4/16, index chunk 4/4.\nBuilding reference seed array... [0.307s]\nBuilding query seed array... [3.548s]\nComputing hash join... [0.779s]\nBuilding seed filter... [0.061s]\nSearching alignments... [7.891s]\nProcessing query block 1, reference block 1/1, shape 5/16, index chunk 1/4.\nBuilding reference seed array... [0.323s]\nBuilding query seed array... [3.446s]\nComputing hash join... [0.785s]\nBuilding seed filter... [0.076s]\nSearching alignments... [8.017s]\nProcessing query block 1, reference block 1/1, shape 5/16, index chunk 2/4.\nBuilding reference seed array... [0.355s]\nBuilding query seed array... [4.215s]\nComputing hash join... [0.793s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.752s]\nProcessing query block 1, reference block 1/1, shape 5/16, index chunk 3/4.\nBuilding reference seed array... [0.39s]\nBuilding query seed array... [4.561s]\nComputing hash join... [0.815s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.653s]\nProcessing query block 1, reference block 1/1, shape 5/16, index chunk 4/4.\nBuilding reference seed array... [0.301s]\nBuilding query seed array... [3.546s]\nComputing hash join... [0.809s]\nBuilding seed filter... [0.061s]\nSearching alignments... [7.804s]\nProcessing query block 1, reference block 1/1, shape 6/16, index chunk 1/4.\nBuilding reference seed array... [0.316s]\nBuilding query seed array... [3.478s]\nComputing hash join... [0.799s]\nBuilding seed filter... [0.067s]\nSearching alignments... [7.844s]\nProcessing query block 1, reference block 1/1, shape 6/16, index chunk 2/4.\nBuilding reference seed array... [0.372s]\nBuilding query seed array... [4.141s]\nComputing hash join... [0.798s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.73s]\nProcessing query block 1, reference block 1/1, shape 6/16, index chunk 3/4.\nBuilding reference seed array... [0.386s]\nBuilding query seed array... [4.425s]\nComputing hash join... [0.807s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.773s]\nProcessing query block 1, reference block 1/1, shape 6/16, index chunk 4/4.\nBuilding reference seed array... [0.325s]\nBuilding query seed array... [3.384s]\nComputing hash join... [0.802s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.839s]\nProcessing query block 1, reference block 1/1, shape 7/16, index chunk 1/4.\nBuilding reference seed array... [0.309s]\nBuilding query seed array... [3.526s]\nComputing hash join... [0.814s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.851s]\nProcessing query block 1, reference block 1/1, shape 7/16, index chunk 2/4.\nBuilding reference seed array... [0.372s]\nBuilding query seed array... [4.084s]\nComputing hash join... [0.796s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.598s]\nProcessing query block 1, reference block 1/1, shape 7/16, index chunk 3/4.\nBuilding reference seed array... [0.38s]\nBuilding query seed array... [4.459s]\nComputing hash join... [0.79s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.803s]\nProcessing query block 1, reference block 1/1, shape 7/16, index chunk 4/4.\nBuilding reference seed array... [0.291s]\nBuilding query seed array... [3.361s]\nComputing hash join... [0.791s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.768s]\nProcessing query block 1, reference block 1/1, shape 8/16, index chunk 1/4.\nBuilding reference seed array... [0.303s]\nBuilding query seed array... [3.437s]\nComputing hash join... [0.817s]\nBuilding seed filter... [0.077s]\nSearching alignments... [7.92s]\nProcessing query block 1, reference block 1/1, shape 8/16, index chunk 2/4.\nBuilding reference seed array... [0.387s]\nBuilding query seed array... [4.174s]\nComputing hash join... [0.816s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.865s]\nProcessing query block 1, reference block 1/1, shape 8/16, index chunk 3/4.\nBuilding reference seed array... [0.389s]\nBuilding query seed array... [4.437s]\nComputing hash join... [0.844s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.843s]\nProcessing query block 1, reference block 1/1, shape 8/16, index chunk 4/4.\nBuilding reference seed array... [0.266s]\nBuilding query seed array... [3.551s]\nComputing hash join... [0.818s]\nBuilding seed filter... [0.063s]\nSearching alignments... [7.73s]\nProcessing query block 1, reference block 1/1, shape 9/16, index chunk 1/4.\nBuilding reference seed array... [0.279s]\nBuilding query seed array... [3.456s]\nComputing hash join... [0.804s]\nBuilding seed filter... [0.068s]\nSearching alignments... [7.909s]\nProcessing query block 1, reference block 1/1, shape 9/16, index chunk 2/4.\nBuilding reference seed array... [0.362s]\nBuilding query seed array... [4.156s]\nComputing hash join... [0.827s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.852s]\nProcessing query block 1, reference block 1/1, shape 9/16, index chunk 3/4.\nBuilding reference seed array... [0.394s]\nBuilding query seed array... [4.498s]\nComputing hash join... [0.817s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.73s]\nProcessing query block 1, reference block 1/1, shape 9/16, index chunk 4/4.\nBuilding reference seed array... [0.315s]\nBuilding query seed array... [3.383s]\nComputing hash join... [0.818s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.97s]\nProcessing query block 1, reference block 1/1, shape 10/16, index chunk 1/4.\nBuilding reference seed array... [0.296s]\nBuilding query seed array... [3.606s]\nComputing hash join... [0.801s]\nBuilding seed filter... [0.071s]\nSearching alignments... [7.786s]\nProcessing query block 1, reference block 1/1, shape 10/16, index chunk 2/4.\nBuilding reference seed array... [0.335s]\nBuilding query seed array... [4.178s]\nComputing hash join... [0.8s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.844s]\nProcessing query block 1, reference block 1/1, shape 10/16, index chunk 3/4.\nBuilding reference seed array... [0.387s]\nBuilding query seed array... [4.481s]\nComputing hash join... [0.827s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.693s]\nProcessing query block 1, reference block 1/1, shape 10/16, index chunk 4/4.\nBuilding reference seed array... [0.282s]\nBuilding query seed array... [3.441s]\nComputing hash join... [0.821s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.882s]\nProcessing query block 1, reference block 1/1, shape 11/16, index chunk 1/4.\nBuilding reference seed array... [0.298s]\nBuilding query seed array... [3.431s]\nComputing hash join... [0.84s]\nBuilding seed filter... [0.076s]\nSearching alignments... [8.045s]\nProcessing query block 1, reference block 1/1, shape 11/16, index chunk 2/4.\nBuilding reference seed array... [0.377s]\nBuilding query seed array... [4.058s]\nComputing hash join... [0.813s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.737s]\nProcessing query block 1, reference block 1/1, shape 11/16, index chunk 3/4.\nBuilding reference seed array... [0.4s]\nBuilding query seed array... [4.53s]\nComputing hash join... [0.829s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.805s]\nProcessing query block 1, reference block 1/1, shape 11/16, index chunk 4/4.\nBuilding reference seed array... [0.287s]\nBuilding query seed array... [3.281s]\nComputing hash join... [0.833s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.806s]\nProcessing query block 1, reference block 1/1, shape 12/16, index chunk 1/4.\nBuilding reference seed array... [0.293s]\nBuilding query seed array... [3.456s]\nComputing hash join... [0.791s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.841s]\nProcessing query block 1, reference block 1/1, shape 12/16, index chunk 2/4.\nBuilding reference seed array... [0.356s]\nBuilding query seed array... [4.255s]\nComputing hash join... [0.824s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.832s]\nProcessing query block 1, reference block 1/1, shape 12/16, index chunk 3/4.\nBuilding reference seed array... [0.403s]\nBuilding query seed array... [4.7s]\nComputing hash join... [0.846s]\nBuilding seed filter... [0.067s]\nSearching alignments... [7.746s]\nProcessing query block 1, reference block 1/1, shape 12/16, index chunk 4/4.\nBuilding reference seed array... [0.261s]\nBuilding query seed array... [3.633s]\nComputing hash join... [0.823s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.772s]\nProcessing query block 1, reference block 1/1, shape 13/16, index chunk 1/4.\nBuilding reference seed array... [0.296s]\nBuilding query seed array... [3.54s]\nComputing hash join... [0.812s]\nBuilding seed filter... [0.084s]\nSearching alignments... [7.877s]\nProcessing query block 1, reference block 1/1, shape 13/16, index chunk 2/4.\nBuilding reference seed array... [0.368s]\nBuilding query seed array... [4.183s]\nComputing hash join... [0.802s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.798s]\nProcessing query block 1, reference block 1/1, shape 13/16, index chunk 3/4.\nBuilding reference seed array... [0.387s]\nBuilding query seed array... [4.605s]\nComputing hash join... [0.824s]\nBuilding seed filter... [0.067s]\nSearching alignments... [7.668s]\nProcessing query block 1, reference block 1/1, shape 13/16, index chunk 4/4.\nBuilding reference seed array... [0.289s]\nBuilding query seed array... [3.508s]\nComputing hash join... [0.809s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.912s]\nProcessing query block 1, reference block 1/1, shape 14/16, index chunk 1/4.\nBuilding reference seed array... [0.276s]\nBuilding query seed array... [3.463s]\nComputing hash join... [0.792s]\nBuilding seed filter... [0.074s]\nSearching alignments... [8.041s]\nProcessing query block 1, reference block 1/1, shape 14/16, index chunk 2/4.\nBuilding reference seed array... [0.342s]\nBuilding query seed array... [4.35s]\nComputing hash join... [0.831s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.881s]\nProcessing query block 1, reference block 1/1, shape 14/16, index chunk 3/4.\nBuilding reference seed array... [0.39s]\nBuilding query seed array... [4.816s]\nComputing hash join... [0.853s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.725s]\nProcessing query block 1, reference block 1/1, shape 14/16, index chunk 4/4.\nBuilding reference seed array... [0.264s]\nBuilding query seed array... [3.511s]\nComputing hash join... [0.805s]\nBuilding seed filter... [0.064s]\nSearching alignments... [7.913s]\nProcessing query block 1, reference block 1/1, shape 15/16, index chunk 1/4.\nBuilding reference seed array... [0.332s]\nBuilding query seed array... [3.465s]\nComputing hash join... [0.804s]\nBuilding seed filter... [0.077s]\nSearching alignments... [7.971s]\nProcessing query block 1, reference block 1/1, shape 15/16, index chunk 2/4.\nBuilding reference seed array... [0.363s]\nBuilding query seed array... [4.145s]\nComputing hash join... [0.818s]\nBuilding seed filter... [0.067s]\nSearching alignments... [7.766s]\nProcessing query block 1, reference block 1/1, shape 15/16, index chunk 3/4.\nBuilding reference seed array... [0.398s]\nBuilding query seed array... [4.894s]\nComputing hash join... [0.848s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.655s]\nProcessing query block 1, reference block 1/1, shape 15/16, index chunk 4/4.\nBuilding reference seed array... [0.282s]\nBuilding query seed array... [3.467s]\nComputing hash join... [0.774s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.825s]\nProcessing query block 1, reference block 1/1, shape 16/16, index chunk 1/4.\nBuilding reference seed array... [0.29s]\nBuilding query seed array... [3.552s]\nComputing hash join... [0.813s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.774s]\nProcessing query block 1, reference block 1/1, shape 16/16, index chunk 2/4.\nBuilding reference seed array... [0.363s]\nBuilding query seed array... [4.236s]\nComputing hash join... [0.822s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.861s]\nProcessing query block 1, reference block 1/1, shape 16/16, index chunk 3/4.\nBuilding reference seed array... [0.377s]\nBuilding query seed array... [4.621s]\nComputing hash join... [0.817s]\nBuilding seed filter... [0.065s]\nSearching alignments... [7.654s]\nProcessing query block 1, reference block 1/1, shape 16/16, index chunk 4/4.\nBuilding reference seed array... [0.301s]\nBuilding query seed array... [3.489s]\nComputing hash join... [0.803s]\nBuilding seed filter... [0.066s]\nSearching alignments... [7.975s]\nDeallocating buffers... [0.021s]\nClearing query masking... [0.208s]\nComputing alignments... terminate called after throwing an instance of 'std::bad_alloc'\n what(): std::bad_alloc\n"
Traceback (most recent call last):
File "/home/digitalstorm/miniconda3/bin/funannotate", line 688, in
main()
File "/home/digitalstorm/miniconda3/bin/funannotate", line 678, in main
mod.main(arguments)
File "/home/digitalstorm/miniconda3/lib/python3.7/site-packages/funannotate/predict.py", line 983, in main
lib.exonerate2hints(Exonerate, hintsP)
File "/home/digitalstorm/miniconda3/lib/python3.7/site-packages/funannotate/library.py", line 3877, in exonerate2hints
with open(file, 'r') as input:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/cb2d0a32-df3a-4e6f-8491-e153d9ae19f4/ana/holbrook_annotations/Chromosomes_NEW_TRI/predict_misc/protein_alignments.gff3'
I am running this on an Ubuntu that has 64 cores and 132 RAM. Could you please give me any ideas on how to solve this?
The text was updated successfully, but these errors were encountered: