Additional ncbi genbank records which fail parsing gbk -> tbl

[hyphaltip]

These are ones I am trying to run funannotate annotate on in order to use funannotate compare

Both I believe are JGI genomes.
Sporidiobolus salmonicolor - CENE01000001 - ftp://ftp.ncbi.nlm.nih.gov/sra/wgs_aux/CE/NE/CENE01/CENE01.1.gbff.gz

Rhodotorula toruloides NP11 - NW_015622961
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/320/785/GCF_000320785.1_RHOziaDV1.0/GCF_000320785.1_RHOziaDV1.0_genomic.gbff.gz

[nextgenusfs]

The first one doesn't have locus_tag annotations, instead seem to use gene tags-- I'm not sure how JGI is getting these to into GenBank, its almost as if they are bypassing QC....

     gene            7248..9101
                     /gene="SPOSA6832_00014"
     mRNA            7248..9101
                     /gene="SPOSA6832_00014"
                     /standard_name="SPOSA6832_00014-mRNA-1"
     CDS             join(7248..7443,7544..7739,7837..8035,8108..8386,
                     8453..8617,8679..9101)
                     /gene="SPOSA6832_00014"
                     /standard_name="SPOSA6832_00014-mRNA-1:cds"
                     /codon_start=1
                     /protein_id="CEQ38564.1"
                     /db_xref="GI:814543670"
                     /translation="MSIIADVGQPVASTSASTLAGEEKQIAIRLSTKDAAYQIPPAKF
                     LVPASWRRFHLSELINKVLENDSPIPFDFLIDQTLLRSSLGAYCAASGTSEEVVLEVE
                     FLPSTLPPQLESTLPSEDWVSDVSVSVRGAILTASYAGTLSLQSSALPPPSTLTFAGH
                     DLSVLSATYVPHPLGSEDKHWIASGGLDRVGRVWEYTIPPLSLTAETAPPAPTTIHTL
                     SLHQAPISSVRSRPLPLSSTTPTSSPHLLTASWDGLIGVWDLTPGVNEGDADLEGADR
                     KKKRRRQSTTVVNKSPVSVLRGHTNKVSRAVFDRNDVTKAYSAGWDHTVRSWDLSIGA
                     ESSSKVCTHSLTSDKVLLDLAQLAAPNLLATGSTDRLISLWDLRSDATQNISLTLSGH
                     TAPVSSIAAHPTSSLLFASGSYDSTVRIWDARSAKQALFTLELPEREGAEPRKEKEKV
                     LAVDWDGERLVAGGEGAQVVVWRVSGQESKAPAAE"
     exon            join(7248..7443,7544..7739,7837..8035,8108..8386,
                     8453..8617,8679..9101)
                     /gene="SPOSA6832_00014"
                     /standard_name="SPOSA6832_00014-mRNA-1:exon"

[nextgenusfs]

Its not only the locus_tag, but also this exon feature isn't standard either, that should be in the mRNA feature. I saw one yesterday that was missing mRNA features entirely and instead had exon and intron features.

[nextgenusfs]

I modified the code to try to deal with these better, and there are more errors, i.e. this seems wrong there is a second CDS for SPOSA6832_03700 yet not a second exon nor mRNA. I don't know how to deal with these seemingly mis-named records -- the error you get is typically an "index is out of range" which basically means that somewhere there is a mistake. In this example, multiple CDSs get added for SPOSA6832_03700 but there is only a single mRNA, so when the script tries to output the multiple loci and doesn't find multiple mRNA coordinates it crashes. Not sure if they should just try to be ignored?

     gene            complement(<82321..>83180)
                     /gene="SPOSA6832_03700"
     mRNA            complement(82321..83180)
                     /gene="SPOSA6832_03700"
                     /standard_name="SPOSA6832_03700-mRNA-1"
     CDS             complement(join(<82321..82399,82554..82841,82904..>83022))
                     /gene="SPOSA6832_03700"
                     /standard_name="SPOSA6832_03700-mRNA-1:cds"
                     /codon_start=1
                     /protein_id="CEQ41952.1"
                     /db_xref="GI:814540666"
                     /translation="SNTLRPYLACVRASLTAALTLESFPSQVAERHNVAEIEAQSTPE
                     VLLTPLVIARNASEQVLIEPSINSVRLSIKIKQADDLERILAHKFTRFLMMRAEAFVV
                     LRRVPVKGYDISFLVTNRNVEDMMKHKLVDFVIDGRQGDLGGQTLDERTSTHRRRVLP
                     VH"
     exon            complement(join(82321..82399,82554..82841,82904..83022))
                     /gene="SPOSA6832_03700"
                     /standard_name="SPOSA6832_03700-mRNA-1:exon"
     gene            83043..84074
                     /gene="SPOSA6832_03689"
     mRNA            83043..84074
                     /gene="SPOSA6832_03689"
                     /standard_name="SPOSA6832_03689-mRNA-1"
     CDS             join(83043..83156,83307..83507,83576..83734,83805..84074)
                     /gene="SPOSA6832_03689"
                     /standard_name="SPOSA6832_03689-mRNA-1:cds"
                     /codon_start=1
                     /protein_id="CEQ41953.1"
                     /db_xref="GI:814540667"
                     /translation="MGMACRGAARAYHDCRRWSRDEEEEQRQSSLDLVRISHTLNERA
                     GFLCNHEVLQILRQQRDDRAARIKVLDEQRRKSRNEGTLDSLMADEKERVQPQDLHTV
                     TFEAIRYLEDDVHSIRRQTPKSVATLLDKLEDLHLTKAERLHIVNLAPTSLVELHVCI
                     EELGERFSEPQQLELIQMVKDHLGKLDQATTRDGDAAHEDRMDEDRDELDEREAAALH
                     EEDELIDEGLGGGGAQEDAVNDIDEVEES"
     exon            join(83043..83156,83307..83507,83576..83734,83805..84074)
                     /gene="SPOSA6832_03689"
                     /standard_name="SPOSA6832_03689-mRNA-1:exon"
     CDS             complement(join(<83043..83156,83079..>83180))
                     /gene="SPOSA6832_03700"
                     /standard_name="SPOSA6832_03700-mRNA-1:cds"
                     /codon_start=1
                     /protein_id="CEQ41954.1"
                     /db_xref="GI:814540668"
                     /translation="LVPSERAHVRDADEVKAALALLLLLIPRPSPAVMVRDADEVKAA
                     LALLLLLIPRPSPAVMVRSRCASASHAH"

[nextgenusfs]

Okay, more updates. I got the first problem "fixed" by alerting user there is a problem with the annotation:

$ funannotate annotate --genbank CENE01.1.gbff -o test4 --cpus 6
-------------------------------------------------------
[Jul 08 05:10 PM]: OS: MacOSX 10.13.5, 8 cores, ~ 17 GB RAM. Python: 2.7.11
[Jul 08 05:10 PM]: Running funannotate v1.4.0-df1972c
[Jul 08 05:10 PM]: No NCBI SBT file given, will use default, however if you plan to submit to NCBI, create one and pass it here '--sbt'
[Jul 08 05:10 PM]: Output directory test4 already exists, will use any existing data.  If this is not what you want, exit, and provide a unique name for output folder
[Jul 08 05:10 PM]: Checking GenBank file for annotation
[Jul 08 05:11 PM]: Incompatible annotation found: SPOSA6832_03700
{'product': ['hypothetical protein', 'hypothetical protein'], 'name': 'SPOSA6832_03700', 'db_xref': [['GI:814540666'], ['GI:814540668']], 'go_terms': [[], []], 'transcript': [Seq('TCCAATACACTCCGCCCGTACCTCGCTTGCGTTCGAGCTTCGCTCACCGCCGCC...CAC', IUPACAmbiguousDNA())], 'codon_start': [1, 1], 'ids': ['SPOSA6832_03700-T1', 'SPOSA6832_03700-T2'], 'protein_id': ['CEQ41952.1', 'CEQ41954.1'], 'note': [[], []], 'source': 'GenBank', 'cds_transcript': [Seq('TCCAATACACTCCGCCCGTACCTCGCTTGCGTTCGAGCTTCGCTCACCGCCGCC...CAC', IUPACAmbiguousDNA()), Seq('CTCGTTCCGAGTGAACGAGCGCACGTGAGAGATGCGGACGAGGTCAAGGCTGCT...CAT', IUPACAmbiguousDNA())], 'location': (82321, 83180), 'mRNA': [[(82904, 83022), (82554, 82841), (82321, 82399)]], 'CDS': [[(82904, 83022), (82554, 82841), (82321, 82399)], [(83079, 83180), (83043, 83156)]], 'partialStop': [False, False], 'protein': ['SNTLRPYLACVRASLTAALTLESFPSQVAERHNVAEIEAQSTPEVLLTPLVIARNASEQVLIEPSINSVRLSIKIKQADDLERILAHKFTRFLMMRAEAFVVLRRVPVKGYDISFLVTNRNVEDMMKHKLVDFVIDGRQGDLGGQTLDERTSTHRRRVLPVH', 'LVPSERAHVRDADEVKAALALLLLLIPRPSPAVMVRDADEVKAALALLLLLIPRPSPAVMVRSRCASASHAH'], 'type': 'mRNA', 'contig': 'CENE01000019.1', 'strand': '-', 'partialStart': [False, False]}
[Jul 08 05:11 PM]: Inspect input file and fix errors manually before proceeding.

It outputs the offending locus and quits. For this Sporidiobolus salmonicolor annotation, if you manually delete that "rogue" second CDS everything else is fine.

The second genome Rhodotorula toruloides NP11 has pseudo annotations on several genes, meaning they don't have CDS. So built a second check to make sure there is a CDS if it is of type == 'mRNA' which means coding gene.

Here's what GBK looks like:

     gene            complement(<470622..>471683)
                     /locus_tag="RHTO_02866"
                     /note="nuclear pore complex protein Nup133; frameshift"
                     /pseudo
                     /db_xref="GeneID:27366879"
     mRNA            complement(join(<470622..470669,470720..470922,
                     471096..471218,471277..471376,471439..471530,
                     471585..>471683))
                     /locus_tag="RHTO_02866"
                     /product="nuclear pore complex protein Nup133"
                     /pseudo
                     /db_xref="GeneID:27366879"
     gene            <474041..>475136
                     /locus_tag="RHTO_02867"
                     /db_xref="GeneID:27366880"
     mRNA            join(<474041..474054,474139..474251,474317..474890,
                     474974..>475136)
                     /locus_tag="RHTO_02867"
                     /product="hypothetical protein"
                     /transcript_id="XM_016416539.1"
                     /db_xref="GeneID:27366880"
     CDS             join(474041..474054,474139..474251,474317..474890,
                     474974..475136)
                     /locus_tag="RHTO_02867"
                     /codon_start=1
                     /product="hypothetical protein"
                     /protein_id="XP_016276258.1"
                     /db_xref="GeneID:27366880"
                     /translation="MCTTRKITTCVHLRVPPNTPASGCVFTTGTASPMSPTSPVVENV
                     QLRTATPVHLQRRQLAALNNPDDDDSASSSSSSESDRDSAFHSSPYTYQSTHRTTTTS
                     SHPHRHHSSSLYPPTISAIPPYLADHSRTSIDAGPSSLSVPHAPDSSRRPSGSASLYN
                     GASASSAAYDRIGPGYGSRVTSSKLRSVSTPVRQHPTTASYQPPAPAFNPNDPLRLPT
                     LGTSTVSGGDASFIQEARLVVHPPAKMDGSAMVAGMHGMQRSFDAFKLDTRFAAHKVG
                     KKISRKLNSFL"

Funannotate will now skip those annotations (writing the error to the log file) - as these are going to cause problems.

[07/08/18 17:12:48]: OS: MacOSX 10.13.5, 8 cores, ~ 17 GB RAM. Python: 2.7.11
[07/08/18 17:12:48]: Running funannotate v1.4.0-df1972c
[07/08/18 17:12:48]: No NCBI SBT file given, will use default, however if you plan to submit to NCBI, create one and pass it here '--sbt'
[07/08/18 17:12:48]: Output directory test4 already exists, will use any existing data.  If this is not what you want, exit, and provide a unique name for output folder
[07/08/18 17:12:48]: Checking GenBank file for annotation
[07/08/18 17:12:56]: Skipping RHTO_03159 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_04621 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_04851 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_07846 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01125 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01319 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01528 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01733 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01738 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_01982 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_02229 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_02363 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_02381 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_02489 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_03461 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_03642 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_03916 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_04322 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_04373 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_04558 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_05121 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_05440 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_06623 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_06681 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_07064 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_07390 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_07392 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_07930 because no CDS found.
[07/08/18 17:12:56]: Skipping RHTO_08171 because no CDS found.
[07/08/18 17:12:57]: Adding Functional Annotation to Rhodotorula toruloides NP11, NCBI accession: WGS:ALAU
[07/08/18 17:12:57]: Annotation consists of: 8,170 gene models
[07/08/18 17:12:57]: 8,140 protein records loaded
[07/08/18 17:12:58]: Running HMMer search of PFAM version 31.0

[nextgenusfs]

Let me know if you find others that fail with this updated code.

[hyphaltip]

Yeah _ i fixed locus tag post-hoc but then hit the second error you mention. terrible... you'd think with all the hoops have to jump through to get annotations in on our end!

[hyphaltip]

updated code seems to be helping, fixing there are redundant mRNA features in this strain of Rhodotorula toruloides CECT1137
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/234/015/GCA_003234015.1_ASM323401v1/GCA_003234015.1_ASM323401v1_genomic.gbff.gz
https://www.ncbi.nlm.nih.gov/genome/13280?genome_assembly_id=383064

manual fixing to remove these is painful for now so seeing what is there. Easier to reannotate!

[nextgenusfs]

seems like this one is a different problem.... let me see if I can figure it out.

[hyphaltip]

It’s duplicated mRNA features for some loci. Who did the qc on this load?! Jason Stajich, PhD jasonstajich.phd@gmail.com

…

On Jul 8, 2018, 4:49 PM -0700, Jon Palmer ***@***.***>, wrote: seems like this one is a different problem.... let me see if I can figure it out. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nextgenusfs]

Perhaps need to parse the /pseudo flag -- that is where the locustag/ID errors are. Probably safe to ignore the annotation marked with pseudo, I think that those are meant for frameshift "genes" -- i.e. best alignment to some curated hit -- probably meant for well studied/curated genes. Seems to be overused in this particular annotation as there are 472 pseudo genes.

And then yes, duplicated mRNA features -- but they seem to have unique names so scripts think they are unique...

I can make it so those get ignored, write the models to logfile, and perhaps alert user there are X number of models have been skipped. But yes, probably better to reannotate as who knows what other problems there are?

[nextgenusfs]

So these specific cases seem to be resolved, re-open when you see other problems...