error : Internally inconsistent attributes formatting: some have repeated keys, some do not.

[CorentinAube]

Hello,

I am currently use IsoQuant to quantify human and viral read by nanopore sequencing.
I made a gtf of my virus that I merge it to the human GTF.

I run IsoQuant with this command line :
python3 isoquant.py --data_type ont --bam barcode01withtrimming_old.bam barcode07withtrimming_old.bam --reference
GRCh38.p13human_genome_primary_asembly_PLUS_IAVcRNAs_seq_corrected.fna --genedb GCF_000001405.39_
GRCh38.p13_genomic_IAV_OMv2.gtf --complete_genedb --output all_data/ --prefix ONT_2samples --labels A1 A2 --report_novel_unspliced true

And here the log:

2023-07-05 16:52:13,056 - INFO - Running IsoQuant version 3.3.0
2023-07-05 16:52:13,057 - WARNING - Output folder already contains a previous run, will be overwritten.
2023-07-05 16:52:13,069 - WARNING - ../Projet_Influenza/all_data/ONT_2samples folder already exists, some files may be overwritten
2023-07-05 16:52:13,069 - WARNING - ../Projet_Influenza/all_data/ONT_2samples/aux folder already exists, some files may be overwritten
2023-07-05 16:52:13,070 - INFO - === IsoQuant pipeline started ===
2023-07-05 16:52:13,072 - INFO - Converting gene annotation file to .db format (takes a while)...
2023-07-05 16:52:13,078 - CRITICAL - IsoQuant failed with the following error, please, submit this issue to https://github.com/ablab/IsoQuant/issuesTraceback (most recent call last):
File "isoquant.py", line 698, in
main(sys.argv[1:])
File "isoquant.py", line 692, in main
run_pipeline(args)
File "isoquant.py", line 631, in run_pipeline
args.genedb = convert_gtf_to_db(args)
File "/mnt/d/Corentin/IsoQuant/src/gtf2db.py", line 110, in convert_gtf_to_db
gtf_filename, genedb_filename = convert_db(gtf_filename, genedb_filename, gtf2db, args)
File "/mnt/d/Corentin/IsoQuant/src/gtf2db.py", line 158, in convert_db
convert_fn(gtf_filename, genedb_filename, args.complete_genedb)
File "/mnt/d/Corentin/IsoQuant/src/gtf2db.py", line 95, in gtf2db
gffutils.create_db(gtf, db, force=True, keep_order=True, merge_strategy='error',
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/create.py", line 1359, in create_db
iterator = iterators.DataIterator(kwargs)
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/iterators.py", line 310, in DataIterator
return _FileIterator(_kwargs)
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/iterators.py", line 89, in init
_peek = self.peek(checklines)
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/iterators.py", line 117, in peek
for i, feature in enumerate(self._custom_iter()):
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/iterators.py", line 154, in _custom_iter
yield feature_from_line(line, dialect=self.dialect)
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/feature.py", line 449, in feature_from_line
attrs, _dialect = parser._split_keyvals(attr_string, dialect=dialect)
File "/home/corentin/.local/lib/python3.8/site-packages/gffutils/parser.py", line 349, in _split_keyvals
raise AttributeStringError(
gffutils.exceptions.AttributeStringError: Internally inconsistent attributes formatting: some have repeated keys, some do not.

Do you think there is a mistake in my annotation file or something else?

Thank you for your help.

Corentin

[andrewprzh]

Dear @CorentinAube

Yes, based on the error, seems like something is not right with the annotation, presumably duplicated IDs between different features (e.g. the same id of a gene and its transcript).
You can send me your GTF for check up.

Best
Andrey

[CorentinAube]

Hello Andrey,

Thank you very much for your answer.

I paste the gtf here because it's a small genome!
CY121204.1 Genbank transcript 28 2307 . + . transcript_id "CDS"; gene_id "CDS"; gene_name "PB2";
CY121204.1 Genbank CDS 28 2307 . + 0 transcript_id "CDS"; gene_name "PB2";
CY121203.1 Genbank transcript 25 2298 . + . transcript_id "gene-PB1-2"; gene_id "gene-PB1-2"; gene_name "PB1";
CY121203.1 Genbank CDS 25 2298 . + 0 transcript_id "gene-PB1-2"; gene_name "PB1";
CY121203.1 Genbank transcript 119 391 . + . transcript_id "gene-PB1-F2-2"; gene_id "gene-PB1-F2-2"; gene_name "PB1-F2";
CY121203.1 Genbank CDS 119 391 . + 0 transcript_id "gene-PB1-F2-2"; gene_name "PB1-F2";
CY121202.1 Genbank transcript 25 2175 . + . transcript_id "gene-PA"; gene_id "gene-PA"; gene_name "PA";
CY121202.1 Genbank CDS 25 2175 . + 0 transcript_id "gene-PA"; gene_id "gene-PA"; gene_name "PA";
CY121201.1 Genbank transcript 27 686 . + . transcript_id "gene-NS1"; gene_id "gene-NS1"; gene_name "NS1";
CY121201.1 Genbank CDS 27 686 . + 0 transcript_id "gene-NS1"; gene_id "gene-NS1"; gene_name "NS1";
CY121201.1 Genbank transcript 27 864 . + . transcript_id "gene-NEP"; gene_id "gene-NEP"; gene_name "NEP";
CY121201.1 Genbank CDS 27 56 . + 0 transcript_id "gene-NEP"; gene_id "gene-NEP"; gene_name "NEP";
CY121201.1 Genbank CDS 529 864 . + 0 transcript_id "gene-NEP"; gene_id "gene-NEP"; gene_name "NEP";
CY121200.1 Genbank transcript 46 1542 . + . transcript_id "gene-NP"; gene_id "gene-NP"; gene_name "NP";
CY121200.1 Genbank CDS 46 1542 . + 0 transcript_id "gene-NP"; gene_id "gene-NP"; gene_name "NP";
CY121199.1 Genbank transcript 20 1429 . + . transcript_id "gene-NA"; gene_id "gene-NA"; gene_name "NA";
CY121199.1 Genbank CDS 20 1429 . + 0 transcript_id "gene-NA"; gene_id "gene-NA"; gene_name "NA";
CY121198.1 Genbank transcript 26 784 . + . transcript_id "gene-M1"; gene_id "gene-M1"; gene_name "M1";
CY121198.1 Genbank CDS 26 784 . + 0 transcript_id "gene-M1"; gene_id "gene-M1"; gene_name "M1";
CY121198.1 Genbank transcript 26 1007 . + . transcript_id "gene-M2"; gene_id "gene-M2"; gene_name "M2";
CY121198.1 Genbank CDS 26 51 . + 0 transcript_id "gene-M2"; gene_id "gene-M2"; gene_name "M2";
CY121198.1 Genbank CDS 740 1007 . + 1 transcript_id "gene-M2"; gene_id "gene-M2"; gene_name "M2";
CY121198.1 Genbank transcript 26 1007 . + . transcript_id "gene-M3"; gene_id "gene-M3"; gene_name "M3";
CY121198.1 Genbank CDS 26 51 . + 0 transcript_id "gene-M3"; gene_id "gene-M3"; gene_name "M3";
CY121198.1 Genbank CDS 740 1007 . + 1 transcript_id "gene-M3"; gene_id "gene-M3"; gene_name "M3";
CY121197.1 Genbank transcript 30 1733 . + . transcript_id "gene-HA"; gene_id "gene-HA"; gene_name "HA";
CY121197.1 Genbank CDS 30 1733 . + 0 transcript_id "gene-HA"; gene_id "gene-HA"; gene_name "HA";

Maybe be the mistake is link to CY121198.1 and CY121201.1 genes because they had differents transcripts for a same gene.

Best,
Corentin

[andrewprzh]

Dear @CorentinAube

Yes, lines like these confuse gffutils

Genbank transcript 27 686 . + . transcript_id "gene-NS1"; gene_id "gene-NS1"; gene_name "NS1";

since gene_id and transcript_id are the same. You can rename them so that transcript id is, for example, gene-NS1.1.

Best
Andrey

[CorentinAube]

Thank you for your answer.

I fixed my gtf but the problem is still going on.
I tested to run IsoQuant just with my human GTF but the problem is still there. So i think my human GTF is not good.

I also tested with only my viral GTF, and it's work until this error :

2023-07-06 14:42:20,536 - CRITICAL - IsoQuant failed with the following error, please, submit this issue to https://github.com/ablab/IsoQuant/issuesconcurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/lib/python3.8/concurrent/futures/process.py", line 198, in _process_chunk
return [fn(*args) for args in chunk]
File "/usr/lib/python3.8/concurrent/futures/process.py", line 198, in
return [fn(*args) for args in chunk]
File "/mnt/d/Corentin/IsoQuant/src/dataset_processor.py", line 109, in collect_reads_in_parallel
AlignmentCollector(chr_id, bam_file_pairs, args, gffutils_db, current_chr_record, read_grouper)
File "/mnt/d/Corentin/IsoQuant/src/alignment_processor.py", line 227, in init
self.bam_merger = BAMOnlineMerger(self.bam_pairs, self.chr_id, 0,
File "/mnt/d/Corentin/IsoQuant/src/alignment_processor.py", line 41, in init
self._set(chr_id, start, end)
File "/mnt/d/Corentin/IsoQuant/src/alignment_processor.py", line 53, in _set
self.current_elements.put_nowait(make_alignment_tuple(i, next(it)))
File "pysam/libcalignmentfile.pyx", line 2107, in pysam.libcalignmentfile.IteratorRowRegion.next
OSError: truncated file
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "isoquant.py", line 698, in
main(sys.argv[1:])
File "isoquant.py", line 692, in main
run_pipeline(args)
File "isoquant.py", line 645, in run_pipeline
dataset_processor.process_all_samples(args.input_data)
File "/mnt/d/Corentin/IsoQuant/src/dataset_processor.py", line 368, in process_all_samples
self.process_sample(sample)
File "/mnt/d/Corentin/IsoQuant/src/dataset_processor.py", line 392, in process_sample
self.collect_reads(sample)
File "/mnt/d/Corentin/IsoQuant/src/dataset_processor.py", line 449, in collect_reads
for storage, read_groups in results:
File "/usr/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
for element in iterable:
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
OSError: truncated file

Thank you for your feed back!
Corentin

[andrewprzh]

pysam.libcalignmentfile.IteratorRowRegion.next
OSError: truncated file

Seems like something is wrong with the BAM files, they look truncated.

Best
Andrey

[CorentinAube]

Thank you, I will look at this.

Best,
Corentin

[CorentinAube]

Hello Andrey,

All is working! Thank you for your feedbacks.

Best,
Corentin

[soniacruciani]

Hello Andrey,
I had the same OSError: truncated file error but the bam file is fine (checked with samtools quickcheck) and it had already worked fine in a previous isoquant job. I solved it so I will leave it here if it can be of any use to anyone:
the bam file was just a softlink, but I regenerated the .bai file from scratch and then it didn't give error. (the previous .bai had worked with the other job so I still don't know what happened but I leave it here for someone else to try in case they have problems).

Thanks,

Sonia

[andrewprzh]

Dear @soniacruciani

Seems like an IO error if the BAM files are in tact. Does not sound like an IsoQuant problem.
Could you send me isoquant.log and alignment.log files, preferably in a separate issue as this one is closed.

Best
Andrey

[CorentinAube]

Hello Andrey, I had the same OSError: truncated file error but the bam file is fine (checked with samtools quickcheck) and it had already worked fine in a previous isoquant job. I solved it so I will leave it here if it can be of any use to anyone: the bam file was just a softlink, but I regenerated the .bai file from scratch and then it didn't give error. (the previous .bai had worked with the other job so I still don't know what happened but I leave it here for someone else to try in case they have problems).

Thanks,

Sonia

Hello Sonia,

From what I remember, the error was due to my method of converting my BAM files to SAM.

It works when I use a pipe between Minimap and Samtools rather than separating these two tools into two different lines:
minimap [option] refseq.fa file.fasta | samtools -sort file.bam -o file.sam

I don't know if it's could help you,
best,
Corentin