shortstack4.0.3 ValueError: invalid literal for int() with base 10: ''

[joweihsieh]

Hi Dr. Axtell,

Thanks for developing this useful software!! I've been attempting to utilize shortstack4.0.3 for both known and de novo miRNA prediction for wheat sRNA data. However, I encountered a ValueError: invalid literal for int() with base 10: ''.

Btw, I am able to run miRNA prediction for the Arabidopsis toy example on shortstack4.0.3, and my wheat sRNA datasets performed well on shortstack3.8.5.

Could you please help me in resolving this issue?

Thank you,
Jo-Wei

======= Below are my output
Beginning run
Options:
{ 'adapter': None,
'align_only': False,
'autotrim': False,
'autotrim_key': 'TCGGACCAGGCTTCATTCCCC',
'bamfile': [ '/home/jhsieh/processed_results/202403_TWG/shortstackv4.0.3/20240424_chrABD/goodDist/merged.bam'],
'dicermax': 24,
'dicermin': 20,
'dn_mirna': True,
'genomefile': '/home/jhsieh/genome/Triticum_aestivum/modifyfromlab/IWGSC2/v1/GenomicSEQ/Original/fasta/iwgsc_refseqv2.1_assembly.fa',
'known_miRNAs': 'tae_known_miRNAs.fasta',
'locifile': None,
'locus': None,
'mincov': 1.0,
'mmap': 'u',
'nohp': False,
'outdir': '/home/jhsieh/processed_results/202403_TWG/shortstackv4.0.3/20240424_chrABD/goodDist_miR3',
'pad': 75,
'readfile': None,
'show_secondaries': False,
'strand_cutoff': 0.8,
'threads': 20}
Required executable RNAfold : /home/jhsieh/miniconda3/envs/ShortStack4/bin/RNAfold
Required executable strucVis : /home/jhsieh/miniconda3/envs/ShortStack4/bin/strucVis
Required executable bowtie : /usr/local/bowtie-1.3.1-linux-x86_64/bowtie
Required executable bowtie-build : /usr/local/bowtie-1.3.1-linux-x86_64/bowtie-build
Required executable samtools : /usr/local/samtools-1.11/bin/samtools

Mon 06 May 2024 12:44:21 -0500 CDT
Defining small RNA clusters de novo
With 411496348 total reads and mincov of 1.0 reads per million, the min read depth is 411

Mon 06 May 2024 13:23:03 -0500 CDT
Analyzing cluster properties using 20 threads

reads processed: 125

reads with at least one alignment: 119 (95.20%)

reads that failed to align: 6 (4.80%)

Reported 43603 alignments
[bam_sort_core] merging from 0 files and 20 in-memory blocks...

Mon 06 May 2024 16:56:59 -0500 CDT
Completed

Mon 06 May 2024 16:56:59 -0500 CDT
Searching for valid microRNA loci
Aligning known_miRNAs sequences to genome

Screening of possible microRNAs from user provided known_miRNAs
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2359, in mir_analysis
q_exact_count = count_exact_location(bedfields, merged_bam)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2684, in count_exact_location
read_count = int(mb.stdout.rstrip())
^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 3619, in
mir_qdata = mirna(args, merged_bam, fai, pmir_bedfile, read_count)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2187, in mirna
user_mloci1 = pool.starmap(mir_analysis,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: invalid literal for int() with base 10: ''

[MikeAxtell]

I have not encountered this error before nor has it been reported (that I can recall). It looks a failure of a simple sub.process call to samtools, which is used to count aligned reads from a specific location. I can't reproduce the error here without your exact original data. Can you send me your data? You'll need to post them on a cloud location somewhere because of file sizes. I need all input files :

• 'bamfile': [ '/home/jhsieh/processed_results/202403_TWG/shortstackv4.0.3/20240424_chrABD/goodDist/merged.bam']
• 'genomefile': '/home/jhsieh/genome/Triticum_aestivum/modifyfromlab/IWGSC2/v1/GenomicSEQ/Original/fasta/iwgsc_refseqv2.1_assembly.fa'
• 'known_miRNAs': 'tae_known_miRNAs.fasta'

If you can get those data to me I can try and reproduce the error and go from there.

[MikeAxtell]

Try commit 35d16a1 and see if your error goes away. I think I might have guessed at the issue, but I don't have any datasets here that are reproducing your error so I can't be sure.

[joweihsieh]

Dear Dr. Axtell,

Thank you for your prompt response and assistance! I will also try commit 35d16a1 later today to see if the errors go away!

Here is the Dropbox link with all the required input files:
https://www.dropbox.com/scl/fo/g4pzi3txipxt5tg7eor3d/ACVt6NH9jtV3p1gtIbP5XOM?rlkey=nosuq7o1v6mnnkuljedhsuwuw&dl=0

I have sent an invitation for folder access to your email (mja18@psu.edu).
Please let me know if you encounter any issues accessing the folder, and also once you have finished downloading the files.

Thank you,
Jo-Wei

[MikeAxtell]

Thanks for posting the data. Unfortunately, your merged.bam file is very large , 189Gb. This is proving difficult to download using the Dropbox web interface. I've tried three times and had failed downloads or corrupted data each time.

I did not see any obvious issue with the genome file or the known_miRNAs file.

Can you instead test the 35d16a1 commit on your end with your data and report if the issue was solved? That commit or any commit later than that and I think the issue may have been solved.

[joweihsieh]

Hi Dr. Axtell,

Thanks for checking the miRNA fasta and genome fasta file for me.

Following your suggestions, I've revised the script accordingly. I am testing it now, but it may take some time due to the large size of the bam file.

I will get back to you once I have the results.

Thanks again,
Jo-Wei

[joweihsieh]

Hi Dr. Axtell,

It still has the same errors.

=========
Screening of possible microRNAs from user provided known_miRNAs
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2359, in mir_analysis
q_exact_count = count_exact_location(bedfields, merged_bam, fai)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2690, in count_exact_location
read_count = int(mb.stdout.rstrip())
^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 3625, in
mir_qdata = mirna(args, merged_bam, fai, pmir_bedfile, read_count)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/bin/ShortStack", line 2187, in mirna
user_mloci1 = pool.starmap(mir_analysis,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jhsieh/miniconda3/envs/ShortStack4/lib/python3.12/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: invalid literal for int() with base 10: ''

=========
I further have these tests for the original script:

• Run 4 fastq files: Successful
• Run 4 individual bam files: Failed with same ValueError

It seems that the ValueError is caused by my bam file.

Jo-Wei

[MikeAxtell]

Yes,
I'm working on this. I believe I have identified the issue but need to complete a few more tests ...

[MikeAxtell]

This issue is that parts of ShortStack's microRNA annotation process are not properly dealing with secondary alignments. Your original large BAM file has a lot of secondary alignments (bit 256 set in the FLAG column). If I filter your BAM file to remove all secondary alignments, I get a much smaller file that only contains primary alignments:

samtools view -b -F 256 merged.bam > nosec.bam

When I use that file with ShortStack there were no bugs:

ShortStack --genomefile iwgsc_refseqv2.1_assembly.fa --bamfile nosec.bam --known_miRNAs tae_known_miRNAs.fasta --outdir RelRun1NoSec --threads 10 --dn_mirna

ShortStack's default alignment method does not output secondary alignments, so it seems you made your sRNA-seq alignments with a different program. Nonetheless, ShortStack should be able to handle secondary alignments in a BAM file. I am working through fixing that bug now.

In the meantime, you can use samtools view -b -F 256 merged.bam > nosec.bam and then use the nosec.bam file as input to ShortStack.

Thanks again for reporting the bug.

[MikeAxtell]

OK, commit a70cd65 fixes this issue. Bamfiles with secondary alignments will now work.

This will be included in the next release, version 4.0.4, coming very soon.

[joweihsieh]

Hi Dr. Axtell,

Wow, thank you, that's awesome! I'll try it right away. Thanks for your help!

Using reads without secondary alignments doesn't seem ideal to me, especially considering the presence of multiple subgenomes in my species. This approach would result in a lot of missing information; signals from some subgenomes might not be successfully counted, leading to the absence of miRNA predictions from those subgenomes.

Anyway, many thanks for fixing this!!! :)

Jo-Wei

[MikeAxtell]

To be clear, ShortStack will *always* ignore secondary alignments. Each read has a single primary alignment and that is the only one that is analyzed. When there are multiple reads with identical sequences and that sequence has more than one valid alignment position, depending on the alignment method used, some reads will have a primary at position A, some at position B, and so on. So you don't need secondary alignments to account for multi-mapping reads. From: joweihsieh ***@***.***> Date: Tuesday, May 14, 2024 at 1:55 PM To: MikeAxtell/ShortStack ***@***.***> Cc: Axtell, Michael ***@***.***>, Assign ***@***.***> Subject: Re: [MikeAxtell/ShortStack] shortstack4.0.3 ValueError: invalid literal for int() with base 10: '' (Issue #151) Hi Dr. Axtell, Wow, thank you, that's awesome! I'll try it right away. Thanks for your help! Using reads without secondary alignments doesn't seem ideal to me, especially considering the presence of multiple subgenomes in my species. This approach would result in a lot of missing information; signals from some subgenomes might not be successfully counted, leading to the absence of miRNA predictions from those subgenomes. Anyway, many thanks for fixing this!!! :) Jo-Wei — Reply to this email directly, view it on GitHub<#151 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUJPCJBJJ474IHVDRC7CQDZCJFXRAVCNFSM6AAAAABHJ44K56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJQHAYDKMRRGE>. You are receiving this because you were assigned.Message ID: ***@***.***>

[joweihsieh]

Hi Dr. Axtell,

Thanks for the clarification and the release of shortstack4.0.4.
I am able to run my data using shortstack4.0.4.

Many thanks,
Jo-Wei