Stuck at Step 4/24: Align reads and segment them into readlets

[undiagnosed]

I am running locally and installed via the installation command at http://rail.bio. I ran with the following command:

rail-rna go local -x /path/Homo_sapiens/UCSC/hg38/Sequence/BowtieIndex/genome /path/Homo_sapiens/UCSC/hg38/Sequence/Bowtie2Index/genome -m run.manifest -o railrna -d tsv,bw,jx

It proceeds until Step 4/24 and then just doesn't do anything. There were no error messages and no CPU utilization after ~10 hrs so I stopped it. Is there any log or anywhere I can look to figure out what is happening?

The manifest file has the following content:

/path/R1.fastq.gz	0	/path/R2.fastq.gz	0	9_27_rna

[dfermin]

I have a similar problem. When I run it I get stuck at 4/24 as well but it eventually crashes.
Did you ever solve your problem?

[dfermin]

I continue to have this issue at Step 4/24.
It doesn't seem to matter what FASTQ files I give it, rail-rna always crashes at the same step:

00h:00m:00s |___| Step 1/24: Count lines in input files
00h:01m:26s |___|     Completed 8 tasks.
00h:01m:26s |___|     Deleted temporary files.
00h:01m:26s |___| Step 2/24: Assign reads to preprocessing tasks
00h:01m:28s |___|     Partitioned 8 inputs into tasks.
00h:01m:28s |___|     Completed 1 task.
00h:01m:28s |___|     Deleted temporary files.
00h:01m:28s |___| Step 3/24: Preprocess reads
00h:15m:34s |___|     Completed 20 tasks.
00h:15m:34s |___|     Deleted temporary files.
00h:15m:34s |___| Step 4/24: Align reads and segment them into readlets
00h:40m:28s |___|     Partitioned 20 inputs into tasks.
*****Errors encountered*****
Streaming command "LC_ALL=C sort -S 307200 -k1,1 -t$'\t' -m /nfs/data1/project100/2019-Oct-5/bin/rail-rna_logs/align_reads/dp.tasks/14.* | /usr/local/apps/rail-rna/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/apps/rail-rna/rail-rna/rna/steps/align_reads.py
--bowtie-idx=/nfs/data1/genomes/iGenomes/Homo_sapiens/UCSC/hg38/Sequence/BowtieIndex/genome --bowtie2-idx=/nfs/data1/genomes/iGenomes/Homo_sapiens/UCSC/hg38/Sequence/Bowtie2Index/genome --bowtie2-exe=/usr/local/apps/rail-rna/bowtie2-2.2.7/bowtie2 --exon-differentials --partition-length=5000 --min-exon-size=9 --search-filter=1 --manifest=/nfs/data1/project100/2019-Oct-5/bin/sample.manifest.down --max-readlet-size=25 --readlet-interval=4 --capping-multiplier=1.1 --gzip-level 3 --index-count 800 --tie-margin 0  --verbose   --output-bam-by
-chr  --no-polyA -- 2>/nfs/data1/project100/2019-Oct-5/bin/rail-rna_logs/align_reads/dp.reduce.log/14.0.log" failed; exit level was 2.
Job flow failed on Wednesday, Oct 09, 2019 at 11:04:33 AM EDT. Run time was 2457.110 seconds.
To start this job flow from where it left off, run:
/usr/local/apps/rail-rna/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/apps/rail-rna/rail-rna/dooplicity/emr_simulator.py -j /nfs/data1/project100/2019-Oct-5/bin/rail-rna_logs/resume_flow_IX05MW50AQVK.json -b /usr/local/apps/rail-rna/rail-rna/rna/driv
er/rail-rna.txt -l /nfs/data1/project100/2019-Oct-5/bin/rail-rna_logs/flow.2019-10-09T10:23:36.539729.log -f --max-attempts 1 --num-processes 20
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/usr/local/apps/rail-rna/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/usr/local/apps/rail-rna/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
    max_attempts=max_attempts
  File "/usr/local/apps/rail-rna/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError

This is on my local machine not using AWS.
I'm running RHEL 7.5 and using python2.7.
Any help would be appreciated.

[muhammed-ali]

I am encountering the same problem. No concrete error message that we could sort out or at least look up for a solution on the internet!
Quick help in this regard will be much appreciated!

00h:48m:05s |___| Step 21/24: Write bigWigs with exome coverage by sample
00h:48m:05s |___| Step 21/24: Write bigWigs with exome coverage by sample
00h:48m:48s |___|     Partitioned 111 inputs into tasks.
00h:48m:48s |___|     Partitioned 111 inputs into tasks.
*****Errors encountered*****
Streaming command "LC_ALL=C sort -S 307200 -k1,1 -k2,3 -t$'\t' -m /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/dp.tasks/1.* | /home/users/mali/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /mnt/irisgpfs/users/mali/raildotbio/rail-rna/rna/steps/coverage.py --bowtie-idx=/scratch/users/mali/Indices/Homo_sapiens_NCBI_GRCh38/NCBI/GRCh38/Sequence/BowtieIndex/genome --percentile=0.75 --out=/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_out/coverage_bigwigs --bigwig-exe=/home/users/mali/raildotbio/bedGraphToBigWig --manifest=/scratch/users/mali/Tasks/Rail_RNA/manifest_files/Adipo2.manifest >/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/1 2>/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/dp.reduce.log/1.0.log" failed; exit level was 1.
*****Errors encountered*****
Streaming command "LC_ALL=C sort -S 307200 -k1,1 -k2,3 -t$'\t' -m /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/dp.tasks/1.* | /home/users/mali/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /mnt/irisgpfs/users/mali/raildotbio/rail-rna/rna/steps/coverage.py --bowtie-idx=/scratch/users/mali/Indices/Homo_sapiens_NCBI_GRCh38/NCBI/GRCh38/Sequence/BowtieIndex/genome --percentile=0.75 --out=/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_out/coverage_bigwigs --bigwig-exe=/home/users/mali/raildotbio/bedGraphToBigWig --manifest=/scratch/users/mali/Tasks/Rail_RNA/manifest_files/Adipo2.manifest >/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/1 2>/mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/coverage/dp.reduce.log/1.0.log" failed; exit level was 1.
Job flow failed on Friday, Oct 18, 2019 at 07:55:31 PM CEST. Run time was 2953.898 seconds.
Job flow failed on Friday, Oct 18, 2019 at 07:55:31 PM CEST. Run time was 2953.898 seconds.
To start this job flow from where it left off, run:
/home/users/mali/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /mnt/irisgpfs/users/mali/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/resume_flow_H25QY279D6LW.json -b /mnt/irisgpfs/users/mali/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/flow.2019-10-18T19:06:15.191256.log -f --max-attempts 1 --num-processes 111
To start this job flow from where it left off, run:
/home/users/mali/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /mnt/irisgpfs/users/mali/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/resume_flow_H25QY279D6LW.json -b /mnt/irisgpfs/users/mali/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /mnt/lscratch/users/mali/Tasks/Rail_RNA/railRNA_out/Adipo2/rail-rna_logs/flow.2019-10-18T19:06:15.191256.log -f --max-attempts 1 --num-processes 111
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/mnt/irisgpfs/users/mali/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/mnt/irisgpfs/users/mali/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
    max_attempts=max_attempts
  File "/mnt/irisgpfs/users/mali/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError

[Rimsk]

Hi, Any solution for the above error. I am getting the same error. Please has anyone got the solution for this. Thank you in advance.

Streaming command "LC_ALL=C sort -T temp_Rail_RNA1 -S 307200 -k1,1 -t$'\t' -m /storage/rxk519/rail_RNA_results1/logs/align_reads/dp.tasks/2.* | /usr/local/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/raildotbio/rail-rna/rna/steps/align_reads.py --bowtie-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_bowtie1_ebwt/hg19 --bowtie2-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/hg19 --bowtie2-exe=/usr/local/raildotbio/bowtie2-2.2.7/bowtie2 --exon-differentials --partition-length=5000 --min-exon-size=9 --search-filter=1 --manifest=/nethome/rxk519/CCLE_1.manifest --max-readlet-size=25 --readlet-interval=4 --capping-multiplier=1.1 --gzip-level 3 --index-count 100 --tie-margin 0 --verbose --scratch temp_Rail_RNA1 --output-bam-by-chr --no-polyA -- 2>/storage/rxk519/rail_RNA_results1/logs/align_reads/dp.reduce.log/2.0.log" failed; exit level was 1.
Job flow failed on Monday, Nov 30, 2020 at 01:57:47 PM EST. Run time was 35.479 seconds.
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in
args.direct_write)
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
max_attempts=max_attempts
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
raise RuntimeError
RuntimeError