Crosspost from kallistobustools: kb count error with CSP reads

[kaijli]

Cross-posting from kallistobustools repo because I realized this repo has more activity:

I've run into an issue with running CSP data (TotalSeq C, adjacent to Chromium 10X protocol for GEX).
The data is FASTQ files with reads of 26 base pairs (and a couple with some 27 bp reads).
They were sequenced as 26x10x10x[25-27] bp configuration CSP libraries.
I've been looking into running kb count and use the H5AD files with the GEX H5AD outputs.
My version of kbtools runs fine on GEX data but always produces the same error when trying to run CSP.
I've tried changing the files, the parameters, reinstalling, etc.

Snippet from standard error

[2024-02-22 16:25:38,978]   DEBUG [main] Printing verbose output
[2024-02-22 16:25:41,189]   DEBUG [main] kallisto binary located at /home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/kallisto/kallisto
[2024-02-22 16:25:41,195]   DEBUG [main] bustools binary located at /home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/bustools/bustools
[2024-02-22 16:25:41,196]   DEBUG [main] Creating `/home/kb_out2/2023_03_15-samples_CSP/1S1_S1/tmp` directory
[2024-02-22 16:25:41,201]   DEBUG [main] Namespace(list=False, command='count', tmp=None, keep_tmp=False, verbose=True, i='/home/ref/kb_custom_index', g='/home/ref/kb_custom_t2g', x='10XV2', o='/home/kb_out2/2023_03_15-samples_CSP/1S1_S1', num=False, w=None, r=None, t=8, m='48G', strand='forward', inleaved=False, genomebam=False, aa=False, gtf=None, chromosomes=None, workflow='standard', em=False, mm=True, tcc=False, filter='bustools', filter_threshold=None, c1=None, c2=None, overwrite=False, dry_run=False, batch_barcodes=False, loom=False, h5ad=True, loom_names='barcode,target_name', sum='none', cellranger=False, gene_names=False, N=None, report=False, no_inspect=False, kallisto='/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/kallisto/kallisto', bustools='/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/bustools/bustools', no_validate=False, no_fragment=False, parity=None, fragment_l=None, fragment_s=None, bootstraps=None, matrix_to_files=False, matrix_to_directories=False, fastqs=['1S1_S1_L001_QC_001.1.trimmed.fastq', '1S1_S1_L001_QC_001.2.trimmed.fastq'])
[2024-02-22 16:25:45,302]    INFO [count] Using index /home/ref/kb_custom_index to generate BUS file to /home/kb_out2/2023_03_15-samples_CSP/1S1_S1 from
[2024-02-22 16:25:45,305]    INFO [count]         1S1_S1_L001_QC_001.1.trimmed.fastq
[2024-02-22 16:25:45,307]    INFO [count]         1S1_S1_L001_QC_001.2.trimmed.fastq
[2024-02-22 16:25:45,307]   DEBUG [count] kallisto bus -i /home/ref/kb_custom_index -o /home/kb_out2/2023_03_15-samples_CSP/1S1_S1 -x 10XV2 -t 8 --fr-stranded 1S1_S1_L001_QC_001.1.trimmed.fastq 1S1_S1_L001_QC_001.2.trimmed.fastq
[2024-02-22 16:25:45,410]   DEBUG [count] 
[2024-02-22 16:25:46,611]   ERROR [count] 
[2024-02-22 16:25:46,611]   ERROR [main] An exception occurred
Traceback (most recent call last):
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 1618, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 703, in parse_count
    count(
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/ngs_tools/logging.py", line 62, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 1277, in count
    bus_result = kallisto_bus(
                 ^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 203, in kallisto_bus
    run_executable(command)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/dry/__init__.py", line 25, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/utils.py", line 203, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))
subprocess.CalledProcessError: Command '/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/kallisto/kallisto bus -i /homeref/kb_custom_index -o /home/kb_out2/2023_03_15-samples_CSP/1S1_S1 -x 10XV2 -t 8 --fr-stranded 1S1_S1_L001_QC_001.1.trimmed.fastq 1S1_S1_L001_QC_001.2.trimmed.fastq' died with <Signals.SIGILL: 4>.
[2024-02-22 16:25:46,615]   DEBUG [main] Removing `/home/kb_out2/2023_03_15-samples_CSP/1S1_S1/tmp` directory

Snippet from srun file

#SBATCH --mem=64G
#SBATCH --partition=tb
#SBATCH --cpus-per-task=32

kb count \
-o /home/kb_out2/2023_03_15-samples_CSP/1S1_S1 \
-i /home/ref/kb_custom_index \
-g /home/ref/kb_custom_t2g \
-x 10XV2 \
-m 48G \
-t 8 \
--workflow standard \
--mm \
--h5ad \
--strand forward \
--filter bustools \
--verbose \
1S1_S1_L001_QC_001.1.trimmed.fastq 1S1_S1_L001_QC_001.2.trimmed.fastq

version information:

kb_python 0.28.0
kallisto: 0.50.1 (/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/kallisto/kallisto)
bustools: 0.43.1 (/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/bustools/bustools)

[Yenaled]

So the exact same command (and index) works with GEX? Can you verify that using your current installation of kb-python?

If so, then it must be something about the FASTQ file of the CSP data. Look at the first 10 reads or so of each file (and paste them here if you can).

[kaijli]

You're right, I get the same error trying to run GEX. The command kb ref still seems to work, but it errors with kb count even with GEX data. It's strange because when the error first occurred, it was in the middle of multiple GEX and CSP runs, and only the last 3 CSP runs errorred, so I assumed it was due to the data type.

This is the first 10 of my GEX

@VH00347:121:AAAY7HYHV:1:1101:19216:1000 1:N:0:CGTGACATGC+TTTAGACCAT
TAGTGGTTCCTTGCCACTACTGAGAA
+
C-CCCCCCCC;-CCC-C;;-CCCCCC
@VH00347:121:AAAY7HYHV:1:1101:19595:1000 1:N:0:CGTGACATGC+TTTAGACCAT
TCTGAGAAGCGATCCCAGCCTCGAGA
+
CCCCCCCCCCCCCCCCCC-C-CCCCC
@VH00347:121:AAAY7HYHV:1:1101:20201:1000 1:N:0:CGTGACATGC+TTTAGACCAT
AACCATGGTGAAATCAAGTGAGTGGG

and the subsequent error:

Traceback (most recent call last):
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 1618, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 703, in parse_count
    count(
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/ngs_tools/logging.py", line 62, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 1277, in count
    bus_result = kallisto_bus(
                 ^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 203, in kallisto_bus
    run_executable(command)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/dry/__init__.py", line 25, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/utils.py", line 203, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))

Here is also the first 10 of my CSP read.

@VH00347:124:AAAY5CHHV:1:1101:18951:1000 1:N:0:GTCGTTGCCT+AAGAAGTTCT
GTACGTAGTAGAAAGGAATTCCTATT
+
CC-C-CCCCCCC;CCCCC;CCCCCCC
@VH00347:124:AAAY5CHHV:1:1101:20087:1000 1:N:0:GTCGTTGCCT+AAGAAGTTCT
TAGTGGTAGATGAGAGTTATAGTCAA
+
C-CC-CCCC-C-CC-CCCCCCCCCCC
@VH00347:124:AAAY5CHHV:1:1101:21412:1000 1:N:0:GTCGTTGCCT+AAGAAGTTCT
ACCGTAAGTCAAAGATACCAAAACTC

Thanks!

[Yenaled]

One suggestion: Install kallisto from source and run your kb count via:

kb count --kallisto=/path/to/kallisto

[Yenaled]

To install from source:

git clone --branch v0.50.1 https://github.com/pachterlab/kallisto
cd kallisto
mkdir build
cd build
cmake ..
make
cd ../../

And then your binary will at: kallisto/build/src/kallisto

[kaijli]

I've installed kallisto from source. Do I need to do the same for bustools?
I am able to run through kb ref without issue, but then kb count errors, saying that the indices are incompatible:

Error: incompatible indices. Found version 10, expected version 13

These two commands are run together, so I'm still very confused.

kb ref \
    --kallisto=/home/kli/kallisto/build/src/kallisto \
    -i ${indexFile} \
    -g ${t2gFile} \
    -d human \
    --overwrite \
    --workflow standard \
    --verbose \
    ${assemblyFile} \
    ${gtfFile}

kb count \
    --kallisto=/home/kli/kallisto/build/src/kallisto \
    -o ${outDir} \
    -i ${indexFile} \
    -g ${t2gFile} \
    -x 10XV2 \
    -m 48G \
    -t 8 \
    --workflow standard \
    --mm \
    --h5ad \
    --strand forward \
    --filter bustools \
    --verbose \
    ${fastqs}

I've verified that the kallisto built from source is the version you specified above. Should I be trying to build the reference just using kallisto index instead of kb ref?

[kaijli]

I somehow made the index error go away, but I still end up with the same kallisto error:

[2024-02-28 12:30:44,720]   ERROR [main] An exception occurred
Traceback (most recent call last):
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 1618, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 703, in parse_count
    count(
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/ngs_tools/logging.py", line 62, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 1277, in count
    bus_result = kallisto_bus(
                 ^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 203, in kallisto_bus
    run_executable(command)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/dry/__init__.py", line 25, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/utils.py", line 203, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))

It's correctly calling the kallisto version. DEBUG [main] kallisto binary located at /home/kli/kallisto/build/src/kallisto

[Yenaled]

Can you run kb count with --verbose and paste the full error message? Otherwise, I can't see where kallisto is failing.

[kaijli]

Yes, this is an output from --verbose with a couple of paths shortened

[2024-02-28 12:30:46,920]   DEBUG [main] Printing verbose output
[2024-02-28 12:30:49,131]   DEBUG [main] kallisto binary located at /home/kli/kallisto/build/src/kallisto
[2024-02-28 12:30:49,131]   DEBUG [main] bustools binary located at /home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/bustools/bustools
[2024-02-28 12:30:49,131]   DEBUG [main] Creating `kb_out2/GEX_test/D3_S1/tmp` directory
[2024-02-28 12:30:49,140]   DEBUG [main] Namespace(list=False, command='count', tmp=None, keep_tmp=False, verbose=True, i='ref/kb_custom_index', g='ref/kb_custom_t2g', x='10XV2', o='kb_out2/GEX_test/D3_S1', num=False, w=None, r=None, t=8, m='48G', strand='forward', inleaved=False, genomebam=False, aa=False, gtf=None, chromosomes=None, workflow='standard', em=False, mm=True, tcc=False, filter='bustools', filter_threshold=None, c1=None, c2=None, overwrite=False, dry_run=False, batch_barcodes=False, loom=False, h5ad=True, loom_names='barcode,target_name', sum='none', cellranger=False, gene_names=False, N=None, report=False, no_inspect=False, kallisto='/home/kli/kallisto/build/src/kallisto', bustools='/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/bins/linux/bustools/bustools', no_validate=False, no_fragment=False, parity=None, fragment_l=None, fragment_s=None, bootstraps=None, matrix_to_files=False, matrix_to_directories=False, fastqs=['D3_S1_L001_QC_001.1.trimmed.fastq', 'D3_S1_L001_QC_001.2.trimmed.fastq'])
[2024-02-28 12:30:52,614]    INFO [count] Using index ref/kb_custom_index to generate BUS file to kb_out2/GEX_test/D3_S1 from
[2024-02-28 12:30:52,615]    INFO [count]         D3_S1_L001_QC_001.1.trimmed.fastq
[2024-02-28 12:30:52,615]    INFO [count]         D3_S1_L001_QC_001.2.trimmed.fastq
[2024-02-28 12:30:52,615]   DEBUG [count] kallisto bus -i ref/kb_custom_index -o kb_out2/GEX_test/D3_S1 -x 10XV2 -t 8 --fr-stranded D3_S1_L001_QC_001.1.trimmed.fastq D3_S1_L001_QC_001.2.trimmed.fastq
[2024-02-28 12:30:52,719]   DEBUG [count] 
[2024-02-28 12:30:54,121]   ERROR [count] 
[2024-02-28 12:30:54,121]   ERROR [main] An exception occurred
Traceback (most recent call last):
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 1618, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/main.py", line 703, in parse_count
    count(
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/ngs_tools/logging.py", line 62, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 1277, in count
    bus_result = kallisto_bus(
                 ^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/count.py", line 203, in kallisto_bus
    run_executable(command)
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/dry/__init__.py", line 25, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kallisto/lib/python3.11/site-packages/kb_python/utils.py", line 203, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))

[Yenaled]

Very strange — there are no messages after the lines you posted?

Otherwise, try running that “kallisto bus” command shown in your output, using your kallisto binary path.

[kaijli]

The only thing after the error message I posted above is the output from /usr/bin/time -v

Command being timed: "kb count --kallisto=/home/kli/kallisto/build/src/kallisto -o kb_out2/GEX_test/D10_S10 -i ref/kb_custom_index -g ref/kb_custom_t2g -x 10XV2 -m 48G -t 8 --workflow standard --mm --h5ad --strand forward --filter bustools --verbose D10_S10_L001_QC_001.1.trimmed.fastq D10_S10_L001_QC_001.2.trimmed.fastq"
	User time (seconds): 2.84
	System time (seconds): 1.44
	Percent of CPU this job got: 45%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.50
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 170424
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 6
	Minor (reclaiming a frame) page faults: 96819
	Voluntary context switches: 20723
	Involuntary context switches: 18
	Swaps: 0
	File system inputs: 2512
	File system outputs: 40
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

I ran the kallisto bus command in an active terminal session.

/home/kli/kallisto/build/src/kallisto bus -i ref/kb_custom_index -o kb_out2/GEX_test/D3_S1 -x 10XV2 -t 8 --fr-stranded D3_S1_L001_QC_001.1.trimmed.fastq D3_S1_L001_QC_001.2.trimmed.fastq

[index] k-mer length: 31
[index] number of targets: 227,665
[index] number of k-mers: 139,900,295
[index] number of D-list k-mers: 5,477,475
[quant] will process sample 1: D3_S1_L001_QC_001.1.trimmed.fastq
                               D3_S1_L001_QC_001.2.trimmed.fastq
[progress] 249M reads processed (5.2% mapped)              done
[quant] processed 250,248,504 reads, 12,911,502 reads pseudoaligned

It ran to completion and ouput files matrix.ec, output.bus, run_info.json, and transcripts.txt.
Does this mean there's something wrong with my version of kb_python?

[Yenaled]

Try pip installing the latest version of kb_python (which is 0.28.2).

Only other thing I can think of is a buggy python installation.

To be honest though, I've never seen something work in a terminal session but not under kb-python (literally all kb-python does is execute that exact same command).

[Yenaled]

I'd also just say try it on a different computer/server and see if the result reproduces there (I suspect it won't). It only takes a couple of gigabytes of RAM so you can just process it on your own personal computer.

[kaijli]

Can I get instructions on how to install from source? I don't have access to pip on my HPC and I don't entirely trust conda. I tried installing into an environment I've been using but I think there's something messed up about it when I was also trying to run the kb-python tutorials with jupyter notebook (not all the versions line up for scanpy and the packages called). I'll try with a fresh conda environment, but I also want to try having kallisto, bustools, and kb-python installed from source. I have a large dataset so it'd be great it I could get it on the system I'm already troubleshooting with.

[Yenaled]

See the above to install kallisto from source -- for kb_python, you have to stick with the pip install b/c that's how it's distributed.

There are also ways to use pip to install stuff locally (it has been a while since I've done it, but it's doable -- you just have to set certain paths/variables and do pip install kb_python --user).

Otherwise, you can run the kallisto bus command to generate the files and run kb count afterwards (kb count will auto-detect the BUS file and proceed from there without rerunning kallisto bus).

[kaijli]

I ended up making a new conda environment with only kb_python and python3.11. I managed to get GEX to run without error, but CSP is still errorring with the kb count commands.
This is the standard error for the CSP run

[2024-03-05 11:03:56,485]   DEBUG Printing verbose output
[2024-03-05 11:03:56,485]   DEBUG kallisto binary located at /home/kli/.conda/envs/kb/bin/kallisto
[2024-03-05 11:03:56,486]   DEBUG bustools binary located at /home/kli/.conda/envs/kb/bin/bustools
[2024-03-05 11:03:56,486]   DEBUG Creating /home/kli/kb_out2/CSP_test/D10_S10/tmp directory
[2024-03-05 11:03:56,491]   DEBUG Namespace(list=False, command='count', tmp=None, keep_tmp=False, verbose=True, kallisto='/home/kli/.conda/envs/kb/bin/kallisto', bustools='bustools', i='/home/kli/ref/kb_custom_index', g='/home/kli/ref/kb_custom_t2g', x='10XV2', o='/home/kli/kb_out2/CSP_test/D10_S10', w=None, t=8, m='48G', workflow='standard', mm=True, tcc=False, filter='bustools', c1=None, c2=None, overwrite=True, dry_run=False, lamanno=False, nucleus=False, loom=False, h5ad=True, cellranger=False, report=False, no_inspect=False, no_validate=False, fastqs=['D10_S10_L001_QC_001.1.trimmed.fastq', 'D10_S10_L001_QC_001.2.trimmed.fastq'])
[2024-03-05 11:03:56,505]    INFO Using index /home/kli/ref/kb_custom_index to generate BUS file to /home/kli/kb_out2/CSP_test/D10_S10 from
[2024-03-05 11:03:56,505]    INFO         D10_S10_L001_QC_001.1.trimmed.fastq
[2024-03-05 11:03:56,505]    INFO         D10_S10_L001_QC_001.2.trimmed.fastq
[2024-03-05 11:03:56,505]   DEBUG kallisto bus -i /home/kli/ref/kb_custom_index -o /home/kli/kb_out2/CSP_test/D10_S10 -x 10XV2 -t 8 D10_S10_L001_QC_001.1.trimmed.fastq D10_S10_L001_QC_001.2.trimmed.fastq
[2024-03-05 11:06:44,859]   DEBUG 
[2024-03-05 11:06:44,862]   DEBUG [index] k-mer length: 31
[2024-03-05 11:06:44,863]   DEBUG [index] number of targets: 227,368
[2024-03-05 11:06:44,863]   DEBUG [index] number of k-mers: 140,125,185
[2024-03-05 11:06:44,863]   DEBUG [index] number of equivalence classes: 964,094
[2024-03-05 11:06:44,863]   DEBUG [quant] will process sample 1: D10_S10_L001_QC_001.1.trimmed.fastq
[2024-03-05 11:06:44,863]   DEBUG D10_S10_L001_QC_001.2.trimmed.fastq
[2024-03-05 11:06:44,863]   DEBUG [quant] finding pseudoalignments for the reads ... done
[2024-03-05 11:06:44,863]   DEBUG [quant] processed 72,208,578 reads, 0 reads pseudoaligned
[2024-03-05 11:06:44,863]   DEBUG [~warn] no reads pseudoaligned.
[2024-03-05 11:06:44,863]   DEBUG 
[2024-03-05 11:06:44,863]   ERROR 
[index] k-mer length: 31
[index] number of targets: 227,368
[index] number of k-mers: 140,125,185
[index] number of equivalence classes: 964,094
[quant] will process sample 1: D10_S10_L001_QC_001.1.trimmed.fastq
D10_S10_L001_QC_001.2.trimmed.fastq
[quant] finding pseudoalignments for the reads ... done
[quant] processed 72,208,578 reads, 0 reads pseudoaligned
[~warn] no reads pseudoaligned.

[2024-03-05 11:06:44,864]   ERROR An exception occurred
Traceback (most recent call last):
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/main.py", line 866, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/main.py", line 314, in parse_count
    count(
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/count.py", line 1123, in count
    bus_result = kallisto_bus(
                 ^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/validate.py", line 112, in inner
    results = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/count.py", line 149, in kallisto_bus
    run_executable(command)
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/dry/__init__.py", line 24, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/kli/.conda/envs/kb/lib/python3.11/site-packages/kb_python/utils.py", line 237, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))
subprocess.CalledProcessError: Command '/home/kli/.conda/envs/kb/bin/kallisto bus -i /home/kli/ref/kb_custom_index -o /home/kli/kb_out2/CSP_test/D10_S10 -x 10XV2 -t 8 D10_S10_L001_QC_001.1.trimmed.fastq D10_S10_L001_QC_001.2.trimmed.fastq' returned non-zero exit status 1.
[2024-03-05 11:06:44,867]   DEBUG Removing /home/kli/kb_out2/CSP_test/D10_S10/tmp directory
	Command being timed: "/home/kli/.conda/envs/kb/bin/kb count --kallisto=/home/kli/.conda/envs/kb/bin/kallisto -o /home/kli/kb_out2/CSP_test/D10_S10 -i /home/kli/ref/kb_custom_index -g /home/kli/ref/kb_custom_t2g -x 10XV2 -m 48G -t 8 --workflow standard --overwrite --mm --h5ad --filter bustools --verbose D10_S10_L001_QC_001.1.trimmed.fastq D10_S10_L001_QC_001.2.trimmed.fastq"
	User time (seconds): 142.84
	System time (seconds): 38.31
	Percent of CPU this job got: 105%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:51.33
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7106372
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 265938
	Voluntary context switches: 654
	Involuntary context switches: 411
	Swaps: 0
	File system inputs: 0
	File system outputs: 106080
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

I was able to run kallisto bus -i /home/kli/ref/kb_custom_index -o /home/kli/kb_out2/CSP_test/D10_S10 -x 10XV2 -t 8 D10_S10_L001_QC_001.1.trimmed.fastq D10_S10_L001_QC_001.2.trimmed.fastq in the command line, and it stops at the same point it does in the standard error above.

It also exits my loop after one set of files, so I'm not entirely sure what's happening there. There's no difference between the code run for GEX and CSP.
This is standard out for one of the looped GEX runs (both runs are without ref, which runs fine)

[2024-03-05 10:16:19,652]   DEBUG Printing verbose output
[2024-03-05 10:16:19,652]   DEBUG kallisto binary located at /home/kli/.conda/envs/kb/bin/kallisto
[2024-03-05 10:16:19,652]   DEBUG bustools binary located at /home/kli/.conda/envs/kb/bin/bustools
[2024-03-05 10:16:19,653]   DEBUG Creating /home/kli/kb_out2/GEX_test/1S1_S1/tmp directory
[2024-03-05 10:16:19,656]   DEBUG Namespace(list=False, command='count', tmp=None, keep_tmp=False, verbose=True, kallisto='/home/kli/.conda/envs/kb/bin/kallisto', bustools='bustools', i='/home/kli/ref/kb_custom_index', g='/home/kli/ref/kb_custom_t2g', x='10XV2', o='/home/kli/kb_out2/GEX_test/1S1_S1', w=None, t=8, m='48G', workflow='standard', mm=True, tcc=False, filter='bustools', c1=None, c2=None, overwrite=True, dry_run=False, lamanno=False, nucleus=False, loom=False, h5ad=True, cellranger=False, report=False, no_inspect=False, no_validate=False, fastqs=['1S1_S1_L001_QC_001.1.trimmed.fastq', '1S1_S1_L001_QC_001.2.trimmed.fastq'])
[2024-03-05 10:16:19,668]    INFO Using index /home/kli/ref/kb_custom_index to generate BUS file to /home/kli/kb_out2/GEX_test/1S1_S1 from
[2024-03-05 10:16:19,668]    INFO         1S1_S1_L001_QC_001.1.trimmed.fastq
[2024-03-05 10:16:19,668]    INFO         1S1_S1_L001_QC_001.2.trimmed.fastq
[2024-03-05 10:16:19,668]   DEBUG kallisto bus -i /home/kli/ref/kb_custom_index -o /home/kli/kb_out2/GEX_test/1S1_S1 -x 10XV2 -t 8 1S1_S1_L001_QC_001.1.trimmed.fastq 1S1_S1_L001_QC_001.2.trimmed.fastq
[2024-03-05 10:24:13,164]   DEBUG 
[2024-03-05 10:24:13,165]   DEBUG [index] k-mer length: 31
[2024-03-05 10:24:13,165]   DEBUG [index] number of targets: 227,368
[2024-03-05 10:24:13,165]   DEBUG [index] number of k-mers: 140,125,185
[2024-03-05 10:24:13,165]   DEBUG [index] number of equivalence classes: 964,094
[2024-03-05 10:24:13,165]   DEBUG [quant] will process sample 1: 1S1_S1_L001_QC_001.1.trimmed.fastq
[2024-03-05 10:24:13,165]   DEBUG 1S1_S1_L001_QC_001.2.trimmed.fastq
[2024-03-05 10:24:13,165]   DEBUG [quant] finding pseudoalignments for the reads ... done
[2024-03-05 10:24:13,165]   DEBUG [quant] processed 280,432,089 reads, 124,368,682 reads pseudoaligned
[2024-03-05 10:24:13,165]   DEBUG 
[2024-03-05 10:24:44,190]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/output.bus passed validation
[2024-03-05 10:24:44,194]    INFO Sorting BUS file /home/kli/kb_out2/GEX_test/1S1_S1/output.bus to /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus
[2024-03-05 10:24:44,194]   DEBUG bustools sort -o /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus -T /home/kli/kb_out2/GEX_test/1S1_S1/tmp -t 8 -m 48G /home/kli/kb_out2/GEX_test/1S1_S1/output.bus
[2024-03-05 10:26:23,478]   DEBUG Read in 124368682 BUS records
[2024-03-05 10:26:28,024]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus passed validation
[2024-03-05 10:26:28,027]    INFO Whitelist not provided
[2024-03-05 10:26:28,027]    INFO Copying pre-packaged 10XV2 whitelist to /home/kli/kb_out2/GEX_test/1S1_S1
[2024-03-05 10:26:28,410]    INFO Inspecting BUS file /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus
[2024-03-05 10:26:28,410]   DEBUG bustools inspect -o /home/kli/kb_out2/GEX_test/1S1_S1/inspect.json -w /home/kli/kb_out2/GEX_test/1S1_S1/10xv2_whitelist.txt -e /home/kli/kb_out2/GEX_test/1S1_S1/matrix.ec /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus
[2024-03-05 10:26:42,935]    INFO Correcting BUS records in /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus to /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.c.bus with whitelist /home/kli/kb_out2/GEX_test/1S1_S1/10xv2_whitelist.txt
[2024-03-05 10:26:42,935]   DEBUG bustools correct -o /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.c.bus -w /home/kli/kb_out2/GEX_test/1S1_S1/10xv2_whitelist.txt /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.bus
[2024-03-05 10:26:57,962]   DEBUG Found 737280 barcodes in the whitelist
[2024-03-05 10:26:57,962]   DEBUG Processed 43135112 BUS records
[2024-03-05 10:26:57,962]   DEBUG In whitelist = 39393962
[2024-03-05 10:26:57,962]   DEBUG Corrected    = 1047370
[2024-03-05 10:26:57,962]   DEBUG Uncorrected  = 2693780
[2024-03-05 10:27:02,054]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.c.bus passed validation
[2024-03-05 10:27:02,055]    INFO Sorting BUS file /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.c.bus to /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:27:02,055]   DEBUG bustools sort -o /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus -T /home/kli/kb_out2/GEX_test/1S1_S1/tmp -t 8 -m 48G /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.s.c.bus
[2024-03-05 10:28:16,702]   DEBUG Read in 40441332 BUS records
[2024-03-05 10:28:20,742]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus passed validation
[2024-03-05 10:28:20,745]    INFO Generating count matrix /home/kli/kb_out2/GEX_test/1S1_S1/counts_unfiltered/cells_x_genes from BUS file /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:28:20,745]   DEBUG bustools count -o /home/kli/kb_out2/GEX_test/1S1_S1/counts_unfiltered/cells_x_genes -g /home/kli/ref/kb_custom_t2g -e /home/kli/kb_out2/GEX_test/1S1_S1/matrix.ec -t /home/kli/kb_out2/GEX_test/1S1_S1/transcripts.txt --genecounts --multimapping /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:29:03,065]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/counts_unfiltered/cells_x_genes.mtx passed validation
[2024-03-05 10:29:03,066]    INFO Reading matrix /home/kli/kb_out2/GEX_test/1S1_S1/counts_unfiltered/cells_x_genes.mtx
[2024-03-05 10:29:26,772]    INFO Writing matrix to h5ad /home/kli/kb_out2/GEX_test/1S1_S1/counts_unfiltered/adata.h5ad
[2024-03-05 10:29:28,824]    INFO Filtering with bustools
[2024-03-05 10:29:28,824]    INFO Generating whitelist /home/kli/kb_out2/GEX_test/1S1_S1/filter_barcodes.txt from BUS file /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:29:28,824]   DEBUG bustools whitelist -o /home/kli/kb_out2/GEX_test/1S1_S1/filter_barcodes.txt /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:29:29,677]   DEBUG Read in 40019744 BUS records, wrote 13714 barcodes to whitelist with threshold 1436
[2024-03-05 10:29:29,678]    INFO Correcting BUS records in /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus to /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.unfiltered.c.bus with whitelist /home/kli/kb_out2/GEX_test/1S1_S1/filter_barcodes.txt
[2024-03-05 10:29:29,678]   DEBUG bustools correct -o /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.unfiltered.c.bus -w /home/kli/kb_out2/GEX_test/1S1_S1/filter_barcodes.txt /home/kli/kb_out2/GEX_test/1S1_S1/output.unfiltered.bus
[2024-03-05 10:29:45,151]   DEBUG Found 13713 barcodes in the whitelist
[2024-03-05 10:29:45,152]   DEBUG Processed 40019744 BUS records
[2024-03-05 10:29:45,152]   DEBUG In whitelist = 29840659
[2024-03-05 10:29:45,152]   DEBUG Corrected    = 0
[2024-03-05 10:29:45,152]   DEBUG Uncorrected  = 10179085
[2024-03-05 10:29:47,855]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.unfiltered.c.bus passed validation
[2024-03-05 10:29:47,855]    INFO Sorting BUS file /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.unfiltered.c.bus to /home/kli/kb_out2/GEX_test/1S1_S1/output.filtered.bus
[2024-03-05 10:29:47,856]   DEBUG bustools sort -o /home/kli/kb_out2/GEX_test/1S1_S1/output.filtered.bus -T /home/kli/kb_out2/GEX_test/1S1_S1/tmp -t 8 -m 48G /home/kli/kb_out2/GEX_test/1S1_S1/tmp/output.unfiltered.c.bus
[2024-03-05 10:30:55,975]   DEBUG Read in 29840659 BUS records
[2024-03-05 10:30:58,905]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/output.filtered.bus passed validation
[2024-03-05 10:30:58,908]    INFO Generating count matrix /home/kli/kb_out2/GEX_test/1S1_S1/counts_filtered/cells_x_genes from BUS file /home/kli/kb_out2/GEX_test/1S1_S1/output.filtered.bus
[2024-03-05 10:30:58,908]   DEBUG bustools count -o /home/kli/kb_out2/GEX_test/1S1_S1/counts_filtered/cells_x_genes -g /home/kli/ref/kb_custom_t2g -e /home/kli/kb_out2/GEX_test/1S1_S1/matrix.ec -t /home/kli/kb_out2/GEX_test/1S1_S1/transcripts.txt --genecounts /home/kli/kb_out2/GEX_test/1S1_S1/output.filtered.bus
[2024-03-05 10:31:22,612]   DEBUG /home/kli/kb_out2/GEX_test/1S1_S1/counts_filtered/cells_x_genes.mtx passed validation
[2024-03-05 10:31:22,612]    INFO Reading matrix /home/kli/kb_out2/GEX_test/1S1_S1/counts_filtered/cells_x_genes.mtx
[2024-03-05 10:31:33,218]    INFO Writing matrix to h5ad /home/kli/kb_out2/GEX_test/1S1_S1/counts_filtered/adata.h5ad
[2024-03-05 10:31:34,336]   DEBUG Removing /home/kli/kb_out2/GEX_test/1S1_S1/tmp directory
	Command being timed: "/home/kli/.conda/envs/kb/bin/kb count --kallisto=/home/kli/.conda/envs/kb/bin/kallisto -o /home/kli/kb_out2/GEX_test/1S1_S1 -i /home/kli/ref/kb_custom_index -g /home/kli/ref/kb_custom_t2g -x 10XV2 -m 48G -t 8 --workflow standard --overwrite --mm --h5ad --filter bustools --verbose 1S1_S1_L001_QC_001.1.trimmed.fastq 1S1_S1_L001_QC_001.2.trimmed.fastq"
	User time (seconds): 2970.85
	System time (seconds): 346.22
	Percent of CPU this job got: 361%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 15:18.10
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 50334632
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 3
	Minor (reclaiming a frame) page faults: 38212739
	Voluntary context switches: 12850
	Involuntary context switches: 6437
	Swaps: 0
	File system inputs: 3984
	File system outputs: 20434480
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0