Support for CSI indexes

[nathanhaigh]

What are the chances of Sambamba supporting .csi index files?

I work with cereal crops where this is starting to be an issue. Many plant species have chromosomes in excess of the ~500Mbp limits of the .bai index.

[lomereiter]

It should be straightforward to add.

<rant>
It's just very annoying to deal with hts-specs where half of the specification is in the PDF, and another half is to be found somewhere in htslib source code (samtools/hts-specs#70). Not sure if it's better with CRAM but that's one of the reasons I gave up and used htslib to read/write it.
</rant>
Fortunately, @kortschak went through this pain already, so at least biogo implementation is readable.

[nathanhaigh]

Nice one @kortschak! Any chance you could provide some help/input here if needs be? Could meet up for a coffee if you are willing - I'll buy! :)

[kortschak]

@nathanhaigh There is not much to add here. I have implemented what is described in the struct definition document and done some looking through the code to clarify things. In the absence of any input from the hts-specs authors I don't see any other things to do, but they are too busy Actually Doing Science to provide a spec for that that could be worked from. Sorry.

I'd be happy to discuss though if you are in town.

[pjotrp]

I think we need to appreciate that writing specs and source code is Actually Doing Science.

Toward effective software solutions for big biology, by Pjotr Prins, Joep de Ligt, Artem Tarasov, Ritsert C Jansen, Edwin Cuppen and Philip E Bourne, Nature Biotechnology, 2015; 33, 686-687 (higly accessed) http://www.nature.com/nbt/journal/v33/n7/full/nbt.3240.html

[kortschak]

@pjotrp Yes, that phrase is a quote that was used to explain why the specs are the way they are. I agree entirely that any kind of science should be reproducible and reviewable, including the code that is used.

[pjotrp]

Feel free to pick this up.

[fgvieira]

Are csi indexes supported by sambamba?

samtools fails when generating fai files, and I am getting an error when merging some bam files with csi indexes:

sambamba 1.0.1
 by Artem Tarasov and Pjotr Prins (C) 2012-2023
    LDC 1.39.0 / DMD v2.109.1 / LLVM17.0.6 / bootstrap LDC - the LLVM D compiler (1.39.0)

sambamba-merge: BAM index file (.bai) must be provided

Depending on the input files, sometimes I can get sambamba to merge them without fai files. Could it be related to sambamba trying to write the output bai file?

[pjotrp]

CSI is not yet supported. Should not be hard to do, so if anyone wants to volunteer a patch...