RAMTools 0.2.0

The format of this release stores the typed SAM tags in columns of their own, the
mate's position and template length as differences from what the record predicts,
and the read names, qualities and bases in blocks of 10,000 records with the codecs
CRAM 3.1 uses. On chromosome 20 of NA12878 at 30x the file goes from 622.5 MB, with
every tag as text as before #111, to 417.9 MB; CRAM 3.1 takes 302.5 MB. New are
ramstats idxstats and flagstat, ramdump -@ N and several regions in one
ramdump, and a script that reproduces the benchmarks. A reader opens only files
of its own format version, 10: files written by 0.1.0 must be converted again.

Added

• bench/ramtools-bench.py reproduces the benchmarks on any Linux or macOS
  machine: it downloads chr20 of NA12878 from EBI and checks it against known
  md5 sums, builds two revisions the same way, measures conversion, size, the
  round trip, dumps, regions and statistics in alternating order, checks every
  output against samtools, and writes a report. See bench/README.md
  (#116). On Linux it runs each command under a small helper it compiles,
  because a command started from Python counted Python's 20 to 30 MB in its
  peak memory: samtools idxstats showed 22 to 30 MB instead of 10 (#117).
• -fqzcomp on both converters compresses lossless quality scores with
  fqzcomp, the codec of CRAM 3.1, in blocks of 10,000 records. On 2 million
  records of chromosome 11 the file is 8.1% smaller, conversion takes 20 to 30%
  longer and a full dump about 30% longer; region counts are unchanged. It is
  off by default, and files written with it need this release or later to read
  their qualities (#90, #99).
• ramstats idxstats prints the records per reference as samtools idxstats
  does. The converters count them while writing and store the counts in
  METADATA, so it reads no record: 0.2 s and 5 MB for any file size. The format
  version becomes 3, and files written before must be converted again (#102).
• ramstats flagstat prints what samtools flagstat prints, in its default,
  TSV (-O tsv) and JSON (-O json) formats. It reads only the FLAG,
  reference, mate reference and MAPQ columns. On chr20 of NA12878 at 30x
  (17.7 million records) it takes 1.3 s of CPU and 20 MB on one thread, against
  9.4 s for samtools flagstat on the BAM and 3.9 s on the CRAM 3.1. The file
  format is unchanged (#108).
• ramdump -@ N reads and formats the whole file on N threads, each taking
  every N-th block of 10,000 records with a reader of its own; the records come
  out in order. On chromosome 20 of NA12878 a full read takes 4.6 s on 8
  threads and 7.9 s on 4 instead of 16.5 s on one, with 546 MB instead of
  107 MB on 8; samtools view -@ 8 takes 3.6 s from BAM and 7.7 s from
  CRAM 3.1. With a region, -@ is ignored (#124).
• ramdump takes several regions and reads them in turn from the file kept
  open, each region's records in order, as samtools view does without -M;
  -c counts them all. On chromosome 20 of NA12878, 100 regions of 1 kb take
  0.73 s in one process instead of 6.55 s in a process each (#125).

Changed

• ramdump with a region searches for it with only the reference and position
  columns open, opens the others at the first record found, and reads without
  RNTuple's cluster cache, which loaded every column of each cluster the search
  visited and of the next one. On chromosome 20 of NA12878 a region of 1 kb
  reads 1.0 MB of the file instead of 53.4 MB and takes 68 ms and 38 MB
  instead of 103 ms and 50 MB, or 80 ms instead of 246 ms from a cold cache.
  A count with -f or -F reads only the flags: 0.31 s instead of 1.76 s in
  a 10 Mb region. Library: RAMFile::Open() takes RNTuple's read options;
  without the cluster cache, a region of 1 kb in a file kept open takes 6.3 ms
  instead of 34.4 ms (#119).
• ramstats flagstat reads its four columns 65,536 rows at a time with
  RNTuple's bulk reads instead of one view call per value, which took four
  fifths of its time. On chromosome 20 of NA12878 it takes 0.44 s instead of
  1.38 s, against 5.9 s for samtools flagstat on the BAM and 2.9 s on the
  CRAM 3.1 (#120).
• bamtoramntuple -threads N fills and compresses the records on N threads, as
  samtoramntuple does: the BAM is read in blocks of 64 MB, each filled on a
  thread of its own and committed in input order, and htslib inflates the
  input on N threads instead of on every core of the machine. On chromosome 20
  of NA12878 a conversion on 8 threads takes 22.8 s instead of 81.2 s and
  203 s of CPU instead of 357 s, with 1136 MB instead of 419 MB. The file has a
  cluster per block, 115 of about 3.8 MB instead of 6 of about 70 MB, as from
  SAM. A read error in the BAM now stops the conversion with a message (#122).
• ramdump reads the fifteen numeric fields of a record in bulk, a run of rows
  at a time, instead of one view call per value. On chromosome 20 of NA12878 a
  full read takes 16.8 s instead of 18.1 s and a 10 Mb region 3.2 s instead of
  3.4 s; a small region reads as many pages as before (#123).
• ramstats flagstat prints counts the converters store in METADATA while
  they write the records, as idxstats does, instead of reading four columns of
  every record: on chromosome 20 of NA12878 it takes 0.05 s instead of 0.41 s,
  and as long on any file. --scan counts them again from the columns. The
  format version becomes 9, and files written before must be converted again
  (#126).
• The converters cap pages at 128 kB before compression instead of 64 kB.
  Opening a file reads the page list of every cluster, and a file now has half
  the pages. On all of NA12878 converted from BAM, 1.99 million pages instead
  of 3.99 million: ramstats idxstats takes 0.37 s and 157 MB instead of
  0.59 s and 268 MB, a region of 1 kb 0.59 s and 209 MB instead of 0.90 s and
  345 MB, and the file is 18.16 GB instead of 18.42 GB. On chromosome 20 a
  region of 1 kb in a file kept open takes 8.3 ms instead of 6.4 ms, because
  its pages are larger, and as long as before in a process of its own. Files
  written before read as before (#127).
• The bases go into blocks of 10,000 records, in a field seqblock, compressed
  with libdeflate at level 6, instead of into a column that RNTuple compresses.
  ROOT runs zstd at twice the level it is given, so the converters' 505 is
  zstd's level 10, which on text of four letters is both larger and twice as
  slow; the converters' help and the README said level 5. On chromosome 20 of
  NA12878 the file is 417.9 MB instead of 429.6 MB, and converting from SAM on
  one thread takes 112.5 s instead of 128.8 s. A full read takes 15.1 s instead
  of 16.3 s on one thread, and 25.7 s of CPU instead of 26.4 s: the block
  readers now decode into the memory of a block they are done with, and rANS
  decodes the qualities in place. Reading only the bases takes 2.9 s instead of
  3.9 s, but 4.3 s of CPU instead of 3.9 s.
  record.seq now holds only *: a program that reads the columns directly
  has to decode seqblock, with SeqBlockReader in the library. The format
  version becomes 10, and files written before must be converted again.
  libdeflate is the system's, or version 1.26 built from source (#128).
• samtoramntuple -threads N holds up to N + 1 blocks of 64 MB in memory
  instead of 2N + 1. On 2 million records of chromosome 11 the peak goes from
  695 to 426 MB on 4 threads and from 1052 to 725 MB on 8, in the same time
  (#98).
• RAM files record a format version, and a reader opens only files of its own
  version: files written by 0.1.0 must be converted again. Until 1.0 the format
  may change between minor releases; see "Versioning" in the README (#101).
• The SAM header is stored in METADATA instead of a ROOT TList, so
  ramdump -H and -h no longer start ROOT's interpreter. On 2 million
  records of chromosome 11, -H goes from 0.86 s and 50 MB to 0.35 s and 6 MB,
  a 1 kb region with -h from 0.86 s and 90 MB to 0.40 s and 47 MB, and the
  whole file with -h from 6.2 s and 96 MB to 5.3 s and 55 MB (#101).
• Library: RAMFile::Open() opens a RAM file and keeps its reference names
  and METADATA in the object it returns instead of in static members, so
  several files can be open in one process. ramntuplescan() and
  QualityBlockReader take that object (#105).
• Library: the converters keep the state of the file they write, the
  reference tables and the METADATA values, in their own objects instead of in
  static members of RAMNTupleRecord, so several conversions can run in one
  process at once. Before, two at once mixed their reference names (#107).
• The tags NM, AS, XS, MQ, RG, PG, MD, MC, SA and XA are stored in columns of
  their own instead of as text, RG and PG as ids into a dictionary in METADATA.
  The other tags stay text, and ramdump -h still reproduces the input byte for
  byte. On chr20 of NA12878 at 30x the file is 9.5% smaller (622.5 to
  563.1 MB), conversion takes 5 to 13% less CPU, and ramdump -h 8% less. The
  format version becomes 5, and files written before must be converted again
  (#111, #112).
• PNEXT and TLEN are stored as the difference from what POS, the strands and
  the CIGAR predict, in the columns pnext_delta and tlen_delta, and still
  come back exactly. On chr20 of NA12878 at 30x the file goes from 563.1 to
  531.0 MB (-5.7%), and conversion and dumps take the same time. A program
  that reads the columns directly gets the SAM values with
  RAMNTupleRecord::GetPNEXT() and GetTLEN(); see CONTRIBUTING. The format
  version becomes 6 (#113).
• Read names are compressed with tok3, the name codec of CRAM 3.1, in blocks
  of 10,000 records, instead of stored in a qname column. On chr20 of NA12878
  at 30x the file goes from 531.0 to 478.2 MB (-9.9%), and ramdump -h still
  reproduces the input byte for byte. Small region queries, ramdump -c,
  flagstat and idxstats take the same time, and so does conversion. Reading
  every name costs CPU: on macOS a dump of the whole file takes about 6% longer
  (28 s) and 28% more CPU, with the names decoded ahead on a second thread. The
  format version becomes 7 (#114).
• Library: QualityBlocks.h becomes RecordBlocks.h and QualityBlockWriter
  becomes BlockWriter, which also writes the name blocks. NameBlockReader
  gives the name of any row; a record read from a file has an empty
  GetQNAME(). METADATA's qual_block_ends becomes block_ends (#114).
• Lossless quality scores are stored in the blocks of 10,000 records with
  rANS order 1, the codec CRAM 3.1 uses for them by default, instead of in the
  qual column. On chr20 of NA12878 at 30x the file goes from 478.2 to
  435.7 MB (-8.9%), and ramdump -h still reproduces the input byte for byte.
  Conversion takes the same CPU or up to 10% less. Reading every quality costs
  CPU: on macOS a dump of the whole file takes about 20% more CPU and 8% more
  time; small region queries, ramdump -c, flagstat and idxstats take the
  same time. The format version becomes 8 (#115).
• -fqzcomp compresses the same blocks with fqzcomp instead of rANS. On chr20
  of NA12878 at 30x, whose qualities are binned, it saves about 1% of the
  qualities and takes three times the CPU to read them (#115).
• Library: lossless qualities go into blocks whatever the quality policy, so
  kQualInBlock only says that a record's quality is in its block, and
  kFqzcomp in the policy asks for fqzcomp. A record read from a file has an
  empty GetQUAL() when its quality is in a block; QualityBlockReader gives
  the quality of any record (#115).
• ramdump writes numbers, CIGARs and qualities into its output without
  temporary strings. On chr20 of NA12878 at 30x a dump of the whole file takes
  27.9 s of CPU and 17.6 s instead of 29.8 s and 19.3 s (-6% and -9%, one
  socket of a Xeon E5-2690 v4), and a 10 Mb region 6% less CPU. The output is
  unchanged (#118).
• Library: QualityBlockReader::Get returns a std::string_view into the
  decoded block, valid until the next call, instead of a copy.
  RAMNTupleRecord::AppendCIGAR and RAMNTupleUtils::AppendInt append to a
  string without making a temporary one (#118).

Removed

• Reading files written by 0.1.0 and earlier, and with them SEQ in the 4-bit
  packing: kSeqRaw, RAMNTupleUtils::EncodeSequence and
  RAMNTupleUtils::DecodeSequence (#101).
• RAMNTupleRecord::OpenRAMFile() and ReadAllRefs(); open files with
  RAMFile::Open() (#105).
• RAMNTupleRecord's static writer state and the calls on it:
  InitializeRefs(), WriteAllRefs(), GetRnameRefs(), GetRnextRefs() and
  the static getters and setters of METADATA values. A converter now fills a
  RAMFileMetadata and calls WriteMetadata(). Also removed are the record's
  SetRNAME(), SetRNEXT(), SetREFID(name), SetREFNEXT(name),
  GetRNAME(), GetRNEXT() and Print(). A record holds ids, and
  RAMFile::GetRname() and GetRnext() give their names (#107).
• RAMNTupleRecord's AddTag(), SetOPT(), ClearTags(), ResetNOPT(),
  GetTags(), GetNOPT() and GetOPT(). tags now holds only the tags
  without a column of their own. Converters split the tags with RAMTagWriter,
  and RAMFile::AppendTags() writes all of them back (#111).
• RAMNTupleRecord's SetPNEXT() and SetTLEN(), and GetPNEXT() and
  GetTLEN() without arguments. Converters set both with SetMate(), and
  readers pass RAMFile::MateOnSameRef() to GetPNEXT() and GetTLEN()
  (#113).

Full list of changes: CHANGELOG.md.