Skip to content

RAMTools 0.1.0

Latest

Choose a tag to compare

@mvassilev mvassilev released this 14 Oct 09:18
48fa5be

The first tagged release. RAMTools began in 2017 at CERN openlab as a format
stored in ROOT's TTree; this release is its rewrite on RNTuple. Files written by
it stay readable by every later release.

Added

  • The RAM format on ROOT's RNTuple and samtoramntuple, which converts SAM to
    it (#1, #2).
  • bamtoramntuple, which converts BAM through htslib (#20).
  • ramdump, which writes a file or a region of it back out as SAM and takes the
    samtools view options -h, -H, -c, -f, -F and -o (#75).
  • ramntupleview, which counts the records in a region and reports the query
    time (#10, #69).
  • -split, one output file per reference, written as a stream and indexed like
    any other file (#9, #79).
  • -compression N on both converters, taking a ROOT compression code (#73).
  • -threads N on both converters; samtoramntuple parses, encodes and
    compresses 64 MB blocks on every thread and writes them in input order
    (#86, #87, #88).
  • Region queries find the region by binary search over the positions of a
    coordinate-sorted file; the converters record whether the input was sorted,
    and an unsorted file is read in full (#79, #83).
  • Benchmarks: a SAM generator and Google Benchmark suites for conversion and
    region queries (#4, #6, #7, #8), and scripts that run them on real data
    (#54).
  • Continuous integration: build and tests, clang-tidy review, formatting checks
    and Codecov gating (#3, #5, #11, #17, #18, #19, #21, #32, #35,
    #47, #57, #77).
  • Tests for the index, the quality encodings and the record getters (#39,
    #41, #43).
  • The Apache License 2.0 (#60); CONTRIBUTING.md, issue and pull request
    templates, and a README that walks through installation, conversion and
    queries (#12, #89).

Changed

  • ROOT 6.38 is the minimum version (#76).
  • SEQ is stored as uppercase text instead of 4-bit packed bases, which
    compresses better: 3.9% smaller files on chromosome 11. Files with packed
    sequences still read (#85).
  • Opening a RAM file no longer starts ROOT's interpreter. On 2 million records
    of chromosome 11, counting a 10 Mb region went from 0.59 s and 150 MB to
    0.26 s and 19 MB (#92), and dumping a 1 kb region from 0.61 s and 204 MB to
    0.30 s and 78 MB (#93). -h and -H still start it, to read the header.
  • Region queries decide overlap by each read's CIGAR span and accept rname,
    rname:pos, rname:start-end and * (#15).

Removed

  • The TTree backend (#71).
  • RAMNTupleConverter, replaced by ramdump (#75).
  • The sparse index of mapped reads (#16), replaced by the binary search
    (#83).

Fixed

  • Sequences decoded wrongly by DecodeSequence (#45).
  • The SAM parser accepted non-numeric and out-of-range integer fields, and a
    query for an unknown reference added it to the reference list (#59).
  • Float tags lost precision (#64).
  • A CIGAR of * came back as an empty field (#66).
  • Illumina quality binning was indexed by the ASCII character instead of the
    Phred score (#68).
  • Fields lost or shifted on the way in or out: lines over 10 kB, consecutive
    tabs, unknown and lowercase bases, unknown CIGAR operators (#70).
  • The SAM header was written as one key per line and could not be read back
    (#72).
  • A mapped read with a malformed CIGAR such as 10Q was stored without one; it
    is now reported with its line number and skipped (#74).
  • A region query past the last index anchor of a reference started from the
    first row and read the whole file (#82); the index was later replaced by the
    binary search (#83).
  • README examples that did not run (#78).
  • bamtoramntuple did not link when htslib is not in a default library
    directory, as on macOS (#95).
  • clang-tidy findings and suppressions (#14, #38).

Full list of changes: CHANGELOG.md.