Features and Updates
--------------------
* Support added for remote reference files. fai_path() can take a
remote reference file and will return the corresponding index file.
Remote indexes can be handled by refs_load_fai(). UR tags in @SQ
lines can now be set to remote URIs. (#1017)
* Added tabix --separate-regions option, which adds header comment
lines separating different regions' output records when multiple
target regions are supplied on the command line. (#1108)
* Added tabix --cache option to set a BGZF block cache size. Most
beneficial when the -R option is used and the same blocks need to
be re-read multiple times. (#1053)
* Improved error checking in tabix and added a --verbosity option
so it is possible to change the amount of logging when it runs.
(#1040)
* A note about the maximum chromosome length usable with TBI indexes
has been added to the tabix manual page. Thanks to John Marshall.
(#1070)
* New method vcf_open_mode() changes the opening mode of a variant
file based on its file extension. Similar to sam_open_mode().
(#1096)
* The VCF parser has been made faster and easier to maintain. (#1057)
* bcf_record_check() has been made faster, giving a 15% speed
increase when reading an uncompressed BCF file. (#1130)
* The VCF parser now recognises the "<NON_REF>" symbolic allele
produced by GATK. (#1045)
* Support has been added for simultaneous reading of unindexed
VCF/BCF files when using the synced_bcf_reader interface. Input
files must have the chromosomes in the same order as each other
and be consistent with the order of sequences in the header.
(#1089)
* The VCF and BCF readers will now attempt to fix up invalid INFO/END
tags where the stored END value is less than POS, resulting in an
apparently negative record length. Such files have been generated
by programs which used END incorrectly, and by broken lift-over
processes that failed to update any END tags present. (#1021; fixed
samtools/bcftools#1154)
* The htsFile interface can now detect the crypt4gh encrypted
format (see https://samtools.github.io/hts-specs/crypt4gh.pdf).
If HTSlib is built with external plug-in support, and the
hfile_crypt4gh plug-in is present, the file will be passed to
it for decryption. The plug-in can be obtained from
https://github.com/samtools/htslib-crypt4gh. (#1046)
* hts_srand48() now seeds the same POSIX-standard sequences of
pseudo-random numbers regardless of platform, including on OpenBSD
where plain srand48() produces a different cryptographically-strong
non-deterministic sequence. Thanks to John Marshall. (#1002)
* Iterators now work with 64 bit positions. (#1018)
* Improved the speed of range queries when using BAI indexes by
making better use of the linear index data included in the file.
The best improvement is on low-coverage data. (#1031)
* Alignments which consume no reference bases are now considered to
have length 1. This would make such alignments cover 1 reference
position in the same manner as alignments that are unmapped or
have no CIGAR strings. These alignments can now be returned by
iterator-based queries. Thanks to John Marshall. (#1063; fixed
samtools/samtools#1240, see also samtools/hts-specs#521).
* A bam_set_seqi() function to modify a single base in the BAM
structure has been added. This is a companion function to
bam_seqi(). (#1022)
* Writing SAM format is around 30% faster. (#1035)
* Added sam_format_aux1() which converts a BAM aux tag to a SAM
format string. (#1134)
* bam_aux_update_str() no longer requires NUL-terminated strings. It
is also now possible to create tags containing part of a longer
string. (#1088)
* It is now possible to use external plug-ins in language bindings
that dynamically load HTSlib. Note that a side-effect of this
change is that some plug-ins now link against libhts.so, which
means that they have to be able to find the shared library when
they are started up. Thanks to John Marshall. (#1072)
* bgzf_close(), and therefore hts_close(), will now return non-zero
when closing a BGZF handle on which errors have been detected.
(Part of #1117)
* Added a special case to the kt_fisher_exact() test for when the
table probability is too small to be represented in a double. This
fixes a bug where it would, for some inputs, fail to correctly
determine which side of the distribution the table was on resulting
in swapped p-values being returned for the left- and right-tailed
tests. The two-tailed test value was not affected by this problem.
(#1126)
* Improved error diagnostics in the CRAM decoder (#1042), BGZF
(#1049), the VCF and BCF readers (#1059), and the SAM parser
(#1073).
* ks_resize() now allocates 1.5 times the requested size when it
needs to expand a kstring instead of rounding up to the next
power of two. This has been done mainly to make the inlined
function smaller, but it also reduces the overhead of storing
data in kstrings at the expense of possibly needing a few more
reallocations. (#1129)
CRAM improvements
-----------------
* Delay CRAM crc32 checks until the data actually needs to be used.
With other changes this leads to a 20x speed up in indexing and
other sub-query based actions. (#988)
* CRAM now handles the transition from mapped to unmapped data in a
better way, improving compression of the unmapped data. (#961)
* CRAM can now use libdeflate. (#961)
* Fixed bug in MD tag generation with "b" read feature codes,
causing the numbers in the tag to be too large. Note that
HTSlib never uses this feature code so it is unlikely that this
bug would be seen on real data. The problem was found when
testing against hand-crafted CRAM files. (#1086)
* Fixed a regression where the CRAM multi-region iterator became
much less efficient when using threads. It now works more like
the single iterator and does not preemptively decode the next
container unless it will be used. (#1061)
* Set CRAM default quality in lossy quality modes. If lossy quality
is enabled and 'B', 'q' or 'Q' features are used, CRAM starts off
with QUAL being all 255 (as per BAM spec and "*" quality) and then
modifies individual qualities as dictated by the specific features.
However that then produces ASCII quality " " (space, q=-1) for the
unmodified bases. Instead ASCII quality "?" (q=30) is used, as per
HTSJDK. Quality 255 is still used for sequences with no modifications
at all. (#1094)
Build changes
-------------
These are compiler, configuration and makefile based changes.
* `make all` now also builds htslib_static.mk and
htslib-uninstalled.pc. Thanks to John Marshall. (#1011)
* Various cppcheck-1.90 warnings have been fixed. (#995, #1011)
* HTSlib now prefers its own headers when being compiled, fixing
build failures on machines that already had a system-installed
HTSlib. Thanks to John Marshall. (#1078; fixed #347)
* Define HTSLIB_EXPORT without using a helper macro to reduce the
length of compiler diagnostics that mention exported functions.
Thanks to John Marshall. (#1029)
* Fix dirty default build by including latest pkg.m4 instead of using
aclocal.m4. Thanks to Damien Zammit. (#1091)
* Struct tags have been added to htslib/*.h public typedefs. This
makes it possible to forward declare htsFile without including
htslib/hts.h. Thanks to Lucas Czech and John Marshall. (#1115;
fixed #1106)
* Fixed compiler warnings emitted by the latest gcc and clang
releases when compiling HTSlib, along with some -Wextra warnings
in the public include files. Thanks to John Marshall. (#1066,
#1063, #1083)
Bug fixes
---------
* Fixed hfile_libcurl breakage when using libcurl 7.69.1 or
later. Thanks to John Marshall for tracking down the exact
libcurl change that caused the incompatibility. (#1105; fixed
samtools/samtools#1254 and samtools/samtools#1284)
* Fixed overflows kroundup32() and kroundup_size_t() which caused
them to return zero when rounding up values where the most
significant bit was set. When this happens they now return the
highest value that can be stored (#1044). All of the kroundup
macro definitions have also been gathered together into a
unified implementation (#1051).
* Fixed missing return parameter value in idx_test_and_fetch().
Thanks to Lilian Janin. (#1014)
* Fixed crashes due to inconsistent selection between BGZF and plain
(hFILE) interfaces when reading files. [fuzz] (#1019)
* Added and/or fixed byte swapping code for big-endian platforms.
Thanks to Jun Aruga, John Marshall, Michael R Crusoe and Gianfranco
Costamagna for their help. (#1023; fixed #119 and #355)
* Fixed a problem with multi-threaded on-the-fly indexes which would
occasionally write virtual offsets pointing at the end of a BGZF
block. Attempting to read from such an offset caused EOF to be
incorrectly reported. These offsets are now handled correctly, and
the indexer has been updated to avoid generating them. (#1028;
fixed samtools/samtools#1197)
* In sam_hdr_create(), free newly allocated SN strings when
encountering an error. [fuzz] (#1034)
* Prevent double free in case of idx_test_and_fetch() failure. Thanks
to @fanwayne for the bug report. (#1047; fixed #1033)
* In the header, link a new PG line only to valid chains. Prevents
an explosive growth of PG lines on headers where PG lines are
already present but not linked together correctly. (#1062; fixed
samtools/samtools#1235)
* Also in the header, when calling sam_hdr_update_line(), update
target arrays only when the name or length is changed. (#1007)
* Fixed buffer overflows in CRAM MD5 calculation triggered by files
with invalid compression headers, or files with embedded references
that were one byte too short. [fuzz] (#1024, #1068)
* Fix mpileup regression between 1.9 and 1.10 where overlap detection
was incorrectly skipped on reads where RNEXT, PNEXT and TLEN were
set to the "unavailable" values ("*", 0, 0 in SAM). (#1097)
* kputs() now checks for null pointer in source string. [fuzz]
(#1087)
* Fix potential bcf_update_alleles() crash on 0 alleles. Thanks to
John Marshall. (#994)
* Added bcf_unpack() calls to some bcf_update functions to fix a bug
where updates made after a call to bcf_dup() could be lost. (#1032;
fixed #1030)
* Error message typo "Number=R" instead of "Number=G" fixed in
bcf_remove_allele_set(). Thanks to Ilya Vorontsov. (#1100)
* Fixed crashes that could occur in BCF files that use IDX= header
annotations to create a sparse set of CHROM, FILTER or FORMAT
indexes, and include records that use one of the missing index
values. [fuzz] (#1092)
* Fixed potential integer overflows in the VCF parser and ensured
that the total length of FORMAT fields cannot go over 2Gbytes.
[fuzz] (#1044, #1104)
* Download index files atomically in idx_test_and_fetch(). This
prevents corruption when running parallel jobs on S3 files.
Thanks to John Marshall. (#1112; samtools/samtools#1242).
* The pileup constructor callback is now given the copy of the
bam1_t struct made by pileup instead of the original one passed
to bam_plp_push(). This makes it the same as the one passed to
the destructor and ensures that cached data, for example the
location of an aux tag, will remain valid. (#1127)
* Fixed possible error in code_sort() on negative CRAM Huffman code
length. (#1008)
* Fixed possible undefined shift in
cram_byte_array_stop_decode_init(). (#1009)
* Fixed a bug where range queries to the end of a given reference
would return incorrect results on CRAM files. (#1016; fixed
samtools/samtools#1173)
* Fixed an integer overflow in cram_read_slice(). [fuzz] (#1026)
* Fixed a memory leak on failure in cram_decode_slice(). [fuzz]
(#1054)
* Fixed a regression which caused cram_transcode_rg() to fail,
resulting in a crash in "samtools cat" on CRAM files. (#1093;
fixed samtools/samtools#1276)
* Fixed an undersized string reallocation in the threaded SAM reader
which caused it to crash when reading SAM files with very long
lines. Numerous memory allocation checks have also been added.
(#1117)
