1.17

Download the source code here: htslib-1.17.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

• A new API for iterating through a BAM record's aux field. (PR #1354, addresses #1319. Thanks to John Marshall)

• Text mode for bgzip. Allows bgzip to compress lines of text with block breaks at newlines. (PR #1493, thanks to Mike Lin for the initial version PR #1369)

• Make tabix support CSI indices with large positions. Unlike SAM and VCF files, BED files do not set a maximum reference length which hindered CSI support. This change sets an arbitrary large size of 100G to enable it to work. (PR #1506)

• Add a fai_line_length function. Exposes the internal line-wrap length. (PR #1516)

• Check for invalid barcode tags in fastq output. (PR #1518, fixes samtools/samtools#1728. Reported by Poshi)

• Warn if reference found in a CRAM file is not contained in the specified reference file. (PR #1517 and PR #1521, adds diagnostics for #1515. Reported by Wei WeiDeng)

• Add a faidx_seq_len64 function that can return sequence lengths longer than INT_MAX. At the same time limit faidx_seq_len to INT_MAX output. Also add a fai_adjust_region to ensure given ranges do not go beyond the end of the requested sequence. (PR #1519)

• Add a bcf_strerror function to give text descriptions of BCF errors. (PR #1510)

• Add CRAM SQ/M5 header checking when specifying a fasta file. This is to prevent creating a CRAM that cannot be decoded again. (PR #1522. In response to samtools/samtools#1748 though not a direct fix)

• Improve support for very long input lines (> 2Gbyte). This is mostly useful for tabix which does not do much interpretation of its input. (PR #1542, a partial fix for #1539)

• Speed up load_ref_portion. This function has been sped up by about 7x, which speeds up low-depth CRAM decoding by about 10%. (PR #1551)

• Expand CRAM API to cope with new samtools cram_size command. (PR #1546)

• Merges neighbouring I and D ops into one op within pileup. This means 4M1D1D1D3M is reported as 4M3D3M. Fixing this in sam.c means not only is samtools mpileup now looking better, but any tool using the mpileup API will be getting consistent results. (PR #1552, fixes the last remaining part of samtools/samtools#139)

• Update the API documentation for bgzf_mt as it refered to a previous iteration. (PR #1556, fixes #1553. Reported by Raghavendra Padmanabhan)

Build changes

• Use POSIX grep in testing as egrep and fgrep are considered obsolete. (PR #1509, thanks to David Seifert)

• Switch to building libdefalte with cmake for Cirris CI. (PR #1511)

• Ensure strings in config_vars.h are escaped correctly. (PR #1530, fixes #1527. Reported by Lucas Czech)

• Easier modification of shared library permissions during install. (PR #1532, fixes #1525. Reported by StephDC)

• Fix build on ancient compilers. Added -std=gnu90 to build tests so older C compilers will still be happy. (PR #1524, fixes #1523. Reported by Martin Jakt)

• Switch MacOS CI tests to an ARM-based image. (PR #1536)

• Cut down the number of embed_ref=2 tests that get run. (PR #1537)

• Add symbol versions to libhts.so. This is to aid package developers. (PR #1560 addresses #1505, thanks to John Marshall. Reported by Stefan Bruens)

• htscodecs now updated to v1.4.0. (PR #1563)

• Cleaned up misleading system error reports in test_bgzf. (PR #1565)
  Bug fixes

Bug fixes

• VCF. Fix n-squared complexity in sample line with many adjacent tabs [fuzz]. (PR #1503)

• Improved bcftools detection and reporting of bgzf decode errors. (PR #1504, thanks to Lilian Janin. PR #1529 thanks to Bergur Ragnarsson, fixes #1528. PR #1554)

• Prevent crash when the only FASTA entry has no sequence [fuzz]. (PR #1507)

• Fixed typo in sam.h documentation. (PR #1512, thanks to kojix2)

• Fix buffer read-overrun in bam_plp_insertion_mod. (PR #1520)

• Fix hash keys being left behind by bcf_hdr_remove. (PR #1535, fixes #1533. Reported by Giulio Genovese in #842)

• Make bcf_hdr_idinfo_exists more robust by checking id value exists. (PR #1544, fixes #1538. Reported by Giulio Genovese)

• CRAM improvements. Fixed crash with multi-threaded CRAM. Fixed a bug in the codec parameter learning for CRAM 3.1 name tokeniser. Fixed Cram compression container substitution matrix generation, (PR #1558, PR #1559 and PR #1562)

1.16

Download the source code here: htslib-1.16.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

• Make hfile_s3 refresh AWS credentials on expiry in order to make HTSlib work better with AWS IAM credentials, which have a limited lifespan. (PR #1462 and PR #1474, addresses #344)

• Allow BAM headers between 2GB and 4GB in size once more. This is not permitted in the BAM specification but was allowed in an earlier version of HTSlib. There is now a warning at 2GB and a hard failure at 4GB. (PR #1421, fixes #1420 and samtools/samtools#1613. Reported by John Marshall and R C Mueller)

• Improve error message when failing to load an index. (PR #1468, example of the problem samtools/samtools#1637)

• Permit MM (base modification) tags containing . and ? suffixes. These define implicit vs explicit coordinates. See the SAM tags specification for details. (PR #1423 and PR #1426, fixes #1418. PR #1469, fixes #1466. Reported by cjw85)

• Warn if spaces instead of tabs are detected in a VCF file to prevent confusion. (PR #1328, fixes bcftools#1575. Reported by ketkijoshi278)

• Add an sclen filter expression keyword. This is the length of a soft-clip, both left and right end. It may be combined with qlen (qlen-sclen) to obtain the number of bases in the query sequence that have been aligned to the genome ie it provides a way to compare local-alignment vs global-alignment length. (PR #1441 and PR samtools/samtools#1661, fixes #1436. Requested by Chang Y)

• Improve error messages for CRAM reference mismatches. If the user specifies the wrong reference, the CRAM slice header MD5sum checks fail. We now report the SQ line M5 string too so it is possible to validate against the whole chr in the ref.fa file. The error message has also been improved to report the reference name instead of #num. Finally, we now hint at the likely cause, which counters the misleading samtools supplied error of "truncated or corrupt" file. (PR #1427, fixes samtools/samtools#1640. Reported by Jian-Guo Zhou)

• Expose more of the CRAM API and add new functionality to extract the reference from a CRAM file. (PR #1429 and PR #1442)

• Improvements to the implementation of embedded references in CRAM where no external reference is specified. (PR #1449, addresses some of the issues in #1445)

• The CRAM writer now allows alignment records with RG:Z: aux tags that don't have a corresponding @RG ID in the file header. Previously these tags would have been silently dropped. HTSlib will complain whenever it has to add one though, as such tags do not conform to recommended practice for the SAM, BAM and CRAM formats. (PR #1480, fixes #1479. Reported by Alex Leonard)

• Set tab delimiter in man page for tabix GFF3 sort. (PR #1457. Thanks to Colin Diesh)

• When using libdeflate, the 1...9 scale of BGZF compression levels is now remapped to the 1...12 range used by libdeflate instead of being passed directly. In particular, HTSlib levels 8 and 9 now map to libdeflate levels 10 and 12, so it is possible to select the highest (but slowest) compression offered by libdeflate. (PR #1488, fixes #1477. Reported by Gert Hulselmans)

• The VCF variant API has been extended so that it can return separate flags for INS and DEL variants as well as the existing INDEL one. These flags have not been added to the old bcf_get_variant_types() interface as it could break existing users. To access them, it is necessary to use new functions bcf_has_variant_type() and bcf_has_variant_types(). (PR #1467)

• The missing, but trivial, le_to_u8() function has been added to hts_endian. (PR #1494, Thanks to John Marshall)

• bcf_format_gt() now works properly on big-endian platforms. (PR #1495, Thanks to John Marshall)

Build changes

These are compiler, configuration and makefile based changes.

• Update htscodecs to version 1.3.0 for new SIMD code + various fixes. Updates the htscodecs submodule and adds changes necessary to make HTSlib build the new SIMD codec implementations. (PR #1438, PR #1489, PR #1500)

• Fix clang builds under mingw. Under mingw, clang requires dllexport to be applied to both function declarations and function definitions. (PR #1435, PR #1497, PR #1498 fixes #1433. Reported by teepean)

• Fix curl type warning with gcc 12.1 on Windows. (PR #1443)

• Detect ARM Neon support and only build appropriate SIMD object files. (PR #1451, fixes #1450. Thanks to John Marshall)

• make print-config now reports extra CFLAGS that are needed to build the SIMD parts of htscodecs. These may be of use to third-party build systems that don't use HTSlib's or htscodecs' build infrastructure. (PR #1485. Thanks to John Marshall)

• Fixed some Makefile dependency issues for the check/test targets and plugins. In particular, make check will now build the all target, if not done already, before running the tests. (PR #1496)

Bug fixes

• Fix bug when reading position -1 in BCF (0 in VCF), which is used to indicate telomeric regions. The BCF reader was incorrectly assuming the value stored in the file was unsigned, so a VCF->BCF->VCF round-trip would change it from 0 to 4294967296. (PR #1476, fixes #1475 and bcftools#1753. Reported by Rodrigo Martin)

• Various bugs and quirks have been fixed in the filter expression engine, mostly related to the handling of absent tags, and the is_true flag. Note that as a result of these fixes, some filter expressions may give different results:

  • Fixed and-expressions including aux tag values which could give an invalid true result depending on the order of terms.
  • The expression ![NM] is now true if only NM does not exist. In earlier versions it would also report true for tags like NM:i:0 which exist but have a value of zero.
  • The expression [X1] != 0 is now false when X1 does not exist. Earlier versions would return true for this comparison when the tag was missing.
  • NULL values due to missing tags now propagate through string, bitwise and mathematical operations. Logical operations always treat them as false. (PR #1463, fixes samtools/samtools#1670. Reported by Gert Hulselmans; PR #1478, fixes samtools/samtools#1677. Reported by johnsonzcode)

• Fix buffer overrun in bam_plp_insertion_mod. Memory now grows to the proper size needed for base modification data. (PR #1430, fixes samtools/samtools#1652. Reported by hd2326)

• Remove limit of returned size from fai_retrieve(). (PR #1446, fixes samtools/samtools#1660. Reported by Shane McCarthy)

• Cap hts_getline() return value at INT_MAX. Prevents hts_getline() from returning a negative number (a fail) for very long string length values. (PR #1448. Thanks to John Marshall)

• Fix breakend detection and test bcf_set_variant_type(). (PR #1456, fixes #1455. Thanks to Martin Pollard)

• Prevent arrays of BCF_BT_NULL values found in BCF files from causing bcf_fmt_array() to call exit() as the type is unsupported. These are now tested for and caught by bcf_record_check(), which returns an error code instead. (PR #1486)

• Improved detection of fasta and fastq files that have very long comments following identifiers. (PR #1491, thanks to John Marshall. Fixes samtools/samtools#1689, reported by cjw85)

• Fixed a SEGV triggered by giving a SAM file to samtools import. (PR #1492)

1.15.1

Download the source code here: htslib-1.15.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

• Security fix: Fixed broken error reporting in the sam_cap_mapq() function, due to a missing hts_log() parameter. Prior to this fix it was possible to abuse the log message format string by passing a specially crafted alignment record to this function. (PR#1406)

• HTSlib now uses libhtscodecs release 1.2.2. This fixes a number of bugs where invalid compressed data could trigger usage of uninitialised values. (PR#1416)

• Fixed excessive memory used by multi-threaded SAM output on long reads. (Part of PR#1384)

• Fixed a bug where tabix would misinterpret region specifiers starting at position 0. It will also now warn if the file being indexed is supposed to be 1-based but has positions less than or equal to 0. (PR#1411)

• The VCF header parser will now issue a warning if it finds an INFO header with Type=Flag but Number not equal to 0. It will also ignore the incorrect Number so the flag can be used. (PR#1415)

1.15

Download the source code here: htslib-1.15.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

Features and Updates

• Bgzip now has a --keep option to not remove the input file after compressing. (PR #1331)

• Improved file format detection so some BED files are no longer detected as FASTQ or FASTA. (PR #1350, thanks to John Marshall)

• Added xz (lzma), zstd and D4 formats to the file type detection functions. We don't actively support reading these data types, but function calls and htsfile can detect them. (PR #1340, thanks to John Marshall)

• CRAM now also uses libdeflate for read-names if the libdeflate version is new enough (1.9 onwards). Previously we used zlib for this due to poor performance of libdeflate. This gives a slight speed up and reduction in file size. (PR #1383)

• The VCF and BCF readers will now issue a warning if contig, INFO or FORMAT IDs do not match the formats described in the VCFv4.3 specification. Note that while the invalid names will mostly still be accepted, future updates will convert the warnings to errors causing files including invalid names to be rejected. (PR #1389)

Build changes

These are compiler, configuration and makefile based changes.

• HTSlib now uses libhtscodecs release 1.2.1.

• Improved support for compiling and linking against HTSlib with Microsoft Visual Studio. (PR #1380, #1377, #1375. Thanks to Aidan Bickford and John Marshall)

• Various internal CI improvements.

Bug fixes

• Fixed CRAM index queries for HTSJDK output (PR #1388, reported by Chris Norman). Note this also fixes writing CRAM writing, to match the specification (and HTSJDK), from version 3.1 onwards.

• Fixed CRAM index queries when required-fields settings are selected to ignore CIGARs (PR #1372, reported by Giulio Genovese).

• Unmapped but placed (having chr/pos) are now included in the BAM indices. (PR #1352, thanks to John Marshall)

• CRAM now honours the filename##idx##index nomenclature for specifying non-standard index locations. (PR #1360, reported by Michael Cariaso)

• Minor CRAM v1.0 read-group fix (PR #1349, thanks to John Marshall)

• Permit .fa and .fq file type detection as synonyms for FASTA and FASTQ. (PR#1386).

• Empty VCF format fields are now output ":.:" as instead of "::". (PR #1370)

• Repeated bcf_sr_seek calls now work. (PR #1363, reported by Giulio Genovese)

• bcf_remove_allele_set now works on unpacked BCF records. (PR #1358, reported by Brent Pedersen).

• The hts_parse_decimal() function used to read numbers in region lists is now better at rejecting non-numeric values. In particular it now rejects a lone 'G' instead of interpreting it as '0G', i.e. zero. (PR #1396, PR #1400, reported by SSSimon Yang; thanks to John Marshall).

• Improve support for GPU issues listed by -Wdouble-promotion. (PR #1365, reported by David Seisert)

• Fix example code in header file documentation. (PR #1381, Thanks to Aidan Bickford)

1.14

Download the source code here: htslib-1.14.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

Features and Updates

• Added a keep option to bgzip to leave the original file untouched. This brings bgzip into line with gzip. (PR #1331, thanks to Alex Petty)

• endpos has been added to the filter language, giving the position of the rightmost mapped base as measured by the CIGAR string. For unmapped reads it is the same as pos. (PR #1307, thanks to John Marshall)

• Interfaces have been added to interpret the new base modification tags added to the SAMtags document in samtools/hts-specs#418. (PR #1132)

• New API functions hts_flush()/sam_flush()/bcf_flush() for flushing output htsFile/samFile/vcfFile streams. (PR #1326, thanks to John Marshall)

• The synced_bcf_reader now sorts lines with symbolic alleles by END tag as well as POS. (PR #1321)

• Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and BCF_SR_TARGETS_OVERLAP for better control of records that start outside the desired region but overlap it are handled. Fixes samtools/bcftools#1420 and samtools/bcftools#1421 raised by John Marshall. (PR #1327)

• HTSlib will now accept long-cigar CG:B: tags made by htsjdk which don't quite follow the specification properly (using signed values instead of unsigned). Thanks to Colin Diesh for reporting an example file. (PR #1317)

• The warning printed when the BGZF reader finds a file with no EOF block has been changed to be less alarming. Unfortunately some third-party BGZF encoders don't write EOF blocks at the end of files. Thanks to Keiran Raine for reporting an example file. (PR #1323)

• The FASTA and FASTQ readers get an option to skip over the first item on the header line, and use the second as the read name. It allows the original name to be restored on some of the fastq files served from the European Nucleotide Archive. (PR #1325)

• HTSlib is now more strict when parsing the VCF samples line (beginning #CHROM). It will only accept tabs between the mandatory field names and sample names must be separated with tabs. (PR #1328)

• HTSlib will now warn if it looks like the header has been corrupted by diagnostic messages from the program that made it. This can happen when using nohup, which by default mixes stdout and stderr into the same stream. (PR#1339, thanks to John Marshall)

• File format detection will now recognise signatures for XZ, Zstd and D4 files (note that HTSlib will not read them yet). (PR #1340, thanks to John Marshall)

Build changes

These are compiler, configuration and makefile based changes.

• Some redundant tests have been removed from the test harness, speeding it up. (PR #1308)

• The version.sh script now works better on shallow checkouts. (PR #1324)

• A check-untracked Makefile target has been added to catch untracked files (mostly) left by the test harness. (PR #1324)

Bug fixes

• Fixed a case where flushing the thread pool could very occasionally cause a deadlock. (PR #1309)

• Fixed a bug where some CRAM files could fail to decode if the required_fields option was in use. Thanks to Matt Sexton for reporting the issue. (PR #1314, fixes samtools/samtools#1475)

• Fixed a regression where the S3 plugin could not read public files unless you supplied some Amazon credentials. Thanks to Chris Saunders for reporting. (PR #1332, fixes samtools/samtools#1491)

• Fixed a possible CRAM thread deadlock discovered by Ryan Cai. (PR #1330, fixes #1329)

• Some set-but-unused variables have been removed. (PR #1334)

• Fixed a bug which prevented flag.read2 from working in the filter language unless it was at the end of the expression. Thanks to Vamsi Kodali for reporting the issue. (PR #1342)

• Fixed a memory leak that could happen if CRAM fails to inflate a LZMA block. (PR #1340, thanks to John Marshall)

1.13

Download the source code here: htslib-1.13.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

Features and Updates

• In case a PG header line has multiple ID tags supplied by other applications, the header API now selects the first one encountered as the identifying tag and issues a warning when detecting subsequent ID tags. (#1256; fixed samtools/samtools#1393)

• VCF header reading function (vcf_hdr_read) no longer tries to download a remote index file by default. (#1266; fixes #380)

• Support reading and writing FASTQ format in the same way as SAM, BAM or CRAM. Records read from a FASTQ file will be treated as unmapped data. (#1156)

• Added GCP requester pays bucket access. Thanks to @indraniel. (#1255)

• Made mpileup's overlap removal choose which copy to remove at random instead of always removing the second one. This avoids strand bias in experiments where the +ve and -ve strand reads always appear in the same order. (#1273; fixes samtools/bcftools#1459)

• It is now possible to use platform specific BAQ parameters. This also selects long-read parameters for read lengths bigger than 1kb, which helps bcftools mpileup call SNPs on PacBio CCS reads. (#1275)

• Improved bcf_remove_allele_set. This fixes a bug that stopped iteration over alleles prematurely, marks removed alleles as 'missing' and does automatic lazy unpacking. (#1288; fixes #1259)

• Improved compression metrics for unsorted CRAM files. This improves the choice of codecs when handling unsorted data. (#1291)

• Linear index entries for empty intervals are now initialised with the file offset in the next non-empty interval instead of the previous one. This may reduce the amount of data iterators have to discard before reaching the desired region, when the starting location is in a sequence gap. Thanks to @CarsonH for reporting the issue. (#1286; fixes #486)

• A new hts_bin_level API function has been added, to compute the level of a given bin in the binning index. (#1286)

• Related to the above, a new API method, hts_idx_nseq, now returns the total number of contigs from an index. (#1295 and #1299)

• Added bracket handling to bcf_hdr_parse_line, for use with ##META lines. Thanks to Alberto Casas Ortiz. (#1240)

Build changes

These are compiler, configuration and makefile based changes.

• HTSlib now uses libhtscodecs release 1.1.1.

• Added a curl/curl.h check to configure and improved INSTALL documentation on build options. Thanks to Melanie Kirsche and John Marshall. (#1265; fixes #1261)

• Some fixes to address GCC 11.1 warnings. (#1280, #1284, #1285; fixes #1283)

• Supports building HTSlib in a separate directory. Thanks to John Marshall. (#1277; fixes #231)

• Supports building HTSlib on MinGW 32-bit environments. Thanks to John Marshall. (#1301)

Bug fixes

• Fixed hts_itr_query() et al region queries: fixed bug introduced in HTSlib 1.12, which led to iterators producing very few reads for some queries (especially for larger target regions) when unmapped reads were present. HTSlib 1.11 had a related problem in which iterators would omit a few unmapped reads that should have been produced; cf #1142. Thanks to Daniel Cooke for reporting the issue. (#1281; fixes #1279)

• Removed compressBound assertions on opening bgzf files. Thanks to Gurt Hulselmans for reporting the issue. (#1258; fixed #1257)

• Duplicate sample name error message for a VCF file now only displays the duplicated name rather the entire same name list. (#1262; fixes samtools/bcftools#1451)

• Fix to make samtools cat work on CRAMs again. (#1276; fixes samtools/samtools#1420)

• Fix for a double memory free in SAM header creation. Thanks to @ihsinme. (#1274)

• Prevent assert in bcf_sr_set_regions. Thanks to Dr K D Murray. (#1270)

• Fixed crash in knet_open() etc stubs. Thanks to John Marshall. (#1289)

• Fixed filter expression "cigar" on unmapped reads. Stop treating an empty CIGAR string as an error. Thanks to Chang Y for reporting the issue. (#1298, fixes samtools/samtools#1445)

• Bug fixes in the bundled copy of htscodecs:

  • Fixed an uninitialized access in the name tokeniser decoder. (samtools/htscodecs#23)

  • Fixed a bug with name tokeniser and variable number of names per slice, causing it to incorrectly report an error on certain valid inputs. (samtools/htscodecs#24)

1.12

Download the source code here: htslib-1.12.tar.bz2.
(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

Features and Updates

• Added experimental CRAM 3.1 and 4.0 support. (#929)
  These should not be used for long term data storage as the specification still needs to be ratified by GA4GH and may be subject to changes in format. (This is highly likely for 4.0). However it may be tested using:

  test/test_view -t ref.fa -C -o version=3.1 in.bam -p out31.cram
  

  For smaller but slower files, try varying the compression profile with an additional -o small. Profile choices are fast, normal, small and archive, and can be applied to all CRAM versions.

• Added a general filtering syntax for alignment records in SAM/BAM/CRAM readers. (#1181, #1203)
  An example to find chromosome spanning read-pairs with high mapping quality: 'mqual >= 30 && mrname != rname'
  To find significant sized deletions: 'cigar =~ "[0-9]{2}D"' or 'rlen - qlen > 10'.
  To report duplicates that aren't part of a "proper pair": 'flag.dup && !flag.proper_pair'
  More details are in the samtools.1 man page under "FILTER EXPRESSIONS".

• The knet networking code has been removed. It only supported the http and ftp protocols, and a better and safer alternative using libcurl has been available since release 1.3. If you need access to ftp:// and http:// URLs, HTSlib should be built with libcurl support. (#1200)

• The old htslib/knetfile.h interfaces have been marked as deprecated. Any code still using them should be updated to use hFILE instead. (#1200)

• Added an introspection API for checking some of the capabilities provided by HTSlib. (#1170) Thanks also to John Marshall for contributions. (#1222)

  • hfile_list_schemes: returns the number of schemes found
  • hfile_list_plugins: returns the number of plugins found
  • hfile_has_plugin: checks if a specific plugin is available
  • hts_features: returns a bit mask with all available features
  • hts_test_feature: test if a feature is available
  • hts_feature_string: return a string summary of enabled features

• Made performance improvements to probaln_glocal method, which speeds up mpileup BAQ calculations. (#1188)

  • Caching of reused loop variables and removal of loop invariants
  • Code reordering to remove instruction latency.
  • Other refactoring and tidyups.

• Added a public method for constructing a BAM record from the component pieces. Thanks to Anders Kaplan. (#1159, #1164)

• Added two public methods, sam_parse_cigar and bam_parse_cigar, as part of a small CIGAR API (#1169, #1182). Thanks to Daniel Cameron for input. (#1147)

• HTSlib, and the included htsfile program, will now recognise the old RAZF compressed file format. Note that while the format is detected, HTSlib is unable to read it. It is recommended that RAZF files are uncompressed with gunzip before using them with HTSlib. Thanks to John Marshall (#1244); and Matthew J. Oldach who reported problems with uncompressing some RAZF files (samtools/samtools#1387).

• The S3 plugin now has options to force the address style. It will recognise the addressing_style and host_bucket entries in the respective AWS .credentials and s3cmd .s3cfg files. There is also a new HTS_S3_ADDRESS_STYLE environment variable. Details are in the htslib-s3-plugin.7 man file (#1249).

Build changes

These are compiler, configuration and makefile based changes.

• Added new Makefile targets for the applications that embed HTSlib and want to run its test suite or clean its generated artefacts. (#1230, #1238)

• The CRAM codecs are now obtained via the htscodecs submodule, hence when cloning it is now best to use git clone --recursive. In an existing clone, you may use git submodule update --init to obtain the htscodecs submodule checkout.

• Updated CI test configuration to recurse HTSlib submodules. (#1359)

• Added Cirrus-CI integration as a replacement for Travis, which was phased out. (#1175; #1212)

• Updated the Windows image used by Appveyor to 'Visual Studio 2019'. (#1172; fixed #1166)

• Fixed a buglet in configure.ac, exposed by the release 2.70 of autoconf. Thanks to John Marshall. (#1198)

• Fixed plugin linking on macOS, to prevent symbol conflict when linking with a static HTSlib. Thanks to John Marshall. (#1184)

• Fixed a clang++9 error in cram_io.h. Thanks to Pjotr Prins. (#1190)

• Introduced $(ALL_CPPFLAGS) to allow for more flexibility in setting the compiler flags. Thanks to John Marshall. (#1187)

• Added 'fall through' comments to prevent warnings issued by Clang on intentional fall through case statements, when building with -Wextra flag. Thanks to John Marshall. (#1163)

• Non-configure builds now define _XOPEN_SOURCE=600 to allow them to work when the gcc -std=c99 option is used. Thanks to John Marshall. (#1246)

Bug fixes

• Fixed VCF #CHROM header parsing to only separate columns at tab characters. Thanks to Sam Morris for reporting the issue. (#1237; fixed samtools/bcftools#1408)

• Fixed a crash reported in bcf_sr_sort_set, which expects REF to be present. (#1204; fixed samtools/bcftools#1361)

• Fixed a bcf synced reader bug when filtering with a region list, and the first record for a chromosome had the same position as the last record for the previous chromosome. (#1254; fixed samtools/bcftools#1441)

• Fixed a bug in the overlapping logic of mpileup, dealing with iterating over CIGAR segments. Thanks to @wulj2 for the analysis. (#1202; fixed #1196)

• Fixed a tabix bug that prevented setting the correct number of lines to be skipped in a region file. Thanks to Jim Robinson for reporting it. (#1189; fixed #1186)

• Made bam_itr_next an alias for sam_itr_next, to prevent it from crashing when working with htsFile pointers. Thanks to Torbjörn Klatt for reporting it. (#1180; fixed #1179)

• Fixed once per outgoing multi-threaded block bgzf_idx_flush assertion, to accommodate situations when a single record could span multiple blocks. Thanks to @lacek. (#1168; fixed samtools/samtools#1328)

• Fixed assumption of pthread_t being a non-structure, as permitted by POSIX. Thanks also to John Marshall and Anders Kaplan. (#1167, #1153, #1153)

• Fixed the minimum offset of a BAI index bin, to account for unmapped reads. Thanks to John Marshall for spotting the issue. (#1158; fixed #1142)

• Fixed the CRLF handling in sam_parse_worker method. Thanks to Anders Kaplan. (#1149; fixed #1148)

• Included unistd.h and errno.h directly in HTSlib files, as opposed to including them indirectly, via third party code. Thanks to Andrew Patterson (#1143) and John Marshall (#1145).

1.11

Download the source code here: htslib-1.11.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they are missing some generated files.)

Features and Updates

• Support added for remote reference files. fai_path() can take a remote reference file and will return the corresponding index file. Remote indexes can be handled by refs_load_fai(). UR tags in @SQ lines can now be set to remote URIs. (#1017)

• Added tabix --separate-regions option, which adds header comment lines separating different regions' output records when multiple target regions are supplied on the command line. (#1108)

• Added tabix --cache option to set a BGZF block cache size. Most beneficial when the -R option is used and the same blocks need to be re-read multiple times. (#1053)

• Improved error checking in tabix and added a --verbosity option so it is possible to change the amount of logging when it runs. (#1040)

• A note about the maximum chromosome length usable with TBI indexes has been added to the tabix manual page. Thanks to John Marshall. (#1070)

• New method vcf_open_mode() changes the opening mode of a variant file based on its file extension. Similar to sam_open_mode(). (#1096)

• The VCF parser has been made faster and easier to maintain. (#1057)

• bcf_record_check() has been made faster, giving a 15% speed increase when reading an uncompressed BCF file. (#1130)

• The VCF parser now recognises the <NON_REF> symbolic allele produced by GATK. (#1045)

• Support has been added for simultaneous reading of unindexed VCF/BCF files when using the synced_bcf_reader interface. Input files must have the chromosomes in the same order as each other and be consistent with the order of sequences in the header. (#1089)

• The VCF and BCF readers will now attempt to fix up invalid INFO/END tags where the stored END value is less than POS, resulting in an apparently negative record length. Such files have been generated by programs which used END incorrectly, and by broken lift-over processes that failed to update any END tags present. (#1021; fixed samtools/bcftools#1154)

• The htsFile interface can now detect the crypt4gh encrypted format (see https://samtools.github.io/hts-specs/crypt4gh.pdf). If HTSlib is built with external plug-in support, and the hfile_crypt4gh plug-in is present, the file will be passed to it for decryption. The plug-in can be obtained from https://github.com/samtools/htslib-crypt4gh. (#1046)

• hts_srand48() now seeds the same POSIX-standard sequences of pseudo-random numbers regardless of platform, including on OpenBSD where plain srand48() produces a different cryptographically-strong non-deterministic sequence. Thanks to John Marshall. (#1002)

• Iterators now work with 64 bit positions. (#1018)

• Improved the speed of range queries when using BAI indexes by making better use of the linear index data included in the file. The best improvement is on low-coverage data. (#1031)

• Alignments which consume no reference bases are now considered to have length 1. This would make such alignments cover 1 reference position in the same manner as alignments that are unmapped or have no CIGAR strings. These alignments can now be returned by iterator-based queries. Thanks to John Marshall. (#1063; fixed samtools/samtools#1240, see also samtools/hts-specs#521).

• A bam_set_seqi() function to modify a single base in the BAM structure has been added. This is a companion function to bam_seqi(). (#1022)

• Writing SAM format is around 30% faster. (#1035)

• Added sam_format_aux1() which converts a BAM aux tag to a SAM format string. (#1134)

• bam_aux_update_str() no longer requires NUL-terminated strings. It is also now possible to create tags containing part of a longer string. (#1088)

• It is now possible to use external plug-ins in language bindings that dynamically load HTSlib. Note that a side-effect of this change is that some plug-ins now link against libhts.so, which means that they have to be able to find the shared library when they are started up. Thanks to John Marshall. (#1072)

• bgzf_close(), and therefore hts_close(), will now return non-zero when closing a BGZF handle on which errors have been detected. (Part of #1117)

• Added a special case to the kt_fisher_exact() test for when the table probability is too small to be represented in a double. This fixes a bug where it would, for some inputs, fail to correctly determine which side of the distribution the table was on resulting in swapped p-values being returned for the left- and right-tailed tests. The two-tailed test value was not affected by this problem. (#1126)

• Improved error diagnostics in the CRAM decoder (#1042), BGZF (#1049), the VCF and BCF readers (#1059), and the SAM parser (#1073).

• ks_resize() now allocates 1.5 times the requested size when it needs to expand a kstring instead of rounding up to the next power of two. This has been done mainly to make the inlined function smaller, but it also reduces the overhead of storing data in kstrings at the expense of possibly needing a few more reallocations. (#1129)

CRAM improvements

• Delay CRAM crc32 checks until the data actually needs to be used. With other changes this leads to a 20x speed up in indexing and other sub-query based actions. (#988)

• CRAM now handles the transition from mapped to unmapped data in a better way, improving compression of the unmapped data. (#961)

• CRAM can now use libdeflate. (#961)

• Fixed bug in MD tag generation with b read feature codes, causing the numbers in the tag to be too large. Note that HTSlib never uses this feature code so it is unlikely that this bug would be seen on real data. The problem was found when testing against hand-crafted CRAM files. (#1086)

• Fixed a regression where the CRAM multi-region iterator became much less efficient when using threads. It now works more like the single iterator and does not preemptively decode the next container unless it will be used. (#1061)

• Set CRAM default quality in lossy quality modes. If lossy quality is enabled and B, q or Q features are used, CRAM starts off with QUAL being all 255 (as per BAM spec and * quality) and then modifies individual qualities as dictated by the specific features.
  However that then produces ASCII quality " " (space, q=-1) for the unmodified bases. Instead ASCII quality "?" (q=30) is used, as per HTSJDK. Quality 255 is still used for sequences with no modifications at all. (#1094)

Build changes

These are compiler, configuration and makefile based changes.

• make all now also builds htslib_static.mk and htslib-uninstalled.pc. Thanks to John Marshall. (#1011)

• Various cppcheck-1.90 warnings have been fixed. (#995, #1011)

• HTSlib now prefers its own headers when being compiled, fixing build failures on machines that already had a system-installed HTSlib. Thanks to John Marshall. (#1078; fixed #347)

• Define HTSLIB_EXPORT without using a helper macro to reduce the length of compiler diagnostics that mention exported functions. Thanks to John Marshall. (#1029)

• Fix dirty default build by including latest pkg.m4 instead of using aclocal.m4. Thanks to Damien Zammit. (#1091)

• Struct tags have been added to htslib/*.h public typedefs. This makes it possible to forward declare htsFile without including htslib/hts.h. Thanks to Lucas Czech and John Marshall. (#1115; fixed #1106)

• Fixed compiler warnings emitted by the latest gcc and clang releases when compiling HTSlib, along with some -Wextra warnings in the public include files. Thanks to John Marshall. (#1066, #1063, #1083)

Bug fixes

• Fixed hfile_libcurl breakage when using libcurl 7.69.1 or later. Thanks to John Marshall for tracking down the exact libcurl change that caused the incompatibility. (#1105; fixed samtools/samtools#1254 and samtools/samtools#1284)

• Fixed overflows kroundup32() and kroundup_size_t() which caused them to return zero when rounding up values where the most significant bit was set. When this happens they now return the highest value that can be stored (#1044). All of the kroundup macro definitions have also been gathered together into a unified implementation (#1051).

• Fixed missing return parameter value in idx_test_and_fetch(). Thanks to Lilian Janin. (#1014)

• Fixed crashes due to inconsistent selection between BGZF and plain (hFILE) interfaces when reading files. [fuzz] (#1019)

• Added and/or fixed byte swapping code for big-endian platforms. Thanks to Jun Aruga, John Marshall, Michael R Crusoe and Gianfranco Costamagna for their help. (#1023; fixed #119 and #355)

• Fixed a problem with multi-threaded on-the-fly indexes which would occasionally write virtual offsets pointing at the end of a BGZF block. Attempting to read from such an offset caused EOF to be incorrectly reported. These offsets are now handled correctly, and the indexer has been updated to avoid generating them. (#1028; fixed samtools/samtools#1197)

• In sam_hdr_create(), free newly allocated SN strings when encountering an error. [fuzz] (#1034)

• Prevent double free in case of idx_test_and_fetch() failure. Thanks to @fanwayne for the bug report. (#1047; fixed #1033)

• In the header, link a new PG line only to valid chains. Prevents an explosive growth of PG lines on headers where PG lines are already present but not linked together correctly. (#1062; fixed samtools/samtools#1235)

• Also in the header, when calling sam_hdr_update_line(), update target arrays only when the name or length is changed. (#1007)

• Fixed buffer overflows in CRAM MD5 calculation triggered by files...

1.10.2

Download the source code here: htslib-1.10.2.tar.bz2.
(The “Source code” downloads are generated by GitHub and are incomplete as they are missing some generated files.)

---

This release fixes crashes reported on files including integer INFO tags with values outside the range officially supported by VCF. It also fixes a bug where invalid BCF files would be created if such values were present.

1.10.1

(Note: The “Source code” downloads below are generated by GitHub and are incomplete as they are missing some generated files.)

---

This release has been withdrawn due to inconsistencies with the generated tar files.