1.17

Download the source code here: samtools-1.17.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• New samtools reset subcommand. Removes alignment information. Alignment location, CIGAR, mate mapping and flags are updated. If the alignment was in reverse direction, sequence and its quality values are reversed and complemented and the reverse flag is reset. Supplementary and secondary alignment data are discarded. (PR #1767, implements #1682. Requested by dkj)

• New samtools cram-size subcommand. It writes out metrics about a CRAM file reporting aggregate sizes per block "Content ID" fields, the data-series contained within them, and the compression methods used. (PR #1777)

• Added a --sanitize option to fixmate and view. This performs some sanity checks on the state of SAM record fields, fixing up common mistakes made by aligners. (PR #1698)

• Permit 1 thread with samtools view. All other subcommands already allow this and it does provide a modest speed increase. (PR #1755, fixes #1743. Reported by Goran Vinterhalter)

• Add CRAM_OPT_REQUIRED_FIELDS option for view -c. This is a big speed up for CRAM (maybe 5-fold), but it depends on which filtering options are being used. (PR #1776, fixes #1775. Reported by Chang Y)

• New filtering options in samtools depth. The new --excl-flags option is a synonym for -G, with --incl-flags and --require-flags added to match view logic. (PR #1718, fixes #1702. Reported by Dario Beraldi)

• Speed up calmd's slow handling of non-position-sorted data by adding caching. This uses more memory but is only activated when needed. (PR #1723, fixes #1595. Reported by lxwgcool)

• Improve samtools consensus for platforms with instrument specific profiles, considerably helping for data with very different indel error models and providing base quality recalibration tables. On PacBio HiFi, ONT and Ultima Genomics consensus qualities are also redistributed within homopolymers and the likelihood of nearby indel errors is raised. (PR #1721, PR #1733)

• Consensus --mark-ins option. This permits he consensus output to include a markup indicating the next base is an insertion. This is necessary as we need a way of outputting both consensus and also how that consensus marries up with the reference coordinates. (PR #1746)

• Make faidx/fqidx output line length default to the input line length. (PR #1738, fixes #1734. Reported by John Marshall)

• Speed up optical duplicate checking where data has a lot of duplicates compared to non-duplicates. (PR #1779, fixes #1771. Reported by Poshi)

• For collate use TMPDIR environment variable, when looking for a temporary folder. (PR #1782, based on PR #1178 and fixes #1172. Reported by Martin Pollard)

Bug Fixes:

• Fix stats breakage on long deletions when given a reference. (PR #1712, fixes #1707. Reported by John Didion)

• In ampliconclip, stop hard clipping from wrongly removing entire reads. (PR #1722, fixes #1717. Reported by Kevin Xu)

• Fix bug in ampliconstats where references mentioned in the input file headers but not in the bed file would cause it to complain that the SAM headers were inconsistent. (PR #1727, fixes #1650. Reported by jPontix)

• Fixed SEGV in samtools collate when no filename given. (PR #1724)

• Changed the default UMI barcode regex in markdup. The old regex was too restrictive. This version will at least allow the default read name UMI as given in the Illumina example documentation. (PR #1737, fixes #1730. Reported by yloemie)

• Fix samtools consensus buffer overrun with MD:Z handling. (PR #1745, fixes #1744. Reported by trilisser)

• Fix a buffer read-overflow in mpileup and tview on sequences with seq "*". (PR #1747)

• Fix view -X command line parsing that was broken in 1.15. (PR #1772, fixes #1720. Reported by Francisco Rodríguez-Algarra and Miguel Machado)

• Stop samtools view -d from reporting meaningless system errors when tag validation fails. (PR #1796)

Documentation:

• Add a description of the samtools tview display layout to the man page. Documents . vs , and upper vs lowercase. Adds a -s sample example, and documents the -w option. (PR #1765, fixes #1759. Reported by Lucas Ferreira da Silva)

• Clarify intention of samtools fasta/q in man page and soft vs hard clipping. (PR #1794, fixes #1792. Reported by Ryan Lorig-Roach)

• Minor fix to wording of mpileup --rf usage and man page. (PR #1795, fixes #1791. Reported by Luka Pavageau)

Non user-visible changes and build improvements:

• Use POSIX grep in testing as egrep and fgrep are considered obsolete. (PR #1726, thanks to David Seifert)

• Switch MacOS CI tests to an ARM-based image. (PR #1770)

1.16.1

Download the source code here: samtools-1.16.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug fixes:

• Fixed a bug with the template-coordinate sort which caused incorrect ordering when using threads, or processing large files that don't fit completely in memory. (PR #1703, thanks to Nils Homer)

• Fixed a crash that occurred when trying to use samtools merge in template-coordinate mode. (PR #1705, thanks to Nils Homer)

1.16

Download the source code here: samtools-1.16.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• samtools reference command added. This subcommand extracts the embedded reference out of a CRAM file. (PR #1649, addresses #723. Requested by Torsten Seemann)

• samtools import now adds grouped by query-name to the header. (PR #1633, thanks to Nils Homer)

• Made samtools view read error messages more generic. Former error message would claim that there was a "truncated file or corrupt BAM index file" with no real justification. Also reset errno in stream_view which could lead to confusing error messages. (PR #1645, addresses some of the issues in #1640. Reported by Jian-Guo Zhou)

• Make samtools view -p also clear mqual, tlen and cigar. (PR #1647, fixes #1606. Reported by eboyden)

• Add bedcov option -c to report read count. (PR #1644, fixes #1629. Reported by Natchaphon Rajudom)

• Add UMI/barcode handling to samtools markdup. (PR #1630, fixes #1358 and #1514. Reported by Gert Hulselmans and Poshi)

• Add a new template coordinate sort order to samtools sort and samtools merge. This is useful when working with unique molecular identifiers (UMIs). (PR #1605, fixes #1591. Thanks to Nils Homer)

• Rename mpileup --ignore-overlaps to --ignore-overlaps-removal or --disable-overlap-removal. The previous name was ambiguous and was often read as an option to enable removal of overlapping bases, while in reality this is on by default and the option turns off the ability to remove overlapping bases. (PR #1666, fixes #1663. Reported by yangdingyangding)

• The dict command can now read BWA's .alt file and add AH:* tags indicating reference sequences that represent alternate loci. (PR #1676. Thanks to John Marshall)

• The samtools index command can now accept multiple alignment filenames with the new -M option, and will index each of them separately. (Specifying the output index filename via out.index or the new -o option is currently only applicable when there is only one alignment file to be indexed.) (PR #1674. Reported by Abigail Ramsøe and Nicola Romanò. Thanks to John Marshall)

• Allow samtools fastq -T "*". This allows all tags from SAM records to be written to fastq headers. This is a counterpart to samtools import -T "*". (PR #1679. Thanks to cjw85)

Bug Fixes:

• Re-enable --reference option for samtools depth. The reference is not used but this makes the command line usage compatible with older releases. (PR #1646, fixes #1643. Reported by Randy Harr)

• Fix regex coordinate bug in samtools markdup. (PR #1657, fixes #1642. Reported by Randy Harr)

• Fix divide by zero in plot-bamstats -m, on unmapped data. (PR #1678, fixes #1675. Thanks to Shane McCarthy)

• Fix missing RG headers when using samtools merge -r. (PR #1683, addresses htslib#1479. Reported by Alex Leonard)

• Fix a possible unaligned access in samtools reference. (PR #1696)

Documentation:

• Add documentation on CRAM compression profiles and some of the newer options that appear in CRAM 3.1 and above. (PR #1659, fixes #1656. Reported by Matthias De Smet)

• Add sclen filter expression keyword documentation. (PR #1661, see also htslib#1441)

• Extend FILTER EXPRESSION man page section to match the changes made in HTSlib. (PR #1687, samtools/htslib#1478)

Non user-visible changes and build improvements:

• Ensure generated test files are ignored (by git) and cleaned (by make testclean) (PR #1692, Thanks to John Marshall)

1.15.1

Download the source code here: samtools-1.15.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug fixes:

• A bug which prevented the samtools view --region-file (and the equivalent -M -L <file>) options from working in version 1.15 has been fixed. (#1617)

• Fixed a crash triggered by using the samtools view -c/--count and --unmap options together. The --unmap option is now ignored in counting mode. (#1619)

Documentation:

• The consensus command was missing from the main samtools.1 manual page. It has now been added. (#1603)

• Corrected instructions for reproducing the samtools stats "raw total sequences" count using samtools view -c. (#1620; reported by krukanna)

• Improved manual page formatting. (#1625; thanks to John Marshall)

Non user-visible changes and build improvements:

• Unnecessary #include lines have been removed from bam_plcmd.c. (#1607; thanks to John Marshall)

1.15

Download the source code here: samtools-1.15.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Notice:

• Samtools mpileup VCF and BCF output (deprecated in release 1.9) has been removed. Please use bcftools mpileup instead.

New work and changes:

• Added --min-BQ and --min-MQ options to depth. These match the equivalent long options found in "samtools mpileup" and gives a consistent way of specifying the base and mapping quality filters. (#1584; fixes #1580. Reported by Chang Y)

• Improved automatic file type detection with view -u or view -1. Setting either of these options would default to BAM format regardless of the usual automatic file type selection based on the file name. The defaults are now only used when the file name does not indicate otherwise. (#1582)

• For markdup optical duplicate marking add regex options for custom coordinates. For the case of non standard read names (QNAME), add options to read the coordinates and, optionally, another part of the string to test for optical duplication. (#1558)

• New samtools consensus subcommand for generating consensus from SAM, BAM or CRAM files based on the contents of the alignment records. The consensus is written as FASTA, FASTQ or as a pileup oriented format. The default FASTA/FASTQ output includes one base per non-gap consensus, with insertions with respect to the aligned reference being included and deletions removed. This could be used to compute a new reference from sequence assemblies to realign against. (#1557)

• New samtools view --fetch-pairs option. This options retrieves pairs even when the mate is outside of the requested region. Using this option enables the multi-region iterator and a region to search must be specified. The input file must be an indexed regular file. (#1542)

• Building on #1530 below, add a tview reflist for Goto. (#1539, thanks to Adam Blanchet)

• Completion of references added to tview Goto. (#1530; thanks to Adam Blanchet)

• New samtools head subcommand for conveniently displaying the headers of a SAM, BAM, or CRAM file. Without options, this is equivalent to samtools view --header-only --no-PG but more succinct and memorable. (#1517; thanks to John Marshall)

Bug Fixes:

• Free memory when stats fails to read the header of a file. (#1592; thanks to Mathias Schmitt)

• Fixed empty field on unsupported aux tags in mpileup --output-extra. Replaces the empty fields on unsupported aux tags with a *. (#1553; fixes #1544. Thanks to Adam Blanchet)

• In mpileup, the --output-BP-5 and --output-BP are no longer mutually exclusive. This fixes the problem of output columns being switched. (#1540; fixes 1534. Reported by Konstantin Riege)

• Fix for hardclip bug in ampliconclip. Odd length sequences resulted in random characters appearing in sequence. (#1538; fixes #1527. Reported by Ivana Mihalek)

Documentation:

• Improved mpileup documentation. (#1566; fixes #1564. Reported by Chang Y)

• Fixed samtools depth -J documentation, which was reversed. (#1552; fixes #1549. Reported by Stephan Hutter)

• Numerous minor man page fixes. (#1528, #1536, #1579, #1590. Thanks to John Marshall for some of these)

Non user-visible changes and build improvements:

• Replace CentOS test build with Rocky Linux. The CentOS Docker images that our test build depended on has stopped working. Switched to Rocky Linux as the nearest available equivalent. (#1589)

• Fix missing autotools on Appveyor. Newer versions of msys2 removed autotools from their base-devel package. This is putting them back. (#1575)

• Fixed bug detected by clang-13 with -Wformat-security. (#1553)

• Switch to using splaysort in bam_lpileup. Improves speed and efficiency in tview. (#1548; thanks to Adam Blanchet)

1.14

Download the source code here: samtools-1.14.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Notice

• Samtools mpileup VCF and BCF output (deprecated in release 1.9) will be removed in the next release. Please use bcftools mpileup instead.

New work and changes

• The legacy samtools API (libbam.a, bam_endian.h, sam.h and most of bam.h) has been removed. We recommend coding against the HTSlib API directly. The legacy API had not been actively maintained since 2015. (#1483)

• New samtools samples command to list the samples used in a SAM/BAM/CRAM file. (#1432; thanks to Pierre Lindenbaum)

• mpileup now supports base modifications via the SAM Mm/MM auxiliary tag. Please see the --output-mods option. (#1311)

• Added mpileup --output-BP-5 option to output the BP field in 5' to 3' order instead of left to right. (#1484; fixes #1481)

• Added samtools view --rf option as an additional FLAG filtering method. This keeps records only if (FLAG & N) != 0. (#1508; fixes #1470)

• New samtools import -N option to use the second word on a FASTQ header line, matching the SRA/ENA FASTQ variant. (#1485)

• Improve view -x option to simplify specifying multiple tags, and added the reverse --keep-tag option to include rather than exclude. (#516)

• Switched the processing order of view -x (tag filtering) and -e (expression) handling. Expressions now happen first so we can filter on tags which are about to be deleted. This is now consistent with the view -d behaviour too. (#1480; fixes #1476. Reported by William Rowell)

• Added filter expression endpos keyword. (#1464. Thanks to John Marshall)

• samtools view errors now appear after any SAM output, improving their visibility. (#1490. Thanks to John Marshall)

• Improved samtools sort use of temporary files, both tidying up if it fails and recovery when facing pre-existing temporary files. (#1510; fixes #1035, #1503. Reported by Vivek Rai and Maarten Kooyman)

• Filtering in samtools view now sets the UNMAP BAM flag when given the -p option. (#1512; fixes #1469)

• Make CRAM references shared during samtools merge so merging many files has a lower memory usage. (#471)

Bug fixes

• Prevent samtools depth from closing stdout when outputting to terminal, avoiding a bad interaction with PySam. (#1465. Thanks to John Marshall)

• In-place samtools reheader now works on CRAMs produced using a higher than default compression level. (#1479)

• Fix setting of the dt tag in markdup. Optical duplicates were being marked too early, negating the tagging and counting elsewhere. (#1487; fixes #1486. Reported by Kevin Lewis)

• Reinstate the samtools stats -I option to filter by sample. (#1496; fixes #1489. Reported by Matthias Bernt)

• Fix "samtools fastq" handling of dual index tags on single-ended input. (#1474)

• Improve "samtools coverage" documentation. (#1521; fixes #1504. Reported by Peter Menzel)

Non user-visible changes and build improvements

• Replace Curses mvprintw() with va_list-based equivalent. (#1509. Thanks to John Marshall and Andreas Tille)

• Fixed some clang-13 warning messages. (#1506)

• Improve quoting of options in samtools import tests. (#1466. Thanks to John Marshall)

• Fixed a faulty test which caused test harness failures on NetBSD. (#1520)

1.13

Download the source code here: samtools-1.13.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

• Fixed samtools view FILE REGION, mpileup -r REGION, coverage -r REGION and other region queries: fixed bug introduced in 1.12, which led to region queries producing very few reads for some queries (especially for larger target regions) when unmapped reads were present. Thanks to @vinimfava (#1451), @JingGuo1997 (#1457) and Ramprasad Neethiraj (#1460) for reporting the respective issues.

• Added options to set and clear flags to samtools view. Along with the existing remove aux tags this gives the ability to remove mark duplicate changes (part of #1358) (#1441)

• samtools view now has long option equivalents for most of its single-letter options. Thanks to John Marshall. (#1442)

• A new tool, samtools import, has been added. It reads one or more FASTQ files and converts them into unmapped SAM, BAM or CRAM. (#1323)

• Fixed samtools coverage error message when the target region name is not present in the file header. Thanks to @Lyn16 for reporting it. (#1462; fixes #1461)

• Made samtools coverage ASCII mode produce true ASCII output. Previously it would produce UTF-8 characters. (#1423; fixes #1419)

• samtools coverage now allows setting the maximum depth, using the -d/--depth option. Also, the default maximum depth has been set to 1000000. (#1415; fixes #1395)

• Complete rewrite of samtools depth. This means it is now considerably faster and does not need a depth limit to avoid high memory usage. Results should mostly be the same as the old command with the potential exception of overlap removal. (#1428; fixes #889, helps ameliorate #1411)

• samtools flags now accepts any number of command line arguments, allowing multiple SAM flag combinations to be converted at once. Thanks to John Marshall. (#1401, fixes #749)

• samtools ampliconclip, ampliconstats and plot-ampliconstats now support inputs that list more than one reference. (#1410 and #1417; fixes #1396 and #1418)

• samtools ampliconclip now accepts the --tolerance option, which allows the user to set the number of bases within which a region is matched. The default is 5. (#1456)

• Updated the documentation on samtools ampliconclip to be clearer about what it does. From a suggestion by Nathan S Watson-Haigh. (#1448)

• Fixed negative depth values in ampliconstats output. (#1400)

• samtools addreplacerg now allows for updating (replacing) an existing @RG line in the output header, if a new @RG line is provided in the command line, via the -r argument. The update still requires the user's approval, which can be given with the new -w option. Thanks to Chuang Yu. (#1404)

• Stopped samtools cat from outputting multiple CRAM EOF markers. (#1422)

• Three new counts have been added to samtools flagstat: primary, mapped primary and duplicate primary. (#1431; fixes #1382)

• samtools merge now accepts a -o FILE option specifying the output file, similarly to most other subcommands. The existing way of specifying it (as the first non-option argument, alongside the input file arguments) remains supported. Thanks to David McGaughey and John Marshall. (#1434)

• The way samtools merge checks for existing files has been changed so that it does not hang when used on a named pipe. (#1438; fixes #1437)

• Updated documentation on mpileup to highlight the fact that the filtering options on FLAGs work with ANY rules. (#1447; fixes #1435)

• samtools can now be configured to use a copy of HTSlib that has been set up with separate build and source trees. When this is the case, the --with-htslib configure option should be given the location of the HTSlib build tree. (Note that samtools itself does not yet support out-of-tree builds). Thanks to John Marshall. (#1427; companion change to samtools/htslib#1277)

1.12

Download the source code here: samtools-1.12.tar.bz2.
(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

• The legacy samtools API (libbam.a, bam.h, sam.h, etc) has not been actively maintained since 2015. It is deprecated and will be removed entirely in a future SAMtools release. We recommend coding against the HTSlib API directly.

• I/O errors and record parsing errors during the reading of SAM/BAM/CRAM files are now always detected. Thanks to John Marshall (#1379; fixed #101)

• New make targets have been added: check-all, test-all, distclean-all, mostlyclean-all, testclean-all, which allow SAMtools installations to call corresponding Makefile targets from embedded HTSlib installations.

• samtools --version now displays a summary of the compilation details and available features, including flags, used libraries and enabled plugins from HTSlib. As an alias, samtools version can also be used. (#1371)

• samtools stats now displays the number of supplementary reads in the SN section. Also, supplementary reads are no longer considered when splitting read pairs by orientation (inward, outward, other). (#1363)

• samtools stats now counts only the filtered alignments that overlap target regions, if any are specified. (#1363)

• samtools view now accepts option -N, which takes a file containing read names of interest. This allows the output of only the reads with names contained in the given file. Thanks to Daniel Cameron. (#1324)

• samtools view -d option now works without a tag associated value, which allows it to output all the reads with the given tag. (#1339; fixed #1317)

• samtools view -d and -D options now accept integer and single character values associated with tags, not just strings. Thanks to @darioMe and Keiran Raine for the suggestions. (#1357, #1392)

• samtools view now works with the filtering expressions introduced by HTSlib. The filtering expression is passed to the program using the specific option -e or the global long option --input-fmt-option. E.g.

  samtools view -e 'qname =~ "#49$" && mrefid != refid && refid != -1 &&  mrefid != -1' align.bam
  

  looks for records with query-name ending in #49 that have their mate aligned in a different chromosome. More details can be found in the FILTER EXPRESSIONS section of the main man page. (#1346)

• samtools markdup now benefits from an increase in performance in the situation when a single read has tens or hundreds of thousands of duplicates. Thanks to @denriquez for reporting the issue. (#1345; fixed #1325)

• The documentation for samtools ampliconstats has been added to the samtools man page. (#1351)

• A new FASTA/FASTQ sanitizer script (fasta-sanitize.pl) was added, which corrects the invalid characters in the reference names. (#1314) Thanks to John Marshall for the installation fix. (#1353)

• The CI scripts have been updated to recurse the HTSlib submodules when cloning HTSlib, to accommodate for the CRAM codecs, which now reside in the htscodecs submodule. (#1359)

• The CI integrations now include Cirrus-CI rather than Travis. (#1335; #1365)

• Updated the Windows image used by Appveyor to 'Visual Studio 2019'. (#1333; fixed #1332)

• Fixed a bug in samtools cat, which prevented the command from running in multi-threaded mode. Thanks to Alex Leonard for reporting the issue. (#1337; fixed #1336)

• A couple of invalid CIGAR strings have been corrected in the test data. (#1343)

• The documentation for samtools depth -s has been improved. Thanks to @wulj2. (#1355)

• Fixed a samtools merge segmentation fault when it failed to merge header @PG records. Thanks to John Marshall. (#1394; reported by Kemin Zhou in #1393)

• Ampliconclip and ampliconstats now guard against the BED file containing more than one reference (chromosome) and fail when found. Adding proper support for multiple references will appear later. (#1398)

1.11

Download the source code here: samtools-1.11.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

• New samtools ampliconclip sub-command for removing primers from amplicon-based sequencing experiments, including the current COVID-19 projects. The primers are listed in a BED file and can be either soft-clipped or hard-clipped. (#1219)

• New samtools ampliconstats sub-command to produce a textual summary of primer and amplicon usage, in a similar style to samtools stats. The misc/plot-ampliconstats script can generate PNG images based on this text report. (#1227)

• Samtools fixmate, addreplacerg, markdup, ampliconclip and sort now accept a -u option to enable uncompressed output, which is useful when sending data over a pipe to another process. Other subcommands which already support this option for the same purpose are calmd, collate, merge, view and depad. (#1265)

• samtools stats has a new GCT section, where it reports ACGT content percentages, similar to GCC but taking into account the read orientation. (#1274)

• samtools split now supports splitting by tag content with the -d option (#1211)

• samtools merge now accepts a BED file as a command line argument (-L) and does the merging only with reads overlapping the specified regions (#1156)

• samtools sort now has a minhash collation (-M) to group unmapped reads with similar sequence together. This can sometimes significantly reduce the file size. (#1093)

• samtools bedcov now has -g and -G options to filter-in and filter-out based on the FLAG field. Also the new -d option adds an extra column per file counting the number of bases with a depth greater than or equal to a given threshold. (#1214)

• Fixed samtools bedcov -j option (discard deletions and ref-skips) with multiple input files (#1212)

• samtools bedcov will now accept BED files with columns separated by spaces as well as tabs (#1246; #1188 reported by Mary Carmack)

• samtools depth can now include deletions (D) when computing the base coverage depth, if the user adds the -J option to the command line (#1163).

• samtools depth will count only the bases of one read, for the overlapping section of a read pair, if the -s option is used in the command line (#1241, thanks to Teng Li).

• samtools depth will now write zeros for the entire reference length, when samtools depth -aa is run on a file with no alignments. (#1252; #1249 reported by Paul Donovan)

• Stopped depth from closing stdout, which triggered test fails in pysam (#1208, thanks to John Marshall).

• samtools view now accepts remote URIs for FASTA and FAI files. Furthermore, the reference and index file can be provided in a single argument, such as samtools view -T ftp://x.com/ref.fa##idx##ftp://y.com/index.fa.fai a.cram (#1176; samtools/htslib#933 reported by @uitde007)

• samtools faidx gets new options --fai-idx and --gzi-idx to allow specification of the locations of the .fai and (if needed) .gzi index files. (#1283)

• The samtools fasta/fastq -T option can now add SAM array (type B) tags to the output header lines. (#1301)

• samtools mpileup can now display MAPQ either as ASCII characters (with -s/--output-MQ; column now restored to its documented order as in 1.9 and previous versions) or comma-separated numbers (with --output-extra MAPQ; in SAM column order alongside other selected --output-extra columns).
  When both -s/--output-MQ and -O/--output-BP are used, samtools 1.10 printed the extra columns in the opposite order. This changes the format produced by 1.10's --output-extra MAPQ. (#1281, thanks to John Marshall; reported by Christoffer Flensburg)

• samtools tview now accepts a -w option to set the output width in text mode (-d T). (#1280)

• The dict command can now add AN tags containing alternative names with "chr" prefixes added to or removed from each sequence name as appropriate and listing both "M" and "MT" alternatives for mitochondria. (#1164, thanks to John Marshall)

• The samtools import command, labelled as obsolete in May 2009 and removed from all help and documentation later that year, has finally been removed. Use samtools view instead. (#1185)

• Replaced the remaining usage of the Samtools 0.1 legacy API with htslib calls. (#1187, thanks to John Marshall)

• Documentation / help improvements (#1154; #1168; #1191; #1199; #1204; #1313):

  • Fixed a few man-page table layout issues
  • Added <file>##idx##<index> filename documentation
  • Fixed usage statement for samtools addreplacerg
  • Miscellaneous spelling and grammar fixes
  • Note fixmate/markdup name collated rather than name sorted input
  • Note that fastq and fasta inputs should also be name collated
  • Reshuffled order of main man-page and added -@ to more sub-pages
  • The misc/seq_cache_populate.pl script now gives REF_CACHE guidance

• Additional documentation improvements, thanks to John Marshall (#1181; #1224; #1248; #1262; #1300)

  • Emphasise that samtools index requires a position-sorted file
  • Document 2^29 chromosome length limit in BAI indexes
  • Numerous typing, spelling and formatting fixes

• Improved the message printed when samtools view fails to read its input (#1296)

• Added build support for the OpenIndiana OS (#1165, thanks to John Marshall)

• Fixed failing tests on OpenBSD (#1151, thanks to John Marshall)

• The samtools sort tests now use less memory so the test suite works better on small virtual machines. (#1159)

• Improved markdup's calculation of insert sizes (#1161) Also improved tests (#1150) and made it run faster when not checking for optical duplicates or adding do tags (#1308)

• Fixed samtools coverage minor inconsistency vs idxstats (#1205; #1203 reported by @calliza)

• Fixed samtools coverage quality thresholding options which were the wrong way round compared to mpileup (-q is the mapping quality threshold and -Q is base quality). (#1279; #1278 reported by @kaspernie)

• Fixed bug where samtools fastq -i would add two copies of the barcode in the fastq header if both reads in a pair had a BC:Z tag (#1309; #1307 reported by @mattsoup)

• samtools calmd no longer errors with a SEQ of * (#1230; #1229 reported by Bob Harris)

• samtools tview now honours $COLUMNS, fixing some CI tests (#1171; #1162 reported by @cljacobs)

• Fixed a samtools depad overflow condition (#1200)

• Improved curses detection in configure script (#1170, #577, #940)

• Fixed samtools stats integer overflows and added support for long references (#1174; #1173)

• Fixed a 1-byte undersized memory allocation in samtools merge. (#1302)

1.10

The samtools-1.10.tar.bz2 download is the full source code release. The “Source code” downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.

---

Changes affecting the whole of samtools, or multiple sub-commands:

• Samtools now uses the new HTSlib header API. As this adds more checks for invalid headers, it is possible that some illegal files will now be rejected when they would have been allowed by earlier versions. (#998)

  Examples of problems that will now be rejected include @SQ lines with no SN: tag, and @RG or @PG lines with no ID: tag.

• samtools sub-commands will now add @PG header lines to output sam/bam/cram files. To disable this, use the --no-PG option. (#1087; #1097)

• samtools now supports alignment records with reference positions greater than 2 gigabases. This allows samtools to process alignments for species which have large chromosomes, like axolotl and lungfish. Note that due to file format limitations, data with large reference positions must use the SAM format. (#1107; #1117)

• Improved the efficiency of reading and writing SAM format data by 2 fold (single thread). This is further improved by the ability to use multiple threads, as previously done with BAM and CRAM.

• samtools can now write BGZF-compressed SAM format. To enable this, either save files with a .sam.gz suffix, or use --output-fmt sam.gz.

• samtools can now index BGZF-compressed SAM files.

• The region parsing code has been improved to handle colons in reference names. Strings can be disambiguated by the use of braces, so for example when reference sequences called chr1 and chr1:100-200 are both present, the regions {chr1}:100-200 and {chr1:100-200} unambiguously indicate which reference is being used. (#864)

• samtools flags, flagstats, idxstats and stats now have aliases flag, flagstat, idxstat and stat. (#934)

• A new global --write-index option has been added. This allows output sam.gz/bam/cram files to be indexed while they are being written out. This should work with addreplacerg, depad, markdup, merge, sort, split, and view. (#1062)

• A global --verbosity option has been added to enable/disable debugging output. (#1124, thanks to John Marshall)

• It is now possible to have data and index files stored in different locations. There are two ways to tell samtools where to find the index:

  1. Samtools bedcov, depth, merge, mpileup, stats, tview, and view accept a new option (-X). When this is used, each input sam/bam/cram listed on the command line should have a corresponding index file. Note that all the data files should be listed first, followed by all the index files. (#978, thanks to Mingfei Shao)

• 2. A delimiter ##idx## can be appended to the data file name followed by the index file name. This can be used both for input files and outputs when indexing on-the-fly.

• HTSlib (and therefore SAMtools) now uses version 4 signatures by default for its s3:// plug-in. It can also write to S3 buckets, as long as version 4 signatures are in use. See HTSlib's NEWS file and htslib-s3-plugin manual page for more information.

• HTSlib (and therefore SAMtools) no longer considers a zero-length file to be a valid SAM file. This has been changed so that pipelines such as somecmd | samtools ... with somecmd aborting before outputting anything will now propagate the error to the second command.

• The samtools manual page has been split up into one for each sub-command. The main samtools.1 manual page now lists the sub-commands and describes the common global options. (#894)

• The meaning of decode_md, store_md and store_nm in the fmt-option section of the samtools.1 man page has been clarified. (#898, thanks to Evan Benn)

• Fixed numerous memory leaks. (#892)

• Fixed incorrect macro definition on Windows. (#950)

• bedcov, phase, misc/ace2sam and misc/wgsim now check for failure to open files. (#1013, thanks to Julie Blommaert and John Marshall)

Changes affecting specific sub-commands:

• A new "coverage" sub-command has been added. This prints a tabular format of the average coverage and percent coverage for each reference sequence, as well as number of aligned reads, average mapping quality and base quality. It can also (with the -m option) plot a histogram of coverage across the genome. (#992, thanks to Florian Breitwieser)

• samtools calmd:

  • Reference bases in MD: tags are now converted to upper case. (#981, #988)

• samtools depth:

  • Add new options to write a header to the output (-H) and to direct the output to a file (-o). (#937, thanks to Pierre Lindenbaum)

  • New options -g and -G can be used to filter reads. (#953)

  • Fix memory leak when failing to set CRAM options. (#985, thanks to Florian Breitwieser)

  • Fix bug when using region filters where the -a option did not work for regions with no coverage. (#1113; #1112 reported by Paweł Sztromwasser)

• samtools fasta and fastq:

  • -1 FILE -2 FILE with the same filename now works properly. (#1042)

  • -o FILE is added as a synonym for -1 FILE -2 FILE. (#1042)

  • The -F option now defaults to 0x900 (SECONDARY,SUPPLEMENTARY). Previously secondary and supplementary records were filtered internally in a way that could not be turned off. (#1042; #939 reported by @finswimmer)

  • Allow reading from a pipe without an explicit - on the command line. (#1042; #874 reported by John Marshall)

  • Turn on multi-threading for bgzf compressed output files. (#908)

  • Fixed bug where the samtools fastq -i would output incorrect information in the Casava tags for dual-index reads. It also now prints the tags for dual indices in the same way as bcl2fastq, using a + sign between the two parts of the index. (#1059; #1047 reported by Denis Loginov)

• samtools flagstat:

  • Samtools flagstat can now optionally write its output in JSON format or as a tab-separated values file. (#1106, thanks to Vivek Rai).

• samtools markdup:

  • It can optionally tag optical duplicates (reads following Illumina naming conventions only). The is enabled with the -d option, which sets the distance for duplicates to be considered as optical. (#1091; #1103; #1121; #1128; #1134)

  • The report stats (-s) option now outputs counts for optical and non-primary (supplementary / secondary) duplicates. It also reports the Picard "estimate library size" statistic. A new -f option can be used to save the statistics in a given file. (#1091)

  • The rules for calling duplicates can be changed using the new --mode option. This mainly changes the position associated with each read in a pair. --mode t (the default) is the existing behaviour where the position used is that of the outermost template base associated with the read. Alternatively --mode s always uses the first unclipped sequence base. In practice, this only makes a difference for read pairs where the two reads are aligned in the same direction. (#1091)

  • A new -c option can be used to clear any existing duplicate tags. (#1091)

  • A new --include-fails option makes markdup include QC-failed reads. (#1091)

  • Fixed buffer overflow in temporary file writer when writing a mixture of long and short alignment records. (#911; #909)

• samtools mpileup:

  • mpileup can now process alignments including CIGAR P (pad) operators correctly. They will now also produce the correct output for alignments where insertions are immediately followed by deletions, or deletions by insertions. Note that due to limitations in HTSlib, they are still unable to output sequences that have been inserted before the first aligned base of a read. (#847; #842 reported by Tiffany Delhomme. See also htslib issue #59 and pull request #699).

  • In samtools mpileup, a deletion or pad on the reverse strand is now marked with a different character (#) than the one used on a forward strand (*), if the --reverse-del option is used. (#1070)

  • New option --output-extra can be used to add columns for user selected alignment fields or aux tags. (#1073)

  • Fixed double-counting of overlapping bases in alignment records with deletions or reference skips longer than twice the insert size. (#989; #987 reported by @dariomel)

  • Improved manual page with documentation about what each output column means. (#1055, thanks to John Marshall)

• samtools quickcheck:

  • Add unmapped (-u) option, which disables the check for @SQ lines in the header. (#920, thanks to Shane McCarthy)

• samtools reheader:

  • A new option -c allows the input header to be passed to a given command. Samtools then takes the output of this command and uses it as the replacement header. (#1007)

  • Make it clear in help message that reheader --in-place only works on CRAM files. (#921, thanks to Julian Gehring)

  • Refuse to in-place reheader BAM files, instead of unexpectedly writing a BAM file to stdout. (#935)

• samtools split:

  • In samtools split, the -u option no longer accepts an extra file name from which a replacement header was read. The two file names were separated using a colon, which caused problems on Windows and prevented the use of URLs. A new -h option has been added to allow the replacement header file to be specified in its own option. (#961)

  • Fixed bug where samtools split would crash if it read a SAM header that contained an @RG line with no ID tag. (#954, reported by @blue-bird1)

• samtools stats:

  • stats will now compute base compositions for BC, CR, OX and RX tags, and quality histograms for QT, CY, BZ and QX tags. (#904)

  • New stats FTC and LTC showing total number of nucleotides for first and last fragments. (#9...