forked from samtools/samtools
-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
1927 lines (1289 loc) · 72.4 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release a.b
-----------
Release 1.13 (7th July 2021)
----------------------------
* Fixed samtools view FILE REGION, mpileup -r REGION, coverage -r REGION and
other region queries: fixed bug introduced in 1.12, which led to region
queries producing very few reads for some queries (especially for larger
target regions) when unmapped reads were present.
Thanks to @vinimfava (#1451), @JingGuo1997 (#1457) and Ramprasad Neethiraj
(#1460) for reporting the respective issues.
* Added options to set and clear flags to samtools view. Along with the
existing remove aux tags this gives the ability to remove mark duplicate
changes (part of #1358)
(#1441)
* samtools view now has long option equivalents for most of its single-letter
options. Thanks to John Marshall.
(#1442)
* A new tool, samtools import, has been added. It reads one or more FASTQ
files and converts them into unmapped SAM, BAM or CRAM.
(#1323)
* Fixed samtools coverage error message when the target region name is not
present in the file header. Thanks to @Lyn16 for reporting it.
(#1462; fixes #1461)
* Made samtools coverage ASCII mode produce true ASCII output. Previously it
would produce UTF-8 characters.
(#1423; fixes #1419)
* samtools coverage now allows setting the maximum depth, using the -d/--depth
option. Also, the default maximum depth has been set to 1000000.
(#1415; fixes #1395)
* Complete rewrite of samtools depth. This means it is now considerably faster
and does not need a depth limit to avoid high memory usage. Results should
mostly be the same as the old command with the potential exception of overlap
removal.
(#1428; fixes #889, helps ameliorate #1411)
* samtools flags now accepts any number of command line arguments,
allowing multiple SAM flag combinations to be converted at once. Thanks to
John Marshall.
(#1401, fixes #749)
* samtools ampliconclip, ampliconstats and plot-ampliconstats now support
inputs that list more than one reference.
(#1410 and #1417; fixes #1396 and #1418)
* samtools ampliconclip now accepts the --tolerance option, which allows the
user to set the number of bases within which a region is matched. The
default is 5.
(#1456)
* Updated the documentation on samtools ampliconclip to be clearer about what
it does. From a suggestion by Nathan S Watson-Haigh.
(#1448)
* Fixed negative depth values in ampliconstats output.
(#1400)
* samtools addreplacerg now allows for updating (replacing) an existing
`@RG` line in the output header, if a new `@RG` line is provided in the
command line, via the -r argument. The update still requires the user's
approval, which can be given with the new -w option. Thanks to Chuang Yu.
(#1404)
* Stopped samtools cat from outputting multiple CRAM EOF markers.
(#1422)
* Three new counts have been added to samtools flagstat: primary, mapped
primary and duplicate primary.
(#1431; fixes #1382)
* samtools merge now accepts a `-o FILE` option specifying the output file,
similarly to most other subcommands. The existing way of specifying it
(as the first non-option argument, alongside the input file arguments)
remains supported. Thanks to David McGaughey and John Marshall.
(#1434)
* The way samtools merge checks for existing files has been changed
so that it does not hang when used on a named pipe.
(#1438; fixes #1437)
* Updated documentation on mpileup to highlight the fact that the filtering
options on FLAGs work with ANY rules.
(#1447; fixes #1435)
* samtools can now be configured to use a copy of HTSlib that has been set
up with separate build and source trees. When this is the case, the
`--with-htslib` configure option should be given the location of the HTSlib
build tree. (Note that samtools itself does not yet support out-of-tree
builds). Thanks to John Marshall.
(#1427; companion change to samtools/htslib#1277)
Release 1.12 (17th March 2021)
------------------------------
* The legacy samtools API (libbam.a, bam.h, sam.h, etc) has not been actively
maintained since 2015. It is deprecated and will be removed entirely in a
future SAMtools release. We recommend coding against the HTSlib API directly.
* I/O errors and record parsing errors during the reading of SAM/BAM/CRAM
files are now always detected. Thanks to John Marshall (#1379; fixed #101)
* New make targets have been added: check-all, test-all, distclean-all,
mostlyclean-all, testclean-all, which allow SAMtools installations to
call corresponding Makefile targets from embedded HTSlib installations.
* samtools --version now displays a summary of the compilation details and
available features, including flags, used libraries and enabled plugins
from HTSlib. As an alias, `samtools version` can also be used. (#1371)
* samtools stats now displays the number of supplementary reads in the
SN section. Also, supplementary reads are no longer considered when
splitting read pairs by orientation (inward, outward, other). (#1363)
* samtools stats now counts only the filtered alignments that overlap
target regions, if any are specified. (#1363)
* samtools view now accepts option -N, which takes a file containing
read names of interest. This allows the output of only the reads with
names contained in the given file. Thanks to Daniel Cameron. (#1324)
* samtools view -d option now works without a tag associated value, which
allows it to output all the reads with the given tag. (#1339; fixed #1317)
* samtools view -d and -D options now accept integer and single character
values associated with tags, not just strings. Thanks to `@dariome` and
Keiran Raine for the suggestions. (#1357, #1392)
* samtools view now works with the filtering expressions introduced by HTSlib.
The filtering expression is passed to the program using the specific option
-e or the global long option --input-fmt-option. E.g.
`samtools view -e 'qname =~ "#49$" && mrefid != refid && refid != -1 &&
mrefid != -1' align.bam` looks for records with query-name ending in `#49`
that have their mate aligned in a different chromosome. More details can be
found in the FILTER EXPRESSIONS section of the main man page. (#1346)
* samtools markdup now benefits from an increase in performance in the
situation when a single read has tens or hundreds of thousands of duplicates.
Thanks to `@denriquez` for reporting the issue. (#1345; fixed #1325)
* The documentation for samtools ampliconstats has been added to the samtools
man page. (#1351)
* A new FASTA/FASTQ sanitizer script (`fasta-sanitize.pl`) was added, which
corrects the invalid characters in the reference names. (#1314) Thanks to
John Marshall for the installation fix. (#1353)
* The CI scripts have been updated to recurse the HTSlib submodules when
cloning HTSlib, to accommodate for the CRAM codecs, which now reside in the
htscodecs submodule. (#1359)
* The CI integrations now include Cirrus-CI rather than Travis. (#1335; #1365)
* Updated the Windows image used by Appveyor to 'Visual Studio 2019'.
(#1333; fixed #1332)
* Fixed a bug in samtools cat, which prevented the command from running in
multi-threaded mode. Thanks to Alex Leonard for reporting the issue.
(#1337; fixed #1336)
* A couple of invalid CIGAR strings have been corrected in the test data.
(#1343)
* The documentation for `samtools depth -s` has been improved.
Thanks to `@wulj2`. (#1355)
* Fixed a `samtools merge` segmentation fault when it failed to merge
header `@PG` records. Thanks to John Marshall. (#1394; reported by
Kemin Zhou in #1393)
* Ampliconclip and ampliconstats now guard against the BED file
containing more than one reference (chromosome) and fail when
found. Adding proper support for multiple references will appear
later. (#1398)
Release 1.11 (22nd September 2020)
----------------------------------
* New samtools ampliconclip sub-command for removing primers from
amplicon-based sequencing experiments, including the current
COVID-19 projects. The primers are listed in a BED file and can be
either soft-clipped or hard-clipped. (#1219)
* New samtools ampliconstats sub-command to produce a textual summary
of primer and amplicon usage, in a similar style to "samtools
stats". The misc/plot-ampliconstats script can generate PNG
images based on this text report. (#1227)
* Samtools fixmate, addreplacerg, markdup, ampliconclip and sort now
accept a -u option to enable uncompressed output, which is useful when
sending data over a pipe to another process. Other subcommands which
already support this option for the same purpose are calmd, collate,
merge, view and depad. (#1265)
* samtools stats has a new GCT section, where it reports ACGT content
percentages, similar to GCC but taking into account the read orientation.
(#1274)
* Samtools split now supports splitting by tag content with the -d
option (#1211)
* samtools merge now accepts a BED file as a command line argument (-L) and
does the merging only with reads overlapping the specified regions (#1156)
* Samtools sort now has a minhash collation (-M) to group unmapped
reads with similar sequence together. This can sometimes
significantly reduce the file size. (#1093)
* Samtools bedcov now has -g and -G options to filter-in and
filter-out based on the FLAG field. Also the new -d option adds
an extra column per file counting the number of bases with a depth
greater than or equal to a given threshold. (#1214)
* Fixed samtools bedcov -j option (discard deletions and ref-skips)
with multiple input files (#1212)
* samtools bedcov will now accept BED files with columns separated by
spaces as well as tabs (#1246; #1188 reported by Mary Carmack)
* samtools depth can now include deletions (D) when computing the base
coverage depth, if the user adds the -J option to the command
line (#1163).
* samtools depth will count only the bases of one read, for the overlapping
section of a read pair, if the -s option is used in the command line
(#1241, thanks to Teng Li).
* samtools depth will now write zeros for the entire reference length,
when "samtools depth -aa" is run on a file with no alignments. (#1252;
#1249 reported by Paul Donovan)
* Stopped depth from closing stdout, which triggered test fails
in pysam (#1208, thanks to John Marshall).
* samtools view now accepts remote URIs for FASTA and FAI files.
Furthermore, the reference and index file can be provided in a single
argument, such as
samtools view -T ftp://x.com/ref.fa##idx##ftp://y.com/index.fa.fai a.cram
(#1176; samtools/htslib#933 reported by @uitde007)
* samtools faidx gets new options --fai-idx and --gzi-idx to allow
specification of the locations of the .fai and (if needed) .gzi index
files. (#1283)
* The samtools fasta/fastq '-T' option can now add SAM array (type 'B') tags
to the output header lines. (#1301)
* samtools mpileup can now display MAPQ either as ASCII characters (with
-s/--output-MQ; column now restored to its documented order as in 1.9 and
previous versions) or comma-separated numbers (with --output-extra MAPQ;
in SAM column order alongside other selected --output-extra columns).
When both -s/--output-MQ and -O/--output-BP are used, samtools 1.10 printed
the extra columns in the opposite order. This changes the format produced
by 1.10's --output-extra MAPQ. (#1281, thanks to John Marshall; reported
by Christoffer Flensburg)
* samtools tview now accepts a -w option to set the output width in
text mode (-d T). (#1280)
* The dict command can now add AN tags containing alternative names with
"chr" prefixes added to or removed from each sequence name as appropriate
and listing both "M" and "MT" alternatives for mitochondria. (#1164,
thanks to John Marshall)
* The samtools import command, labelled as obsolete in May 2009 and
removed from all help and documentation later that year, has
finally been removed. Use samtools view instead. (#1185)
* Replaced the remaining usage of the Samtools 0.1 legacy API with
htslib calls. (#1187, thanks to John Marshall)
* Documentation / help improvements (#1154; #1168; #1191; #1199;
#1204; #1313):
- Fixed a few man-page table layout issues
- Added <file>##idx##<index> filename documentation
- Fixed usage statement for samtools addreplacerg
- Miscellaneous spelling and grammar fixes
- Note fixmate/markdup name collated rather than name sorted input
- Note that fastq and fasta inputs should also be name collated
- Reshuffled order of main man-page and added -@ to more sub-pages
- The misc/seq_cache_populate.pl script now gives REF_CACHE guidance
* Additional documentation improvements, thanks to John Marshall (#1181;
#1224; #1248; #1262; #1300)
- Emphasise that samtools index requires a position-sorted file
- Document 2^29 chromosome length limit in BAI indexes
- Numerous typing, spelling and formatting fixes
* Improved the message printed when samtools view fails to read its
input (#1296)
* Added build support for the OpenIndiana OS (#1165, thanks to John Marshall)
* Fixed failing tests on OpenBSD (#1151, thanks to John Marshall)
* The samtools sort tests now use less memory so the test suite works better
on small virtual machines. (#1159)
* Improved markdup's calculation of insert sizes (#1161)
Also improved tests (#1150) and made it run faster when not checking
for optical duplicates or adding 'do' tags (#1308)
* Fixed samtools coverage minor inconsistency vs idxstats (#1205; #1203
reported by @calliza)
* Fixed samtools coverage quality thresholding options which were the
wrong way round compared to mpileup (-q is the mapping quality threshold
and -Q is base quality). (#1279; #1278 reported by @kaspernie)
* Fixed bug where `samtools fastq -i` would add two copies of the barcode
in the fastq header if both reads in a pair had a "BC:Z" tag (#1309;
#1307 reported by @mattsoup)
* Samtools calmd no longer errors with a SEQ of "*" (#1230; #1229
reported by Bob Harris)
* Samtools tview now honours $COLUMNS, fixing some CI tests (#1171; #1162
reported by @cljacobs)
* Fixed a samtools depad overflow condition (#1200)
* Improved curses detection in configure script (#1170, #577, #940)
* Fixed samtools stats integer overflows and added support for long
references (#1174; #1173)
* Fixed a 1-byte undersized memory allocation in samtools merge. (#1302)
Release 1.10 (6th December 2019)
--------------------------------
Changes affecting the whole of samtools, or multiple sub-commands:
* Samtools now uses the new HTSlib header API. As this adds more checks
for invalid headers, it is possible that some illegal files will now
be rejected when they would have been allowed by earlier versions. (#998)
Examples of problems that will now be rejected include @SQ lines with
no SN: tag, and @RG or @PG lines with no ID: tag.
* samtools sub-commands will now add '@PG' header lines to output sam/bam/cram
files. To disable this, use the '--no-PG' option. (#1087; #1097)
* samtools now supports alignment records with reference positions greater
than 2 gigabases. This allows samtools to process alignments for
species which have large chromosomes, like axolotl and lungfish. Note that
due to file format limitations, data with large reference positions
must use the SAM format. (#1107; #1117)
* Improved the efficiency of reading and writing SAM format data by 2 fold
(single thread). This is further improved by the ability to use multiple
threads, as previously done with BAM and CRAM.
* samtools can now write BGZF-compressed SAM format. To enable this,
either save files with a '.sam.gz' suffix, or use '--output-fmt sam.gz'.
* samtools can now index BGZF-compressed SAM files.
* The region parsing code has been improved to handle colons in reference
names. Strings can be disambiguated by the use of braces, so for
example when reference sequences called "chr1" and "chr1:100-200"
are both present, the regions "{chr1}:100-200" and "{chr1:100-200}"
unambiguously indicate which reference is being used. (#864)
* samtools flags, flagstats, idxstats and stats now have aliases
flag, flagstat, idxstat and stat. (#934)
* A new global '--write-index' option has been added. This allows output
sam.gz/bam/cram files to be indexed while they are being written out.
This should work with addreplacerg, depad, markdup, merge, sort, split,
and view. (#1062)
* A global '--verbosity' option has been added to enable/disable
debugging output. (#1124, thanks to John Marshall)
* It is now possible to have data and index files stored in different
locations. There are two ways to tell samtools where to find the
index:
1. Samtools bedcov, depth, merge, mpileup, stats, tview, and view
accept a new option (-X). When this is used, each input sam/bam/cram
listed on the command line should have a corresponding index file.
Note that all the data files should be listed first, followed by all
the index files. (#978, thanks to Mingfei Shao)
2. A delimiter '##idx##' can be appended to the data file name followed
by the index file name. This can be used both for input files and
outputs when indexing on-the-fly.
* HTSlib (and therefore SAMtools) now uses version 4 signatures by default
for its s3:// plug-in. It can also write to S3 buckets, as long as
version 4 signatures are in use. See HTSlib's NEWS file and
htslib-s3-plugin manual page for more information.
* HTSlib (and therefore SAMtools) no longer considers a zero-length
file to be a valid SAM file. This has been changed so that pipelines such
as `somecmd | samtools ...` with `somecmd` aborting before outputting
anything will now propagate the error to the second command.
* The samtools manual page has been split up into one for each
sub-command. The main samtools.1 manual page now lists the sub-commands
and describes the common global options. (#894)
* The meaning of decode_md, store_md and store_nm in the fmt-option section
of the samtools.1 man page has been clarified. (#898, thanks to Evan Benn)
* Fixed numerous memory leaks. (#892)
* Fixed incorrect macro definition on Windows. (#950)
* bedcov, phase, misc/ace2sam and misc/wgsim now check for failure to open
files. (#1013, thanks to Julie Blommaert and John Marshall)
Changes affecting specific sub-commands:
* A new "coverage" sub-command has been added. This prints a tabular format
of the average coverage and percent coverage for each reference sequence,
as well as number of aligned reads, average mapping quality and base
quality. It can also (with the '-m' option) plot a histogram of
coverage across the genome. (#992, thanks to Florian Breitwieser)
* samtools calmd:
- Reference bases in MD: tags are now converted to upper case. (#981, #988)
* samtools depth:
- Add new options to write a header to the output (-H) and to direct
the output to a file (-o). (#937, thanks to Pierre Lindenbaum)
- New options '-g' and '-G' can be used to filter reads. (#953)
- Fix memory leak when failing to set CRAM options. (#985, thanks
to Florian Breitwieser)
- Fix bug when using region filters where the '-a' option did not
work for regions with no coverage. (#1113; #1112 reported by
Paweł Sztromwasser)
* samtools fasta and fastq:
- '-1 FILE -2 FILE' with the same filename now works properly. (#1042)
- '-o FILE' is added as a synonym for '-1 FILE -2 FILE'. (#1042)
- The '-F' option now defaults to 0x900 (SECONDARY,SUPPLEMENTARY).
Previously secondary and supplementary records were filtered internally
in a way that could not be turned off. (#1042; #939 reported
by @finswimmer)
- Allow reading from a pipe without an explicit '-' on the command line.
(#1042; #874 reported by John Marshall)
- Turn on multi-threading for bgzf compressed output files. (#908)
- Fixed bug where the samtools fastq -i would output incorrect information
in the Casava tags for dual-index reads. It also now prints the tags
for dual indices in the same way as bcl2fastq, using a '+' sign between
the two parts of the index. (#1059; #1047 reported by Denis Loginov)
* samtools flagstat:
- Samtools flagstat can now optionally write its output in JSON format or
as a tab-separated values file. (#1106, thanks to Vivek Rai).
* samtools markdup:
- It can optionally tag optical duplicates (reads following Illumina
naming conventions only). The is enabled with the '-d' option,
which sets the distance for duplicates to be considered as optical.
(#1091; #1103; #1121; #1128; #1134)
- The report stats (-s) option now outputs counts for optical and
non-primary (supplementary / secondary) duplicates. It also reports
the Picard "estimate library size" statistic. A new '-f' option
can be used to save the statistics in a given file. (#1091)
- The rules for calling duplicates can be changed using the new --mode
option. This mainly changes the position associated with each read in
a pair. '--mode t' (the default) is the existing behaviour where the
position used is that of the outermost template base associated with the
read. Alternatively '--mode s' always uses the first unclipped sequence
base. In practice, this only makes a difference for read pairs where the
two reads are aligned in the same direction. (#1091)
- A new '-c' option can be used to clear any existing duplicate tags.
(#1091)
- A new '--include-fails' option makes markdup include QC-failed reads.
(#1091)
- Fixed buffer overflow in temporary file writer when writing a mixture
of long and short alignment records. (#911; #909)
* samtools mpileup:
- mpileup can now process alignments including CIGAR P (pad) operators
correctly. They will now also produce the correct output for alignments
where insertions are immediately followed by deletions, or deletions by
insertions. Note that due to limitations in HTSlib, they are still
unable to output sequences that have been inserted before the first
aligned base of a read. (#847; #842 reported by Tiffany Delhomme.
See also htslib issue #59 and pull request #699).
- In samtools mpileup, a deletion or pad on the reverse strand is now
marked with a different character ('#') than the one used on a forward
strand ('*'), if the '--reverse-del' option is used. (#1070)
- New option '--output-extra' can be used to add columns for user
selected alignment fields or aux tags. (#1073)
- Fixed double-counting of overlapping bases in alignment records with
deletions or reference skips longer than twice the insert size.
(#989; #987 reported by @dariomel)
- Improved manual page with documentation about what each output column
means. (#1055, thanks to John Marshall)
* samtools quickcheck:
- Add unmapped (-u) option, which disables the check for @SQ lines in
the header. (#920, thanks to Shane McCarthy)
* samtools reheader:
- A new option '-c' allows the input header to be passed to a given
command. Samtools then takes the output of this command and uses it
as the replacement header. (#1007)
- Make it clear in help message that reheader --in-place only works on
CRAM files. (#921, thanks to Julian Gehring)
- Refuse to in-place reheader BAM files, instead of unexpectedly writing
a BAM file to stdout. (#935)
* samtools split:
- In samtools split, the '-u' option no longer accepts an extra file name
from which a replacement header was read. The two file names were
separated using a colon, which caused problems on Windows and prevented
the use of URLs. A new '-h' option has been added to allow the replacement
header file to be specified in its own option. (#961)
- Fixed bug where samtools split would crash if it read a SAM header that
contained an @RG line with no ID tag. (#954, reported by @blue-bird1)
* samtools stats:
- stats will now compute base compositions for BC, CR, OX and RX tags,
and quality histograms for QT, CY, BZ and QX tags. (#904)
- New stats FTC and LTC showing total number of nucleotides for first and
last fragments. (#946)
- The rules for classifying reads as "first" or "last" fragment have been
tightened up. (#949)
- Fixed bug where stats could over-estimate coverage when using the
target-regions option or when a region was specified on the command-line.
(#1027; #1025, reported by Miguel Machado; #1029, reported by Jody Phelan).
- Fixed error in stats GCD percentile depth calculation when the depth to be
reported fell between two bins. It would report the depth entirely from
the lower bin instead of taking a weighted average of the two. (#1048)
- Better catching and reporting of out of memory conditions. (#984;
#982, reported by Jukka Matilainen)
- Improved manual page. (#927)
* samtools tview:
- tview can now display alignments including CIGAR P operators, D followed
by I and I followed by D correctly. See mpileup above for more
information. (#847; #734, reported by Ryan Lorig-Roach)
- The "go to position" text entry box has been made wider. (#968, thanks
to John Marshall)
- Fixed samtools tview -s option which was not filtering reads correctly.
It now only shows reads from the requested sample or read group. (#1089)
* samtools view:
- New options '-d' and '-D' to only output alignments which have a tag
with a given type and value. (#1001, thanks to Gert Hulselmans)
* misc/plot-bamstats script:
- Fixed merge (-m) option. (#923, #924 both thanks to Marcus D Sherman)
- Made the quality heatmap work with gnuplot version 5.2.7 and later.
(#1068; #1065 reported by Martin Mokrejš)
- Fixed --do-ref-stats bug where fasta header lines would be counted
as part of the sequence when the --targets option was used. (#1120,
thanks to Neil Goodgame)
* Removed the misc/varfilter.py Python script, as it takes consensus-pileup
as input, which was removed from samtools in release 0.1.17 in 2011. (#1125)
Release 1.9 (18th July 2018)
----------------------------
* Samtools mpileup VCF and BCF output is now deprecated. It is still
functional, but will warn. Please use bcftools mpileup instead. (#884)
* Samtools mpileup now handles the '-d' max_depth option differently. There
is no longer an enforced minimum, and '-d 0' is interpreted as limitless
(no maximum - warning this may be slow). The default per-file depth is
now 8000, which matches the value mpileup used to use when processing
a single sample. To get the previous default behaviour use the higher
of 8000 divided by the number of samples across all input files, or 250.
(#859)
* Samtools stats new features:
- The '--remove-overlaps' option discounts overlapping portions of
templates when computing coverage and mapped base counting. (#855)
- When a target file is in use, the number of bases inside the
target is printed and the percentage of target bases with coverage
above a given threshold specified by the '--cov-threshold' option. (#855)
- Split base composition and length statistics by first and last reads.
(#814, #816)
* Samtools faidx new features:
- Now takes long options. (#509, thanks to Pierre Lindenbaum)
- Now warns about zero-length and truncated sequences due to the
requested range being beyond the end of the sequence. (#834)
- Gets a new option (--continue) that allows it to carry on
when a requested sequence was not in the index. (#834)
- It is now possible to supply the list of regions to output in a text
file using the new '--region-file' option. (#840)
- New '-i' option to make faidx return the reverse complement of
the regions requested. (#878)
- faidx now works on FASTQ (returning FASTA) and added a new
fqidx command to index and return FASTQ. (#852)
* Samtools collate now has a fast option '-f' that only operates on
primary pairs, dropping secondary and supplementary. It tries to write
pairs to the final output file as soon as both reads have been found. (#818)
* Samtools bedcov gets a new '-j' option to make it ignore deletions (D) and
reference skips (N) when computing coverage. (#843)
* Small speed up to samtools coordinate sort, by converting it to use
radix sort. (#835, thanks to Zhuravleva Aleksandra)
* Samtools idxstats now works on SAM and CRAM files, however this
isn't fast due to some information lacking from indices. (#832)
* Compression levels may now be specified with the level=N
output-fmt-option. E.g. with -O bam,level=3.
* Various documentation improvements.
* Bug-fixes:
- Improved error reporting in several places. (#827, #834, #877, cd7197)
- Various test improvements.
- Fixed failures in the multi-region iterator (view -M) when regions
provided via BED files include overlaps (#819, reported by Dave Larson).
- Samtools stats now counts '=' and 'X' CIGAR operators when
counting mapped bases. (#855)
- Samtools stats has fixes for insert size filtering (-m, -i). (#845; #697
reported by Soumitra Pal)
- Samtools stats -F now longer negates an earlier -d option. (#830)
- Fix samtools stats crash when using a target region. (#875, reported by
John Marshall)
- Samtools sort now keeps to a single thread when the -@ option is absent.
Previously it would spawn a writer thread, which could cause the CPU
usage to go slightly over 100%. (#833, reported by Matthias Bernt)
- Fixed samtools phase '-A' option which was incorrectly defined to take
a parameter. (#850; #846 reported by Dianne Velasco)
- Fixed compilation problems when using C_INCLUDE_PATH. (#870; #817 reported
by Robert Boissy)
- Fixed --version when built from a Git repository. (#844, thanks to
John Marshall)
- Use noenhanced mode for title in plot-bamstats. Prevents unwanted
interpretation of characters like underscore in gnuplot version 5. (#829,
thanks to M. Zapukhlyak)
- blast2sam.pl now reports perfect match hits (no indels or mismatches).
(#873, thanks to Nils Homer)
- Fixed bug in fasta and fastq subcommands where stdout would not be flushed
correctly if the -0 option was used.
- Fixed invalid memory access in mpileup and depth on alignment records
where the sequence is absent.
Release 1.8 (3rd April 2018)
----------------------------
* samtools calmd now has a quiet mode. This can be enabled by passing `-Q` to
calmd. (Thanks to Colin Davenport)
* In samtools depth `-d 0` will effectively remove the depth limit. (#764)
* Improvements made to samtools collate's interface and documentation. It is
now possible to specify an output file name using `-o`, instead of deriving
it from the prefix used for temporary files. The prefix itself is now
optional if `-o` or `-O` (to stdout) is used. (#780)
* Bug-fixes:
- Make samtools addreplacerg choose output format by file extension. (#767;
reported by Argy Megalios)
- Merge tests now work on ungzipped data, allowing tests to be run against
different deflate libraries.
- samtools markdup error messages about missing tags have been updated with
the suggestion that samtools fixmate is run beforehand. (#765; reported by
Yudong Cai)
- Enables the `--reference` option for samtools fastq. Now works like other
programs when a reference sequence is needed for CRAM files. (#791,
reported by Milana Kaljevic)
Release 1.7 (26th January 2018)
--------------------
* HTSlib, and so samtools, now support BAMs which include CIGARs with more
than 65535 operations as per HTS-Specs 18th November (dab57f4 and 2f915a8).
* samtools quickcheck will now write a warning to stderr if it finds
any problems. These messages can be suppressed with a new `-q` option.
* samtools markdup can now mark supplementary alignments of reads where
the primary alignment is found to be a duplicate. Supplementary marking
can be turned on by passing the `-S` option to markdup. When this
option is enabled, all the alignment data will be written to a temporary
file so that supplementary alignments that occur before a duplicated
primary can be correctly marked in the final output. The location
of this temporary file can be influenced using the new `-T` option.
* samtools view now supports HTSlib's new multi-region iterator.
This can be enabled by passing the `-M` option to view. When using
this option:
- The BED filter (`-L` option) will use the index to skip through the file
- Reads from overlapping regions will only be output once
* samtools bedcov will now ignore BED comment and header lines (#571; thanks
to Daniel Baker).
* samtools collate now updates the @HD SO: and GO: tags, and sort will
remove a GO: tag if present. (#757; reported by Imran Haque).
* Bug-fixes:
- maq2sam now checks for input files that end early. (#751; patch supplied
by Alexandre Rebert of the Mayhem team, via Andreas Tille from Debian.)
- Fixed incorrect check when looking up header tags that could lead
to a crash in samtools stats. (#208; thanks to Dave Larson.)
- Fixed bug in samtools fastq `-O` option where it would fail if
the OQ tag in the input file had an unexpected type. (#758;
reported by Taejeong Bae)
- The MD5 calculations in samtools dict and md5fa did not handle
non-alphabetic characters in the same way as the CRAM MD5 function.
They have now been updated to match. (#704; reported by Chris Norman).
- Fix possible infinite loop in samtools targetcut.
- Building bam_tview_curses should no longer fail if a curses header file
cannot be found.
Release 1.6 (28th September 2017)
--------------------
* Added new markdup sub-command and '-m' option for fixmate. Used together,
they allow duplicates to be marked and optionally removed. This
fixes a number of problems with the old 'rmdup' sub-command, for
example samtools issue #497. 'rmdup' is kept for backwards compatibility
but 'markdup' should be used in preference.
* Sort is now much better at keeping within the requested memory limit. It
should also be slightly faster and need fewer temporary files when the file
to be sorted does not fit in memory. (#593; thanks to Nathan Weeks.)
* Sort no longer rewrites the header when merging from files. It can also
now merge from memory, so fewer temporary files need to be written and
it is better at sorting in parallel when everything fits in memory.
* Both sort and merge now resolve ties when merging based on the position
in the input file(s). This makes them fully stable for all ordering
options. (Previously position sort was stable, but name and by tag
sorts were not).
* New --output-qname option for mpileup.
* Support for building on Windows using msys2/mingw64 or cygwin has
been improved.
Release 1.5 [Solstice Release] (21st June 2017)
--------------------
* Samtools fastq now has a -i option to create a fastq file from an index
tag, and a -T option (similar to -t) to add user specified aux tags to
the fastq header line.
* Samtools fastq can now create compressed fastq files, by giving the
output filenames an extension of .gq, .bgz, or .bgzf
* Samtools sort has a -t TAG option, that allows records to be sorted by
the value of the specified aux tag, then by position or name. Merge
gets a similar option, allowing files sorted this way to be merged.
(#675; thanks to Patrick Marks of 10xgenomics).
Release 1.4.1 (8th May 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Added options to fastq to create fastq files from BC (or other)
tags.
* Samtools view has gained a -G <flags> option to exclude on all bits
set. For example to discard reads where neither end has been
mapped use "-G 12".
* Samtools cat has a -b <fofn> option to ease concatenation of many
files.
* Added misc/samtools_tab_completion for bash auto-completion of
samtools sub-commands. (#560)
* Samtools tview now has J and K keys for verticale movement by 20
lines. (#257)
* Various compilation / portability improvements.
* Fixed issue with more than 65536 CIGAR operations and SAM/CRAM files.
(#667)
Release 1.4 (13 March 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Noteworthy changes in samtools:
* Fixed Issue #345 - out-by-one error in insert-size in samtools stats
* bam_split now add a @PG header to the bam file
* Added mate cigar tag support to fixmate
* Multi-threading is now supported for decoding BAM and CRAM (as well
as the previously supported encoding). Most commands that read BAM
or CRAM have gained an -@ or --threads arguments, providing a
significant speed bonus. For commands that both read and write
files the threads are shared between decoding and encoding tasks.
* Added -a option to samtools mpileup to show all locations, including
sites with zero depth; repeating the option as -aa or -a -a additionally
shows reference sequences without any reads mapped to them (#496).
* The mpileup text output no longer contains empty columns at zero coverage
positions. Previously it would output "...0\t\t..." in some circumstances
(zero coverage due to being below a minimum base quality); this has been
fixed to output as "...0\t*\t*..." with placeholder '*' characters as in
other zero coverage circumstances (see PR #537).
* To stop it from creating too many temporary files, samtools sort
will now not run unless its per-thread memory limit (-m) is set to
at least 1 megabyte (#547).
* The misc/plot-bamstats script now has a -l / --log-y option to change
various graphs to display their Y axis log-scaled. Currently this
affects the Insert Size graph (PR #589; thanks to Anton Kratz).
* Fixmate will now also add and update MC (mate CIGAR) tags.
Beta Release 1.3.1 (22 April 2016)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Noteworthy changes in samtools:
* The sort command creates any needed temporary files alongside the final
output file (similarly to the pre-1.3 behaviour), and now aborts when
it detects a collision with another sort invocation's temporary files.
When the -T PREFIX option specified is a directory (or when sorting to
standard output), a random component is now added to temporary filenames
to try to avoid collisions (#432, #523, #529, #535, PR #530).
* All samtools commands now check for I/O errors more carefully, especially
when writing output files (#111, #253, #470, PR #467).
* Build fixes for 32-bit systems; be sure to run configure on such systems
to enable large file support and access to 2GiB+ files.
* The fasta/fastq/bam2fq command no longer ignores reads when the -s option
is used (#532).
* The fastq -O option no longer crashes on reads that do not have an OQ tag
field (#517).
* The merge and sort commands now handle (unusual) BAM files that have no
textual @SQ headers (#548, #550).
* Sorting files containing @CO headers no longer duplicates the comment
headers, which previously happened on large sorts for which temporary
files were needed (#563).
* The rmdup and view -l commands no longer crash on @RG headers that do not
have a LB field (#538).
* Fixed miscellaneous issues #128, #130, #131, #489, and #514.
Beta Release 1.3 (15 December 2015)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Noteworthy changes in samtools:
* The obsolete "samtools sort in.bam out.prefix" usage has been removed.
If you are still using -f, -o, or out.prefix, convert to use -T PREFIX
and/or -o FILE instead. (#295, #349, #356, #418, PR #441; see also
discussions in #171, #213.)
* When writing CRAM output, samtools now defaults to writing CRAM v3.0
rather than v2.1.
* The "bamshuf" command has been renamed to "collate" (hence the term
bamshuf no longer appears in the documentation, though it still works
on the command line for compatibility with existing scripts).
* The mpileup command now outputs the unseen allele in VCF/BCF as <*>
rather than X or <X> as previously, and now has AD, ADF, ADR, INFO/AD,
INFO/ADF, INFO/ADR --output-tags annotations that largely supersede
the existing DV, DP4, DPR annotations.
* The mpileup command now applies BAQ calculations at all base positions,
regardless of which -l or -r options are used (previously with -l it was
not applied to the first few tens of bases of each chromosome, leading
to different mpileup results with -l vs. -r; #79, #125, #286, #407).
* Samtools now has a configure script which checks your build environment
and facilitates choosing which HTSlib to build against. See INSTALL
for details.
* Samtools's Makefile now fully supports the standard convention of
allowing CC/CPPFLAGS/CFLAGS/LDFLAGS/LIBS to be overridden as needed.
Previously it listened to $(LDLIBS) instead; if you were overriding
that, you should now override LIBS rather than LDLIBS.
* A new addreplacerg command that adds or alters @RG headers and RG:Z
record tags has been added.
* The rmdup command no longer immediately aborts (previously it always
aborted with "bam_get_library() not yet implemented"), but remains
not recommended for most use (#159, #252, #291, #393).
* Merging files with millions of headers now completes in a reasonable
amount of time (#337, #373, #419, #453; thanks to Nathan Weeks,
Chris Smowton, Martin Pollard, Rob Davies).
* Samtools index's optional index output path argument works again (#199).
* Fixed calmd, targetcut, and potential mpileup segfaults when given broken
alignments with POS far beyond the end of their reference sequences.
* If you have source code using bam_md.c's bam_fillmd1_core(), bam_cap_mapQ(),
or bam_prob_realn_core() functions, note that these now take an additional
ref_len parameter. (The versions named without "_core" are unchanged.)