-
Notifications
You must be signed in to change notification settings - Fork 33
/
NEWS
2148 lines (1499 loc) · 83.1 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
XZ Utils Release Notes
======================
5.6.0 (2024-02-24)
This bumps the minor version of liblzma because new features were
added. The API and ABI are still backward compatible with liblzma
5.4.x and 5.2.x and 5.0.x.
NOTE: As described in the NEWS for 5.5.2beta, the core components
are now under the BSD Zero Clause License (0BSD).
Since 5.5.2beta:
* liblzma:
- Disabled the branchless C variant in the LZMA decoder based
on the benchmark results from the community.
- Disabled x86-64 inline assembly on x32 to fix the build.
* Sandboxing support in xz:
- Landlock is now used even when xz needs to create files.
In this case the sandbox is has to be more permissive than
when no files need to be created. A similar thing was
already in use with pledge(2) since 5.3.4alpha.
- Landlock and pledge(2) are now stricter when reading from
more than one input file and only writing to standard output.
- Added support for Landlock ABI version 4.
* CMake:
- Default to -O2 instead of -O3 with CMAKE_BUILD_TYPE=Release.
-O3 is not useful for speed and makes the code larger.
- Now builds lzmainfo and lzmadec.
- xzdiff, xzgrep, xzless, xzmore, and their symlinks are now
installed. The scripts are also tested during "make test".
- Added translation support for xz, lzmainfo, and the
man pages.
- Applied the symbol versioning workaround for MicroBlaze that
is used in the Autotools build.
- The general XZ Utils and liblzma API documentation is now
installed.
- The CMake component names were changed a little and several
were added. liblzma_Runtime and liblzma_Development are
unchanged.
- Minimum required CMake version is now 3.14. However,
translation support is disabled with CMake versions
older than 3.20.
- The CMake-based build is now close to feature parity with the
Autotools-based build. Most importantly a few tests aren't
run yet. Testing the CMake-based build on different operating
systems would be welcome now. See the comment at the top of
CMakeLists.txt.
* Fixed a bug in the Autotools feature test for ARM64 CRC32
instruction support for old versions of Clang. This did not
affect the CMake build.
* Windows:
- The build instructions in INSTALL and windows/INSTALL*.txt
were revised completely.
- windows/build-with-cmake.bat along with the instructions
in windows/INSTALL-MinGW-w64_with_CMake.txt should make
it very easy to build liblzma.dll and xz.exe on Windows
using CMake and MinGW-w64 with either GCC or Clang/LLVM.
- windows/build.bash was updated. It now works on MSYS2 and
on GNU/Linux (cross-compiling) to create a .zip and .7z
package for 32-bit and 64-bit x86 using GCC + MinGW-w64.
* The TODO file is no longer installed as part of the
documentation. The file is out of date and does not reflect
the actual tasks that will be completed in the future.
* Translations:
- Translated lzmainfo man pages are now installed. These
had been forgotten in earlier versions.
- Updated Croatian, Esperanto, German, Hungarian, Korean,
Polish, Romanian, Spanish, Swedish, Vietnamese, and Ukrainian
translations.
- Updated German, Korean, Romanian, and Ukrainian man page
translations.
* Added a few tests.
Summary of new features added in the 5.5.x development releases:
* liblzma:
- LZMA decoder: Speed optimizations to the C code and
added GCC & Clang compatible inline assembly for x86-64.
- Added lzma_mt_block_size() to recommend a Block size for
multithreaded encoding.
- Added CLMUL-based CRC32 on x86-64 and E2K with runtime
processor detection. Similar to CRC64, on 32-bit x86 it
isn't available unless --disable-assembler is used.
- Optimized the CRC32 calculation on ARM64 platforms using the
CRC32 instructions. Runtime detection for the instruction is
used on GNU/Linux, FreeBSD, Windows, and macOS. If the
compiler flags indicate unconditional CRC32 instruction
support (+crc) then the generic version is not built.
- Added definitions of mask values like
LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>.
* xz:
- Multithreaded mode is now the default. This improves
compression speed and creates .xz files that can be
decompressed in multithreaded mode. The downsides are
increased memory usage and slightly worse compression ratio.
- Added a new command line option --filters to set the filter
chain using the liblzma filter string syntax.
- Added new command line options --filters1 ... --filters9 to
set additional filter chains using the liblzma filter string
syntax. The --block-list option now allows specifying filter
chains that were set using these new options.
- Ported the command line tools to Windows MSVC.
Visual Studio 2015 or later is required.
* Added lz4 support to xzdiff/xzcmp and xzgrep.
5.5.2beta (2024-02-14)
* Licensing change: The core components are now under the
BSD Zero Clause License (0BSD). In XZ Utils 5.4.6 and older
and 5.5.1alpha these components are in the public domain and
obviously remain so; the change affects the new releases only.
0BSD is an extremely permissive license which doesn't require
retaining or reproducing copyright or license notices when
distributing the code, thus in practice there is extremely
little difference to public domain.
* liblzma
- Significant speed optimizations to the LZMA decoder were
made. There are now three variants that can be chosen at
build time:
* Basic C version: This is a few percent faster than
5.4.x due to some new optimizations.
* Branchless C: This is currently the default on platforms
for which there is no assembly code. This should be a few
percent faster than the basic C version.
* x86-64 inline assembly. This works with GCC and Clang.
The default choice can currently be overridden by setting
LZMA_RANGE_DECODER_CONFIG in CPPFLAGS: 0 means the basic
version and 3 means that branchless C version.
- Optimized the CRC32 calculation on ARM64 platforms using the
CRC32 instructions. The instructions are optional in ARMv8.0
and are required in ARMv8.1 and later. Runtime detection for
the instruction is used on GNU/Linux, FreeBSD, Windows, and
macOS. If the compiler flags indicate unconditional CRC32
instruction support (+crc) then the generic version is not
built.
* Added lz4 support to xzdiff/xzcmp and xzgrep.
* Man pages of xzdiff/xzcmp, xzgrep, and xzmore were rewritten
to simplify licensing of the man page translations.
* Translations:
- Updated Chinese (simplified), German, Korean, Polish,
Romanian, Spanish, Swedish, and Ukrainian translations.
- Updated German, Korean, Romanian, and Ukrainian man page
translations.
* Small improvements to the tests.
* Added doc/examples/11_file_info.c. It was added to the Git
repository in 2017 but forgotten to be added into distribution
tarballs.
* Removed doc/examples_old. These were from 2012.
* Removed the macos/build.sh script. It had not been updated
since 2013.
5.5.1alpha (2024-01-26)
* Added a new filter for RISC-V binaries. The filter can be used
for 32-bit and 64-bit binaries with either little or big
endianness. In liblzma, the Filter ID is LZMA_FILTER_RISCV (0x0B)
and the xz option is --riscv. liblzma filter string syntax
recognizes this filter as "riscv".
* liblzma:
- Added lzma_mt_block_size() to recommend a Block size for
multithreaded encoding
- Added CLMUL-based CRC32 on x86-64 and E2K with runtime
processor detection. Similar to CRC64, on 32-bit x86 it
isn't available unless --disable-assembler is used.
- Implemented GNU indirect function (IFUNC) as a runtime
function dispatching method for CRC32 and CRC64 fast
implementations on x86. Only GNU/Linux (glibc) and FreeBSD
builds will use IFUNC, unless --enable-ifunc is specified to
configure.
- Added definitions of mask values like
LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>.
- The XZ logo is now included in the Doxygen generated
documentation. It is licensed under Creative Commons
Attribution-ShareAlike 4.0.
* xz:
- Multithreaded mode is now the default. This improves
compression speed and creates .xz files that can be
decompressed multithreaded at the cost of increased memory
usage and slightly worse compression ratio.
- Added new command line option --filters to set the filter
chain using liblzma filter string syntax.
- Added new command line options --filters1 ... --filters9 to
set additional filter chains using liblzma filter string
syntax. The --block-list option now allows specifying filter
chains that were set using these new options.
- Added support for Linux Landlock as a sandboxing method.
- xzdec now supports pledge(2), Capsicum, and Linux Landlock as
sandboxing methods.
- Progress indicator time stats remain accurate after pausing
xz with SIGTSTP.
- Ported xz and xzdec to Windows MSVC. Visual Studio 2015 or
later is required.
* CMake Build:
- Supports pledge(2), Capsicum, and Linux Landlock sandboxing
methods.
- Replacement functions for getopt_long() are used on platforms
that do not have it.
* Enabled unaligned access by default on PowerPC64LE and on RISC-V
targets that define __riscv_misaligned_fast.
* Tests:
- Added two new fuzz targets to OSS-Fuzz.
- Implemented Continuous Integration (CI) testing using
GitHub Actions.
* Changed quoting style from `...' to '...' in all messages,
scripts, and documentation.
* Added basic Codespell support to help catch typo errors.
5.4.6 (2024-01-26)
* Fixed a bug involving internal function pointers in liblzma not
being initialized to NULL. The bug can only be triggered if
lzma_filters_update() is called on a LZMA1 encoder, so it does
not affect xz or any application known to us that uses liblzma.
* xz:
- Fixed a regression introduced in 5.4.2 that caused encoding
in the raw format to unnecessarily fail if --suffix was not
used. For instance, the following command no longer reports
that --suffix must be used:
echo foo | xz --format=raw --lzma2 | wc -c
- Fixed an issue on MinGW-w64 builds that prevented reading
from or writing to non-terminal character devices like NUL.
* Added a new test.
5.4.5 (2023-11-31)
* liblzma:
- Use __attribute__((__no_sanitize_address__)) to avoid address
sanitization with CRC64 CLMUL. It uses 16-byte-aligned reads
which can extend past the bounds of the input buffer and
inherently trigger address sanitization errors. This isn't
a bug.
- Fixed an assertion failure that could be triggered by a large
unpadded_size argument. It was verified that there was no
other bug than the assertion failure.
- Fixed a bug that prevented building with Windows Vista
threading when __attribute__((__constructor__)) is not
supported.
* xz now properly handles special files such as "con" or "nul" on
Windows. Before this fix, the following wrote "foo" to the
console and deleted the input file "con_xz":
echo foo | xz > con_xz
xz --suffix=_xz --decompress con_xz
* Build systems:
- Allow builds with Windows win95 threading and small mode when
__attribute__((__constructor__)) is supported.
- Added a new line to liblzma.pc for MSYS2 (Windows):
Cflags.private: -DLZMA_API_STATIC
When compiling code that will link against static liblzma,
the LZMA_API_STATIC macro needs to be defined on Windows.
- CMake specific changes:
* Fixed a bug that allowed CLOCK_MONOTONIC to be used even
if the check for it failed.
* Fixed a bug where configuring CMake multiple times
resulted in HAVE_CLOCK_GETTIME and HAVE_CLOCK_MONOTONIC
not being set.
* Fixed the build with MinGW-w64-based Clang/LLVM 17.
llvm-windres now has more accurate GNU windres emulation
so the GNU windres workaround from 5.4.1 is needed with
llvm-windres version 17 too.
* The import library on Windows is now properly named
"liblzma.dll.a" instead of "libliblzma.dll.a"
* Fixed a bug causing the Ninja Generator to fail on
UNIX-like systems. This bug was introduced in 5.4.0.
* Added a new option to disable CLMUL CRC64.
* A module-definition (.def) file is now created when
building liblzma.dll with MinGW-w64.
* The pkg-config liblzma.pc file is now installed on all
builds except when using MSVC on Windows.
* Added large file support by default for platforms that
need it to handle files larger than 2 GiB. This includes
MinGW-w64, even 64-bit builds.
* Small fixes and improvements to the tests.
* Updated translations: Chinese (simplified) and Esperanto.
5.4.4 (2023-08-02)
* liblzma and xzdec can now build against WASI SDK when threading
support is disabled. xz and tests don't build yet.
* CMake:
- Fixed a bug preventing other projects from including liblzma
multiple times using find_package().
- Don't create broken symlinks in Cygwin and MSYS2 unless
supported by the environment. This prevented building for the
default MSYS2 environment. The problem was introduced in
xz 5.4.0.
* Documentation:
- Small improvements to man pages.
- Small improvements and typo fixes for liblzma API
documentation.
* Tests:
- Added a new section to INSTALL to describe basic test usage
and address recent questions about building the tests when
cross compiling.
- Small fixes and improvements to the tests.
* Translations:
- Fixed a mistake that caused one of the error messages to not
be translated. This only affected versions 5.4.2 and 5.4.3.
- Updated the Chinese (simplified), Croatian, Esperanto, German,
Korean, Polish, Romanian, Spanish, Swedish, Ukrainian, and
Vietnamese translations.
- Updated the German, Korean, Romanian, and Ukrainian man page
translations.
5.4.3 (2023-05-04)
* All fixes from 5.2.12
* Features in the CMake build can now be disabled as CMake cache
variables, similar to the Autotools build.
* Minor update to the Croatian translation.
5.4.2 (2023-03-18)
* All fixes from 5.2.11 that were not included in 5.4.1.
* If xz is built with support for the Capsicum sandbox but running
in an environment that doesn't support Capsicum, xz now runs
normally without sandboxing instead of exiting with an error.
* liblzma:
- Documentation was updated to improve the style, consistency,
and completeness of the liblzma API headers.
- The Doxygen-generated HTML documentation for the liblzma API
header files is now included in the source release and is
installed as part of "make install". All JavaScript is
removed to simplify license compliance and to reduce the
install size.
- Fixed a minor bug in lzma_str_from_filters() that produced
too many filters in the output string instead of reporting
an error if the input array had more than four filters. This
bug did not affect xz.
* Build systems:
- autogen.sh now invokes the doxygen tool via the new wrapper
script doxygen/update-doxygen, unless the command line option
--no-doxygen is used.
- Added microlzma_encoder.c and microlzma_decoder.c to the
VS project files for Windows and to the CMake build. These
should have been included in 5.3.2alpha.
* Tests:
- Added a test to the CMake build that was forgotten in the
previous release.
- Added and refactored a few tests.
* Translations:
- Updated the Brazilian Portuguese translation.
- Added Brazilian Portuguese man page translation.
5.4.1 (2023-01-11)
* liblzma:
- Fixed the return value of lzma_microlzma_encoder() if the
LZMA options lc/lp/pb are invalid. Invalid lc/lp/pb options
made the function return LZMA_STREAM_END without encoding
anything instead of returning LZMA_OPTIONS_ERROR.
- Windows / Visual Studio: Workaround a possible compiler bug
when targeting 32-bit x86 and compiling the CLMUL version of
the CRC64 code. The CLMUL code isn't enabled by the Windows
project files but it is in the CMake-based builds.
* Build systems:
- Windows-specific CMake changes:
* Don't try to enable CLMUL CRC64 code if _mm_set_epi64x()
isn't available. This fixes CMake-based build with Visual
Studio 2013.
* Created a workaround for a build failure with windres
from GNU binutils. It is used only when the C compiler
is GCC (not Clang). The workaround is incompatible
with llvm-windres, resulting in "XZx20Utils" instead
of "XZ Utils" in the resource file, but without the
workaround llvm-windres works correctly. See the
comment in CMakeLists.txt for details.
* Included the resource files in the xz and xzdec build
rules. Building the command line tools is still
experimental but possible with MinGW-w64.
- Visual Studio: Added stream_decoder_mt.c to the project
files. Now the threaded decompressor lzma_stream_decoder_mt()
gets built. CMake-based build wasn't affected.
- Updated windows/INSTALL-MSVC.txt to mention that CMake-based
build is now the preferred method with Visual Studio. The
project files will probably be removed after 5.4.x releases.
- Changes to #defines in config.h:
* HAVE_DECL_CLOCK_MONOTONIC was replaced by
HAVE_CLOCK_MONOTONIC. The old macro was always defined
in configure-generated config.h to either 0 or 1. The
new macro is defined (to 1) only if the declaration of
CLOCK_MONOTONIC is available. This matches the way most
other config.h macros work and makes things simpler with
other build systems.
* HAVE_DECL_PROGRAM_INVOCATION_NAME was replaced by
HAVE_PROGRAM_INVOCATION_NAME for the same reason.
* Tests:
- Fixed test script compatibility with ancient /bin/sh
versions. Now the five test_compress_* tests should
no longer fail on Solaris 10.
- Added and refactored a few tests.
* Translations:
- Updated the Catalan and Esperanto translations.
- Added Korean and Ukrainian man page translations.
5.4.0 (2022-12-13)
This bumps the minor version of liblzma because new features were
added. The API and ABI are still backward compatible with liblzma
5.2.x and 5.0.x.
Since 5.3.5beta:
* All fixes from 5.2.10.
* The ARM64 filter is now stable. The xz option is now --arm64.
Decompression requires XZ Utils 5.4.0. In the future the ARM64
filter will be supported by XZ for Java, XZ Embedded (including
the version in Linux), LZMA SDK, and 7-Zip.
* Translations:
- Updated Catalan, Croatian, German, Romanian, and Turkish
translations.
- Updated German man page translations.
- Added Romanian man page translations.
Summary of new features added in the 5.3.x development releases:
* liblzma:
- Added threaded .xz decompressor lzma_stream_decoder_mt().
It can use multiple threads with .xz files that have multiple
Blocks with size information in Block Headers. The threaded
encoder in xz has always created such files.
Single-threaded encoder cannot store the size information in
Block Headers even if one used LZMA_FULL_FLUSH to create
multiple Blocks, so this threaded decoder cannot use multiple
threads with such files.
If there are multiple Streams (concatenated .xz files), one
Stream will be decompressed completely before starting the
next Stream.
- A new decoder flag LZMA_FAIL_FAST was added. It makes the
threaded decompressor report errors soon instead of first
flushing all pending data before the error location.
- New Filter IDs:
* LZMA_FILTER_ARM64 is for ARM64 binaries.
* LZMA_FILTER_LZMA1EXT is for raw LZMA1 streams that don't
necessarily use the end marker.
- Added lzma_str_to_filters(), lzma_str_from_filters(), and
lzma_str_list_filters() to convert a preset or a filter chain
string to a lzma_filter[] and vice versa. These should make
it easier to write applications that allow users to specify
custom compression options.
- Added lzma_filters_free() which can be convenient for freeing
the filter options in a filter chain (an array of lzma_filter
structures).
- lzma_file_info_decoder() to makes it a little easier to get
the Index field from .xz files. This helps in getting the
uncompressed file size but an easy-to-use random access
API is still missing which has existed in XZ for Java for
a long time.
- Added lzma_microlzma_encoder() and lzma_microlzma_decoder().
It is used by erofs-utils and may be used by others too.
The MicroLZMA format is a raw LZMA stream (without end marker)
whose first byte (always 0x00) has been replaced with
bitwise-negation of the LZMA properties (lc/lp/pb). It was
created for use in EROFS but may be used in other contexts
as well where it is important to avoid wasting bytes for
stream headers or footers. The format is also supported by
XZ Embedded (the XZ Embedded version in Linux got MicroLZMA
support in Linux 5.16).
The MicroLZMA encoder API in liblzma can compress into a
fixed-sized output buffer so that as much data is compressed
as can be fit into the buffer while still creating a valid
MicroLZMA stream. This is needed for EROFS.
- Added lzma_lzip_decoder() to decompress the .lz (lzip) file
format version 0 and the original unextended version 1 files.
Also lzma_auto_decoder() supports .lz files.
- lzma_filters_update() can now be used with the multi-threaded
encoder (lzma_stream_encoder_mt()) to change the filter chain
after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH.
- In lzma_options_lzma, allow nice_len = 2 and 3 with the match
finders that require at least 3 or 4. Now it is internally
rounded up if needed.
- CLMUL-based CRC64 on x86-64 and E2K with runtime processor
detection. On 32-bit x86 it currently isn't available unless
--disable-assembler is used which can make the non-CLMUL
CRC64 slower; this might be fixed in the future.
- Building with --disable-threads --enable-small
is now thread-safe if the compiler supports
__attribute__((__constructor__)).
* xz:
- Using -T0 (--threads=0) will now use multi-threaded encoder
even on a single-core system. This is to ensure that output
from the same xz binary is identical on both single-core and
multi-core systems.
- --threads=+1 or -T+1 is now a way to put xz into
multi-threaded mode while using only one worker thread.
The + is ignored if the number is not 1.
- A default soft memory usage limit is now used for compression
when -T0 is used and no explicit limit has been specified.
This soft limit is used to restrict the number of threads
but if the limit is exceeded with even one thread then xz
will continue with one thread using the multi-threaded
encoder and this limit is ignored. If the number of threads
is specified manually then no default limit will be used;
this affects only -T0.
This change helps on systems that have very many cores and
using all of them for xz makes no sense. Previously xz -T0
could run out of memory on such systems because it attempted
to reserve memory for too many threads.
This also helps with 32-bit builds which don't have a large
amount of address space that would be required for many
threads. The default soft limit for -T0 is at most 1400 MiB
on all 32-bit platforms.
- Previously a low value in --memlimit-compress wouldn't cause
xz to switch from multi-threaded mode to single-threaded mode
if the limit cannot otherwise be met; xz failed instead. Now
xz can switch to single-threaded mode and then, if needed,
scale down the LZMA2 dictionary size too just like it already
did when it was started in single-threaded mode.
- The option --no-adjust no longer prevents xz from scaling down
the number of threads as that doesn't affect the compressed
output (only performance). Now --no-adjust only prevents
adjustments that affect compressed output, that is, with
--no-adjust xz won't switch from multi-threaded mode to
single-threaded mode and won't scale down the LZMA2
dictionary size.
- Added a new option --memlimit-mt-decompress=LIMIT. This is
used to limit the number of decompressor threads (possibly
falling back to single-threaded mode) but it will never make
xz refuse to decompress a file. This has a system-specific
default value because without any limit xz could end up
allocating memory for the whole compressed input file, the
whole uncompressed output file, multiple thread-specific
decompressor instances and so on. Basically xz could
attempt to use an insane amount of memory even with fairly
common files. The system-specific default value is currently
the same as the one used for compression with -T0.
The new option works together with the existing option
--memlimit-decompress=LIMIT. The old option sets a hard limit
that must not be exceeded (xz will refuse to decompress)
while the new option only restricts the number of threads.
If the limit set with --memlimit-mt-decompress is greater
than the limit set with --memlimit-compress, then the latter
value is used also for --memlimit-mt-decompress.
- Added new information to the output of xz --info-memory and
new fields to the output of xz --robot --info-memory.
- In --lzma2=nice=NUMBER allow 2 and 3 with all match finders
now that liblzma handles it.
- Don't mention endianness for ARM and ARM-Thumb filters in
--long-help. The filters only work for little endian
instruction encoding but modern ARM processors using
big endian data access still use little endian
instruction encoding. So the help text was misleading.
In contrast, the PowerPC filter is only for big endian
32/64-bit PowerPC code. Little endian PowerPC would need
a separate filter.
- Added decompression support for the .lz (lzip) file format
version 0 and the original unextended version 1. It is
autodetected by default. See also the option --format on
the xz man page.
- Sandboxing enabled by default:
* Capsicum (FreeBSD)
* pledge(2) (OpenBSD)
* Scripts now support the .lz format using xz.
* A few new tests were added.
* The liblzma-specific tests are now supported in CMake-based
builds too ("make test").
5.3.5beta (2022-12-01)
* All fixes from 5.2.9.
* liblzma:
- Added new LZMA_FILTER_LZMA1EXT for raw encoder and decoder to
handle raw LZMA1 streams that don't have end of payload marker
(EOPM) alias end of stream (EOS) marker. It can be used in
filter chains, for example, with the x86 BCJ filter.
- Added lzma_str_to_filters(), lzma_str_from_filters(), and
lzma_str_list_filters() to make it easier for applications
to get custom compression options from a user and convert
it to an array of lzma_filter structures.
- Added lzma_filters_free().
- lzma_filters_update() can now be used with the multi-threaded
encoder (lzma_stream_encoder_mt()) to change the filter chain
after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH.
- In lzma_options_lzma, allow nice_len = 2 and 3 with the match
finders that require at least 3 or 4. Now it is internally
rounded up if needed.
- ARM64 filter was modified. It is still experimental.
- Fixed LTO build with Clang if -fgnuc-version=10 or similar
was used to make Clang look like GCC >= 10. Now it uses
__has_attribute(__symver__) which should be reliable.
* xz:
- --threads=+1 or -T+1 is now a way to put xz into multi-threaded
mode while using only one worker thread.
- In --lzma2=nice=NUMBER allow 2 and 3 with all match finders
now that liblzma handles it.
* Updated translations: Chinese (simplified), Korean, and Turkish.
5.3.4alpha (2022-11-15)
* All fixes from 5.2.7 and 5.2.8.
* liblzma:
- Minor improvements to the threaded decoder.
- Added CRC64 implementation that uses SSSE3, SSE4.1, and CLMUL
instructions on 32/64-bit x86 and E2K. On 32-bit x86 it's
not enabled unless --disable-assembler is used but then
the non-CLMUL code might be slower. Processor support is
detected at runtime so this is built by default on x86-64
and E2K. On these platforms, if compiler flags indicate
unconditional CLMUL support (-msse4.1 -mpclmul) then the
generic version is not built, making liblzma 8-9 KiB smaller
compared to having both versions included.
With extremely compressible files this can make decompression
up to twice as fast but with typical files 5 % improvement
is a more realistic expectation.
The CLMUL version is slower than the generic version with
tiny inputs (especially at 1-8 bytes per call, but up to
16 bytes). In normal use in xz this doesn't matter at all.
- Added an experimental ARM64 filter. This is *not* the final
version! Files created with this experimental version won't
be supported in the future versions! The filter design is
a compromise where improving one use case makes some other
cases worse.
- Added decompression support for the .lz (lzip) file format
version 0 and the original unextended version 1. See the
API docs of lzma_lzip_decoder() for details. Also
lzma_auto_decoder() supports .lz files.
- Building with --disable-threads --enable-small
is now thread-safe if the compiler supports
__attribute__((__constructor__))
* xz:
- Added support for OpenBSD's pledge(2) as a sandboxing method.
- Don't mention endianness for ARM and ARM-Thumb filters in
--long-help. The filters only work for little endian
instruction encoding but modern ARM processors using
big endian data access still use little endian
instruction encoding. So the help text was misleading.
In contrast, the PowerPC filter is only for big endian
32/64-bit PowerPC code. Little endian PowerPC would need
a separate filter.
- Added --experimental-arm64. This will be renamed once the
filter is finished. Files created with this experimental
filter will not be supported in the future!
- Added new fields to the output of xz --robot --info-memory.
- Added decompression support for the .lz (lzip) file format
version 0 and the original unextended version 1. It is
autodetected by default. See also the option --format on
the xz man page.
* Scripts now support the .lz format using xz.
* Build systems:
- New #defines in config.h: HAVE_ENCODER_ARM64,
HAVE_DECODER_ARM64, HAVE_LZIP_DECODER, HAVE_CPUID_H,
HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR, HAVE_USABLE_CLMUL
- New configure options: --disable-clmul-crc,
--disable-microlzma, --disable-lzip-decoder, and
'pledge' is now an option in --enable-sandbox (but
it's autodetected by default anyway).
- INSTALL was updated to document the new configure options.
- PACKAGERS now lists also --disable-microlzma and
--disable-lzip-decoder as configure options that must
not be used in builds for non-embedded use.
* Tests:
- Fix some of the tests so that they skip instead of fail if
certain features have been disabled with configure options.
It's still not perfect.
- Other improvements to tests.
* Updated translations: Croatian, Finnish, Hungarian, Polish,
Romanian, Spanish, Swedish, and Ukrainian.
5.3.3alpha (2022-08-22)
* All fixes from 5.2.6.
* liblzma:
- Fixed 32-bit build.
- Added threaded .xz decompressor lzma_stream_decoder_mt().
It can use multiple threads with .xz files that have multiple
Blocks with size information in Block Headers. The threaded
encoder in xz has always created such files.
Single-threaded encoder cannot store the size information in
Block Headers even if one used LZMA_FULL_FLUSH to create
multiple Blocks, so this threaded decoder cannot use multiple
threads with such files.
If there are multiple Streams (concatenated .xz files), one
Stream will be decompressed completely before starting the
next Stream.
- A new decoder flag LZMA_FAIL_FAST was added. It makes the
threaded decompressor report errors soon instead of first
flushing all pending data before the error location.
* xz:
- Using -T0 (--threads=0) will now use multi-threaded encoder
even on a single-core system. This is to ensure that output
from the same xz binary is identical on both single-core and
multi-core systems.
- A default soft memory usage limit is now used for compression
when -T0 is used and no explicit limit has been specified.
This soft limit is used to restrict the number of threads
but if the limit is exceeded with even one thread then xz
will continue with one thread using the multi-threaded
encoder and this limit is ignored. If the number of threads
is specified manually then no default limit will be used;
this affects only -T0.
This change helps on systems that have very many cores and
using all of them for xz makes no sense. Previously xz -T0
could run out of memory on such systems because it attempted
to reserve memory for too many threads.
This also helps with 32-bit builds which don't have a large
amount of address space that would be required for many
threads. The default limit is 1400 MiB on all 32-bit
platforms with -T0.
Now xz -T0 should just work. It might use too few threads
in some cases but at least it shouldn't easily run out of
memory. It's possible that this will be tweaked before 5.4.0.
- Changes to --memlimit-compress and --no-adjust:
In single-threaded mode, --memlimit-compress can make xz
scale down the LZMA2 dictionary size to meet the memory usage
limit. This obviously affects the compressed output. However,
if xz was in threaded mode, --memlimit-compress could make xz
reduce the number of threads but it wouldn't make xz switch
from multi-threaded mode to single-threaded mode or scale
down the LZMA2 dictionary size. This seemed illogical.
Now --memlimit-compress can make xz switch to single-threaded
mode if one thread in multi-threaded mode uses too much
memory. If memory usage is still too high, then the LZMA2
dictionary size can be scaled down too.
The option --no-adjust was also changed so that it no longer
prevents xz from scaling down the number of threads as that
doesn't affect compressed output (only performance). After
this commit --no-adjust only prevents adjustments that affect
compressed output, that is, with --no-adjust xz won't switch
from multithreaded mode to single-threaded mode and won't
scale down the LZMA2 dictionary size.
- Added a new option --memlimit-mt-decompress=LIMIT. This is
used to limit the number of decompressor threads (possibly
falling back to single-threaded mode) but it will never make
xz refuse to decompress a file. This has a system-specific
default value because without any limit xz could end up
allocating memory for the whole compressed input file, the
whole uncompressed output file, multiple thread-specific
decompressor instances and so on. Basically xz could
attempt to use an insane amount of memory even with fairly
common files.
The new option works together with the existing option
--memlimit-decompress=LIMIT. The old option sets a hard limit
that must not be exceeded (xz will refuse to decompress)
while the new option only restricts the number of threads.
If the limit set with --memlimit-mt-decompress is greater
than the limit set with --memlimit-compress, then the latter
value is used also for --memlimit-mt-decompress.
* Tests:
- Added a few more tests.