Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter to string command line interface #12

Merged
merged 23 commits into from
Jul 17, 2023
Merged

Filter to string command line interface #12

merged 23 commits into from
Jul 17, 2023

Commits on Jul 17, 2023

  1. xz: Add --filters option to CLI. 

    The --filters option uses the new lzma_str_to_filters() function
to convert a string into a full filter chain. Using this option
will reset all previous filters set by --preset, --[filter], or
--filters.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    a3bcd59 View commit details
    Browse the repository at this point in the history

  2. xz: Update --long-help and man page for new --filters option.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    a88630d View commit details
    Browse the repository at this point in the history

  3. Tests: Use new --filters option in test_compress.sh

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    a38dc05 View commit details
    Browse the repository at this point in the history

  4. xz: Separate string to filter conversion into a helper function. 

    Converting from string to filter will also need to be done for block
specific filter chains.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    177dc1e View commit details
    Browse the repository at this point in the history

  5. xz: Use lzma_filters_free() in forget_filter_chain(). 

    This is a little cleaner than the previous implementation of
forget_filter_chain(). It is also more consistent since
lzma_str_to_filters() will always terminate the filter chain so there
is no need to terminate it later in coder_set_compression_settings().
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    16f7c3c View commit details
    Browse the repository at this point in the history

  6. xz: Create command line options for filters[1-9]. 

    The new command line options are meant to be combined with --block-list.
They work as an optional extension to --block-list to specify a custom
filter chain for each block listed. The new options allow the creation
of up to 9 reusable filter chains. For instance:

xz --block-list=1:10MiB,3:5MiB,,2:5MiB,1:0 --filters1=delta--lzma2 \
--filters2=x86--lzma2 --filters3=arm64--lzma2

Will create the following blocks:
1. A block of size 10 MiB with filter chain delta, lzma2.
2. A block of size 5 MiB with filter chain arm64, lzma2.
3. A block of size 5 MiB with filter chain arm64, lzma2.
4. A block of size 5 MiB with filter chain x86, lzma2.
5. A block containing the rest of the file contents with filter chain
   delta, lzma2.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    c56b0b3 View commit details
    Browse the repository at this point in the history

  7. xz: Add a message if --block-list is used outside of xz compresssion. 

    --block-list is only supported with compression in xz format. This avoids
silently ignoring when --block-list is unused.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7cc8ac8 View commit details
    Browse the repository at this point in the history

  8. xz: Free filters[] in debug mode. 

    This will only free filter chains created with --filters1-9 since the
default filter chain may be set from a static function variable. The
complexity to free the default filter chain is not worth the burden on
code maintenance.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    1758b5c View commit details
    Browse the repository at this point in the history

  9. xz: Do not include block splitting if encoders are disabled. 

    The block splitting logic and split_block() function are not needed if
encoders are disabled. This will help slightly reduce the binary size
when built without encoders and allow split_block() to use functions
that require encoders being enabled.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    6dc935d View commit details
    Browse the repository at this point in the history

  10. xz: Allows --block-list filters to scale down memory usage. 

    Previously, only the default filter chain could have its memory usage
adjusted. The filter chains specified with --filtersX were not checked
for memory usage. Now, all used filter chains will be adjusted if
necessary.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    2b6b050 View commit details
    Browse the repository at this point in the history

  11. xz: Validate --flush-timeout for all specified filter chains.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    5c2ec96 View commit details
    Browse the repository at this point in the history

  12. xz: Set the Block size for mt encoding correctly. 

    When opt_block_size is not used, the Block size for mt encoder is
derived from the minimum of the largest Block specified by
--block-list and the recommended Block size on all filter chains
calculated by lzma_mt_block_size(). This avoids using unnecessary
memory and ensures that all Blocks are large enough for the most memory
needy filter chain.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    e63a0f7 View commit details
    Browse the repository at this point in the history

  13. xz: Ignore filter chains that are set but never used in --block-list. 

    If a filter chain is set but not used in --block-list, it introduced
unexpected behavior such as requiring an unneeded amount of memory to
compress, reducing the number of threads in multi-threaded encoding, and
printing an incorrect amount of memory needed to decompress.

This also renames filters_init_mask => filters_used_mask. A filter is
assumed to be used if it is specified in --filtersX until
coder_set_compression_settings() determines which filters are referenced
in --block-list.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    6d2922d View commit details
    Browse the repository at this point in the history

  14. xz: Update --long-help for the new --filtersX option.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    794b418 View commit details
    Browse the repository at this point in the history

  15. xz: Update the man page for --block-list and --filtersX 

    The --block-list option description needed updating since the new
--filtersX option changes how it can be used. The new entry for
--filters1=FILTERS ... --filter9=FILTERS was created right after
the --filters option.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    55e066e View commit details
    Browse the repository at this point in the history

  16. xz: Add a new --filters-help option. 

    The --filters-help can be used to help create filter chains with the
--filters and --filtersX options. The message in --long-help is too
short to fully explain the syntax to construct complex filter chains.

In --robot mode, xz will only print the output from liblzma function
lzma_str_list_filters.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    589b2e0 View commit details
    Browse the repository at this point in the history

  17. xz: Update man page for new --filters-help option.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    bd4f72f View commit details
    Browse the repository at this point in the history

  18. xz: Reorder robot mode subsections in the man page. 

    The order is now consistent with the order the command line arguments
are documented earlier in the man page. The new order is:
1. --list
2. --info-memory
3. --version

Instead of the previous order:
1. --version
2. --info-memory
3. --list
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7c2b5f5 View commit details
    Browse the repository at this point in the history

  19. xz: Slight reword in xz man page for consistency. 

    Changed will print => prints in xz --robot --version description to
match --robot --info-memory description.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    20ee158 View commit details
    Browse the repository at this point in the history

  20. xz: Add a section to man page for robot mode --filters-help.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    761451c View commit details
    Browse the repository at this point in the history

  21. xz: Update man page Authors and date.

    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    c5c4576 View commit details
    Browse the repository at this point in the history

  22. xz: Minor clean up for coder.c 

    * Moved max_block_list_size from a global to local variable.
* Reworded error message in validate_block_list_filter().
* Removed helper function filter_chain_error().
* Changed 1 << X to 1U << X in many places
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    1acbef4 View commit details
    Browse the repository at this point in the history

  23. xz: Fix typo in man page. 

    The Memory limit information section described three output
columns when it actually has six. This was reworded to
"multiple" to make it more future proof.
    @JiaT75
    JiaT75 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    2d1178e View commit details
    Browse the repository at this point in the history