Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8293499: Provide jmod --compress option #10213

Closed

Conversation

shipilev
Copy link
Member

@shipilev shipilev commented Sep 8, 2022

I have been looking into make clean-images images performance, and realized jmod keeps compressing files with default compression level. Tuning that toward lighter compression levels improves build performance considerably, without a heavy loss in *.jmod sizes.

This PR allows JMOD to select the compression level. Follow-ups would use this in the build system, see #10214.

The interesting asymmetry against jlink is: jlink provides --compress option that only takes 2 for "ZIP compression". (Separately, we could argue if it would be beneficial to extend --compress to jlink as well, so to select the compression level there too.)


Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change requires a CSR request to be approved

Issues

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/10213/head:pull/10213
$ git checkout pull/10213

Update a local copy of the PR:
$ git checkout pull/10213
$ git pull https://git.openjdk.org/jdk pull/10213/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 10213

View PR using the GUI difftool:
$ git pr show -t 10213

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/10213.diff

@bridgekeeper
Copy link

bridgekeeper bot commented Sep 8, 2022

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr Pull request is ready for review label Sep 8, 2022
@openjdk
Copy link

openjdk bot commented Sep 8, 2022

@shipilev The following label will be automatically applied to this pull request:

  • core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the core-libs core-libs-dev@openjdk.org label Sep 8, 2022
@mlbridge
Copy link

mlbridge bot commented Sep 8, 2022

@AlanBateman
Copy link
Contributor

AlanBateman commented Sep 8, 2022

I don't have time to look at this closely today but I think an option to configure the compression level wouldn't look out of place on the jmod tool.

Doing the equivalent with jlink would require more thought to see if it makes sense or is really needed. The jimage format is for execution, not a packaging format like jmod that is primarily consumed at link-time. I agree the jlink --compress option is a bit overloaded where --compress=0 and --compress=2 are more like jar --no-compress or its default to deflate. The jar tool doesn't have an option to set the compression level.

@mlchung
Copy link
Member

mlchung commented Sep 9, 2022

It's reasonable for jmod tool to support --compress-level option to specify the compression level. The JMOD file format is not specified but the implementation uses ZIP. Compression level from 0-9 makes sense for ZIP but may not be applicable if the JMOD file format is changed. But it may be okay to worry about that when the file format is changed.

For jlink, when resources are compressed in writing to jimage, there would be performance overhead in reading and decompressing the resources at runtime. I'd be interested in knowing the tradeoff between the smaller jimage vs the runtime overhead in decompressing the resources. I'd advice to separate the jmod compression level change from jlink compression level discussion.

@AlanBateman
Copy link
Contributor

I went through the changes and I don't see anything obviously wrong. The command line option doesn't look out of place although as Mandy points us, the format is not documented so there is flexibility to change it in the future. If some future format doesn't support something like compression level then that option could just emit a warning and do nothing.

@AlanBateman
Copy link
Contributor

/csr

@openjdk openjdk bot added the csr Pull request needs approved CSR before integration label Sep 9, 2022
@openjdk
Copy link

openjdk bot commented Sep 9, 2022

@AlanBateman has indicated that a compatibility and specification (CSR) request is needed for this pull request.

@shipilev please create a CSR request for issue JDK-8293499 with the correct fix version. This pull request cannot be integrated until the CSR request is approved.

@shipilev
Copy link
Member Author

I submitted the CSR request for this change: https://bugs.openjdk.org/browse/JDK-8293629

@cl4es
Copy link
Member

cl4es commented Sep 12, 2022

Could we word this so that the accepted values is compression-dependent? The value appears to be ignored for --compress=0|1 (should it throw if specified or not specified to, say, 0?). We might also add other compressions in the future which allows a wider or narrower range of compression levels (zstd uses 0 - 22).

@jaikiran
Copy link
Member

I gave this a try locally. It's my understanding that this new option is only relevant when creating the jmod archive. However, right now, in its current form, I can pass this option to commands like jmod extract and jmod list without those commands complaining about that option. For example, this following command doesn't complain about the --compression-level being passed to it:

jmod extract --compression-level=9 jdk/jmods/java.base.jmod

Should we instead complain/fail when this option is used for anything other than create?

@shipilev
Copy link
Member Author

Could we word this so that the accepted values is compression-dependent? The value appears to be ignored for --compress=0|1 (should it throw if specified or not specified to, say, 0?). We might also add other compressions in the future which allows a wider or narrower range of compression levels (zstd uses 0 - 22).

Um, this is jmod, it does not have --compress=... option? jlink does. So, I don't quite understand this comment :)

@shipilev
Copy link
Member Author

Should we instead complain/fail when this option is used for anything other than create?

Good idea, see new commit.

@cl4es
Copy link
Member

cl4es commented Sep 12, 2022

Um, this is jmod, it does not have --compress=... option? jlink does. So, I don't quite understand this comment :)

Doh! Yes, I mixed up the two and assumed you'd be considering this for jlink as well. The part about being open-ended about alternate compression algorithms still applies, but since the compression for jmod is currently always zip we can defer such wording for if/when that day comes.

@jaikiran
Copy link
Member

Thank you for the changes Aleksey. They look fine to me (except for that one minor error message related issue which I added a comment for). I haven't given the latest changes a try, but the new test you added to verify that the command fails in other modes when --compression-level will cover it.

@shipilev
Copy link
Member Author

The part about being open-ended about alternate compression algorithms still applies, but since the compression for jmod is currently always zip we can defer such wording for if/when that day comes.

Well, we can future-proof it by doing what e.g. ZFS does:

$ zfs set compression
...
	compression     YES      YES   on | off | lzjb | gzip | gzip-[1-9] | zle | lz4 | zstd | zstd-[1-19] | zstd-fast-[1-10,20,30,40,50,60,70,80,90,100,500,1000]
...

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

@cl4es
Copy link
Member

cl4es commented Sep 12, 2022

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

I like this scheme. It's a good thing there's prior art, too.

@mbreinhold
Copy link
Member

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

I like this scheme. It's a good thing there's prior art, too.

Agreed. With this approach we can also revise jlink --compress to take the same arguments, keeping [012] for compatibility. We can also name this option --compress for jmod, rather than --compression-level, for consistency.

@shipilev shipilev changed the title 8293499: Provide jmod compression level option 8293499: Provide jmod --compress option Sep 12, 2022
@shipilev
Copy link
Member Author

Agreed. With this approach we can also revise jlink --compress to take the same arguments, keeping [012] for compatibility. We can also name this option --compress for jmod, rather than --compression-level, for consistency.

I updated this PR and related CSR with this new option format.

Copy link
Member

@mlchung mlchung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also like the new --compress option taking zip-0, zip-1,... zip-9 values.

Copy link
Member

@jaikiran jaikiran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest state of this PR looks fine to me.

Copy link
Member

@mlchung mlchung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The jmod usage should specify the default.

@jaikiran
Copy link
Member

Hello Aleksey, could you also please create a release note for this? You may already know how to do that, if not, the steps are noted here https://openjdk.org/guide/#release-notes

@shipilev
Copy link
Member Author

Hello Aleksey, could you also please create a release note for this? You may already know how to do that, if not, the steps are noted here https://openjdk.org/guide/#release-notes

Yes, but later, once CSR and this PR integrates in its final form.

@shipilev
Copy link
Member Author

(posting a comment in the hopes that bots would check the CSR state)

@openjdk openjdk bot removed the csr Pull request needs approved CSR before integration label Sep 16, 2022
@openjdk
Copy link

openjdk bot commented Sep 16, 2022

@shipilev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8293499: Provide jmod --compress option

Reviewed-by: alanb, mchung, jpai, redestad

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 66 new commits pushed to the master branch:

  • 01e7b88: 8290917: x86: Memory-operand arithmetic instructions have too low costs
  • 4b8399b: 8293251: Use stringStream::base() instead of as_string() when applicable
  • a8f0f57: 8278165: Clarify that ZipInputStream does not access the CEN fields for a ZipEntry
  • 746f5f5: 8293816: CI: ciBytecodeStream::get_klass() is not consistent
  • 4b297c1: 8293892: Add links to JVMS 19 and 20 from ClassFileFormatVersion enum constants
  • dfb9c06: 8293535: jdk/javadoc/doclet/testJavaFX/TestJavaFxMode.java fail with jfx
  • f42caef: 8293550: Optionally add get-task-allow entitlement to macos binaries
  • 5feca68: 8293840: RISC-V: Remove cbuf parameter from far_call/far_jump/trampoline_call
  • 39cd163: 8293578: Duplicate ldc generated by javac
  • 7765942: 8290367: Update default value and extend the scope of com.sun.jndi.ldap.object.trustSerialData system property
  • ... and 56 more: https://git.openjdk.org/jdk/compare/37df5f56259429482cfdbe38e8b6256f1efaf9e8...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Sep 16, 2022
@shipilev
Copy link
Member Author

I did a minor cleanup, unquoted: "zip-6" -> zip-6 to be consistent in help message. Otherwise, this is ready to go and I will integrate it soon. Thanks!

@shipilev
Copy link
Member Author

Thank you!

/integrate

@openjdk
Copy link

openjdk bot commented Sep 19, 2022

Going to push as commit 43f7f47.
Since your change was applied there have been 71 commits pushed to the master branch:

  • 26e08cf: 8293844: C2: Verify Location::{oop,normal} types in PhaseOutput::FillLocArray
  • 357a2cc: 8293937: x86: Drop LP64 conditions from clearly x86_32 code
  • b1ed40a: 8293466: libjsig should ignore non-modifying sigaction calls
  • b6ff8fa: 8292073: NMT: remove unused constructor parameter from MallocHeader
  • cfd44bb: 8293218: serviceability/tmtools/jstat/GcNewTest.java fails with "Error in the percent calculation"
  • 01e7b88: 8290917: x86: Memory-operand arithmetic instructions have too low costs
  • 4b8399b: 8293251: Use stringStream::base() instead of as_string() when applicable
  • a8f0f57: 8278165: Clarify that ZipInputStream does not access the CEN fields for a ZipEntry
  • 746f5f5: 8293816: CI: ciBytecodeStream::get_klass() is not consistent
  • 4b297c1: 8293892: Add links to JVMS 19 and 20 from ClassFileFormatVersion enum constants
  • ... and 61 more: https://git.openjdk.org/jdk/compare/37df5f56259429482cfdbe38e8b6256f1efaf9e8...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Sep 19, 2022
@openjdk openjdk bot closed this Sep 19, 2022
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 19, 2022
@openjdk
Copy link

openjdk bot commented Sep 19, 2022

@shipilev Pushed as commit 43f7f47.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@shipilev shipilev deleted the JDK-8293499-jmod-compression-level branch October 21, 2022 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-libs core-libs-dev@openjdk.org integrated Pull request has been integrated
6 participants