New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dist has gotten too large due to new maven features #2487

Closed
headius opened this Issue Jan 20, 2015 · 18 comments

Comments

Projects
None yet
4 participants
@headius
Member

headius commented Jan 20, 2015

The newish jar_dependencies feature in JRuby resulted in us having to ship Maven itself plus its dependencies. That increased the size of our distribution by more than 20MB. Other libraries contribute, but the ruby-maven-libs gem is the biggest issue.

Our users like the jar-dependencies feature, so it seems like we need to ship this stuff...but we need to shrink it down. Suggestions so far:

  • Install them as default gems. This will eliminate the cached gem file from our dist.
  • Use pack200 on the jars in the maven gem. That reduces size of the cached gem file (so moot point if we install as a default gem)
  • Install lazily or request user to install X.
@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jan 23, 2015

Member

did look at the complete distribution and used pack200 to reduce the jars which leads to:

$ ll -h *zip
41M  jruby-dist-9.0.0.0-SNAPSHOT-bin.zip
13M  jruby-dist-9.0.0.0-SNAPSHOT-src.zip
18M  jruby-dist-9.0.0.0-SNAPSHOT-pack200.zip

to use the pack200.zip you need to run a "post_install" script

find -name "*pack200.gz" | sed "s/\(.*\).pack200.gz/\1.pack200.gz \1.jar/" | xargs -L1 unpack200 -r

the reduction in size is impressive and worth for using with rbenv, rvm and alike. maybe even windows installer. the post_installer could be also an executable jar.

Member

mkristian commented Jan 23, 2015

did look at the complete distribution and used pack200 to reduce the jars which leads to:

$ ll -h *zip
41M  jruby-dist-9.0.0.0-SNAPSHOT-bin.zip
13M  jruby-dist-9.0.0.0-SNAPSHOT-src.zip
18M  jruby-dist-9.0.0.0-SNAPSHOT-pack200.zip

to use the pack200.zip you need to run a "post_install" script

find -name "*pack200.gz" | sed "s/\(.*\).pack200.gz/\1.pack200.gz \1.jar/" | xargs -L1 unpack200 -r

the reduction in size is impressive and worth for using with rbenv, rvm and alike. maybe even windows installer. the post_installer could be also an executable jar.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jan 23, 2015

Member

another point are the native libraries

  • both lib/jruby.jar has bundled the jni/* directory inside the jar and the dist archive contains lib/jni/* as well
Member

mkristian commented Jan 23, 2015

another point are the native libraries

  • both lib/jruby.jar has bundled the jni/* directory inside the jar and the dist archive contains lib/jni/* as well
@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Jan 23, 2015

Member

I believe windows installer does use pack200 internally as part of install4j.

Member

enebo commented Jan 23, 2015

I believe windows installer does use pack200 internally as part of install4j.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 3, 2015

Member

did some pack200 stuff on https://github.com/jruby/jruby/tree/pack20

jruby-dist-bin.tar.gz 37MB
jruby-dist-bin.zip 38MB
jruby-dist-bin200.tar.gz 17MB
jruby-dist-bin200.zip 18MB
jruby-dist-src.tar.gz 8MB
jruby-dist-src.zip 13MB

after unzipping jruby-dist-bin200.* archives you need to run bin/unpack200.sh
which takes a couple of seconds. did not yet tried for a windows unpack200.bat - first I need to know whether this is OK to go this route !

I personally would go for the bin200 archives - the reduction is really impressive and makes a difference in installing jruby (for me).

I also started to repack ruby-maven and ruby-maven-libs so they can be installed a default gems and refactored the script for installing default gems from lib/pom.rb into its own ruby file (need to reuse it for maven/jruby-dist) but this is still work in process.

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

Member

mkristian commented Mar 3, 2015

did some pack200 stuff on https://github.com/jruby/jruby/tree/pack20

jruby-dist-bin.tar.gz 37MB
jruby-dist-bin.zip 38MB
jruby-dist-bin200.tar.gz 17MB
jruby-dist-bin200.zip 18MB
jruby-dist-src.tar.gz 8MB
jruby-dist-src.zip 13MB

after unzipping jruby-dist-bin200.* archives you need to run bin/unpack200.sh
which takes a couple of seconds. did not yet tried for a windows unpack200.bat - first I need to know whether this is OK to go this route !

I personally would go for the bin200 archives - the reduction is really impressive and makes a difference in installing jruby (for me).

I also started to repack ruby-maven and ruby-maven-libs so they can be installed a default gems and refactored the script for installing default gems from lib/pom.rb into its own ruby file (need to reuse it for maven/jruby-dist) but this is still work in process.

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

@chrisseaton

This comment has been minimized.

Show comment
Hide comment
@chrisseaton

chrisseaton Mar 3, 2015

Contributor

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

That's a really good idea.

Contributor

chrisseaton commented Mar 3, 2015

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

That's a really good idea.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 3, 2015

Member

Very nice @mkristian, and I agree the size reduction is significant. Only thing that gives me pause is releasing too many distribution archives since I feel like we'd need to still provide the unpacked versions too.

What do you think @enebo?

Member

headius commented Mar 3, 2015

Very nice @mkristian, and I agree the size reduction is significant. Only thing that gives me pause is releasing too many distribution archives since I feel like we'd need to still provide the unpacked versions too.

What do you think @enebo?

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 3, 2015

Member
Member

mkristian commented Mar 3, 2015

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Mar 3, 2015

Member

The original problem was dist was too big and the answer is to continue disting the big file and add more dist files to the release? I don't think this is actually solving a problem. I also don't think most users know what pack200 is and so they won't download it because they won't think they can unpack it. pack200, to me, fits better into installers since it can be hidden from the person installing.

I always feel like the dissenting voice :) We release waaay too many files already. In another couple of years I will be uploading 1GB of diced permutations of our bits (let us not forget we are not even doing truffle dists yet).

Member

enebo commented Mar 3, 2015

The original problem was dist was too big and the answer is to continue disting the big file and add more dist files to the release? I don't think this is actually solving a problem. I also don't think most users know what pack200 is and so they won't download it because they won't think they can unpack it. pack200, to me, fits better into installers since it can be hidden from the person installing.

I always feel like the dissenting voice :) We release waaay too many files already. In another couple of years I will be uploading 1GB of diced permutations of our bits (let us not forget we are not even doing truffle dists yet).

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 3, 2015

Member
Member

mkristian commented Mar 3, 2015

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 3, 2015

Member
Member

mkristian commented Mar 3, 2015

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 16, 2015

Member

@mkristian Yes, cherry-pick any size improvements other than pack200 stuff to master. Perhaps some day we'll explore pack200 distribution.

Member

headius commented Mar 16, 2015

@mkristian Yes, cherry-pick any size improvements other than pack200 stuff to master. Perhaps some day we'll explore pack200 distribution.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 17, 2015

Member

@headius the explosion of the size of most distribution artifacts for 9.0.0.0.pre1 came through the embedded jruby-complete-1.7.19 inside the jruby-readline setup. this I fixed a while ago when I released jruby-readline-1.0.gem to maven central.

beside this there are quite a few place to reduce amount of maven modules and attached artifacts but that is not part of the pack200 branch.

Member

mkristian commented Mar 17, 2015

@headius the explosion of the size of most distribution artifacts for 9.0.0.0.pre1 came through the embedded jruby-complete-1.7.19 inside the jruby-readline setup. this I fixed a while ago when I released jruby-readline-1.0.gem to maven central.

beside this there are quite a few place to reduce amount of maven modules and attached artifacts but that is not part of the pack200 branch.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Mar 17, 2015

Member

so I reworked the artifacts which gets "deployed" to maven central:

  • two less maven modules:
    • maven/jruby-noasm is gone
    • maven/jruby-stdlib is part of lib/
  • three less deployed artifacts:
    • jruby-noasm.jar
    • jruby-core-noasm.jar which is the jruby-core.jar now
    • jruby-core-complete.jar is gone

the jruby-core pom uses the dependency-reduced pom after shading asm and jnr-jffi (which uses asm).

this change reduced the upload on release about 20MB.

finally I added jruby-dist-bin200.tar.gz which is 17MB. here I also replaced the bin/jruby script with one which unpacks all the jars when used the first time before executing the jruby. any further bin/jruby execution works as usual - on linux ONLY or when there is a bash.

so overall it is a reduction of attached artifacts and total upload size :)

see more details on pack200 branch.

once maven-3.3.0 is released I will work on the ruby-maven-libs.gem and reduce the dist size even more.

Member

mkristian commented Mar 17, 2015

so I reworked the artifacts which gets "deployed" to maven central:

  • two less maven modules:
    • maven/jruby-noasm is gone
    • maven/jruby-stdlib is part of lib/
  • three less deployed artifacts:
    • jruby-noasm.jar
    • jruby-core-noasm.jar which is the jruby-core.jar now
    • jruby-core-complete.jar is gone

the jruby-core pom uses the dependency-reduced pom after shading asm and jnr-jffi (which uses asm).

this change reduced the upload on release about 20MB.

finally I added jruby-dist-bin200.tar.gz which is 17MB. here I also replaced the bin/jruby script with one which unpacks all the jars when used the first time before executing the jruby. any further bin/jruby execution works as usual - on linux ONLY or when there is a bash.

so overall it is a reduction of attached artifacts and total upload size :)

see more details on pack200 branch.

once maven-3.3.0 is released I will work on the ruby-maven-libs.gem and reduce the dist size even more.

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jun 1, 2015

Member

just a follow up:

with the new ruby-maven and ruby-maven-libs gem it is finally possible to install those gems as default gems inside the jruby-dist files which will reduce the size by at least 8MB

Member

mkristian commented Jun 1, 2015

just a follow up:

with the new ruby-maven and ruby-maven-libs gem it is finally possible to install those gems as default gems inside the jruby-dist files which will reduce the size by at least 8MB

@mkristian mkristian added this to the JRuby 9.0.0.0.rc1 milestone Jun 1, 2015

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Jun 26, 2015

Member

@mkristian are these new versions on master now as default gems? Can this be closed now?

Member

enebo commented Jun 26, 2015

@mkristian are these new versions on master now as default gems? Can this be closed now?

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jun 26, 2015

Member

@enebo the new versions are on master but not as default gems. give me the
weekend to finalize it. had already done something but those scripts are on
my old laptop which is not with me right now.

On Fri, Jun 26, 2015 at 5:34 PM, Thomas E Enebo notifications@github.com
wrote:

@mkristian https://github.com/mkristian are these new versions on
master now as default gems? Can this be closed now?


Reply to this email directly or view it on GitHub
#2487 (comment).

Member

mkristian commented Jun 26, 2015

@enebo the new versions are on master but not as default gems. give me the
weekend to finalize it. had already done something but those scripts are on
my old laptop which is not with me right now.

On Fri, Jun 26, 2015 at 5:34 PM, Thomas E Enebo notifications@github.com
wrote:

@mkristian https://github.com/mkristian are these new versions on
master now as default gems? Can this be closed now?


Reply to this email directly or view it on GitHub
#2487 (comment).

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Jun 26, 2015

Member

@mkristian okie dokie...just checking

Member

enebo commented Jun 26, 2015

@mkristian okie dokie...just checking

@mkristian

This comment has been minimized.

Show comment
Hide comment
@mkristian

mkristian Jul 2, 2015

Member

done on both branches

Member

mkristian commented Jul 2, 2015

done on both branches

@mkristian mkristian closed this Jul 2, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment