Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce dependencies from Full JRE to compact1 profile #10559

Closed
avkonst opened this issue Oct 19, 2017 · 9 comments
Closed

Reduce dependencies from Full JRE to compact1 profile #10559

avkonst opened this issue Oct 19, 2017 · 9 comments
Assignees
@adriaanm @dwijnand
Labels
jdk9 [obsolete]
Milestone
2.13.0-M5

Comments

@avkonst
Copy link

avkonst commented Oct 19, 2017 •
edited
Loading

We have got few applications written in Scala. Now we would like to reduce footprint of deliverable binaries. Two reasons: be able to deploy to embedded IoT-like devices and reduce size of docker images.

Resolution of this ticket will also make very positive effect on the size of Java 9 modules required by scala-library, when scala becomes ready for Java 9 jlink tool.

We downsized binaries of compiled OpenJDK JRE from 90MB to 13MB (we achieve it with minimal1 variant of stripped libjvm and compressed compact1 profile of rt.jar). This is big gain, but scala libraries have got unfortunate dependencies which trigger Full JRE requirement instead of compact1.

There 3 places of interest:

  • scala.beans (org.scala-lang.scala-library-2.12.2.jar) -> java.beans (which is only avaialble in Full JRE profile of rt.jar)
  • scala.sys.process (org.scala-lang.scala-library-2.12.2.jar) -> java.lang.management (which is available in compact3 profile of rt.jar, FYI: compact3 profile is 2 times bigger in size than compact1)
  • scala.reflect.internal (org.scala-lang.scala-reflect-2.12.2.jar) -> java.rmi (which is available in compact2 profile of rt.jar, FYI: compact2 profile is about 50% bigger than compact1)

I wonder if it is possible to reduce these dependencies?

Maybe there is a way to disable part of functionally by testing dependency class availability in runtime and using reflection to invoke it?

Alternatively, it maybe possible to split 2 scala jars into more pieces, for example: libs required only for compilation and libs required for runtimes with various profiles (like scala-library-compact1, scala-library-compact2, scala-library-compact3, scala-library-extra instead of a single big one).

I am keen on to contribute the change if you could advise what can be done.

Here are the details and commands, which I used to analyse this:

>  jdeps -profile org.scala-lang.scala-library-2.12.2.jar | egrep -v ".jar$"
org.scala-lang.scala-library-2.12.2.jar -> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar (Full JRE)
   scala (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.lang.reflect                                  compact1
      -> java.util                                          compact1
      -> java.util.concurrent.locks                         compact1
      -> java.util.stream                                   compact1
   scala.annotation (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
   scala.annotation.meta (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
   scala.annotation.unchecked (org.scala-lang.scala-library-2.12.2.jar)
   scala.beans (org.scala-lang.scala-library-2.12.2.jar)
      -> java.beans                                         Full JRE
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.reflect                                  compact1
   scala.collection (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
   scala.collection.concurrent (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.concurrent.atomic                        compact1
   scala.collection.convert (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.function                                 compact1
   scala.collection.generic (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util.concurrent.atomic                        compact1
   scala.collection.immutable (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.regex                                    compact1
   scala.collection.mutable (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.reflect                                  compact1
      -> java.util                                          compact1
      -> java.util.regex                                    compact1
      -> java.util.stream                                   compact1
   scala.collection.parallel (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.concurrent.atomic                        compact1
   scala.collection.parallel.immutable (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util.concurrent.atomic                        compact1
   scala.collection.parallel.mutable (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.collection.script (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.compat (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.reflect                                  compact1
      -> java.nio.charset                                   compact1
      -> java.util                                          compact1
   scala.concurrent (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.concurrent.atomic                        compact1
   scala.concurrent.duration (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util.concurrent                               compact1
   scala.concurrent.forkjoin (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
   scala.concurrent.impl (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.concurrent.atomic                        compact1
      -> java.util.concurrent.locks                         compact1
   scala.io (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.net                                           compact1
      -> java.nio                                           compact1
      -> java.nio.charset                                   compact1
      -> java.text                                          compact1
   scala.math (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.math                                          compact1
      -> java.util                                          compact1
   scala.ref (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.util                                          compact1
   scala.reflect (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.annotation                               compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.reflect                                  compact1
   scala.reflect.macros.internal (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
   scala.runtime (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.annotation                               compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.lang.reflect                                  compact1
      -> java.util                                          compact1
      -> java.util.stream                                   compact1
   scala.runtime.java8 (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
   scala.sys (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.security                                      compact1
      -> java.util                                          compact1
   scala.sys.process (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.management                               compact3
      -> java.net                                           compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
   scala.text (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.util (org.scala-lang.scala-library-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.jar                                      compact1
   scala.util.control (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.util.hashing (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.util.matching (org.scala-lang.scala-library-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.util                                          compact1
      -> java.util.regex                                    compact1

and:

> jdeps -profile org.scala-lang.scala-reflect-2.12.2.jar | egrep -v ".jar$" | grep -v "not found"
org.scala-lang.scala-reflect-2.12.2.jar -> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar (compact2)
   scala.reflect.api (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.reflect.internal (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.annotation                               compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.lang.reflect                                  compact1
      -> java.rmi                                           compact2
      -> java.security                                      compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
   scala.reflect.internal.annotations (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
   scala.reflect.internal.pickling (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.reflect.internal.settings (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
   scala.reflect.internal.tpe (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.reflect.internal.transform (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
   scala.reflect.internal.util (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.lang.reflect                                  compact1
      -> java.net                                           compact1
      -> java.security                                      compact1
      -> java.security.cert                                 compact1
      -> java.util                                          compact1
      -> java.util.concurrent                               compact1
      -> java.util.concurrent.atomic                        compact1
      -> java.util.regex                                    compact1
   scala.reflect.io (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.reflect                                  compact1
      -> java.net                                           compact1
      -> java.nio.charset                                   compact1
      -> java.nio.file                                      compact1
      -> java.nio.file.attribute                            compact1
      -> java.util                                          compact1
      -> java.util.jar                                      compact1
      -> java.util.zip                                      compact1
   scala.reflect.macros (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
      -> java.lang.invoke                                   compact1
      -> java.net                                           compact1
   scala.reflect.macros.blackbox (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
   scala.reflect.macros.whitebox (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.lang                                          compact1
   scala.reflect.runtime (org.scala-lang.scala-reflect-2.12.2.jar)
      -> java.io                                            compact1
      -> java.lang                                          compact1
      -> java.lang.annotation                               compact1
      -> java.lang.invoke                                   compact1
      -> java.lang.ref                                      compact1
      -> java.lang.reflect                                  compact1
      -> java.net                                           compact1
      -> java.nio.charset                                   compact1
      -> java.util                                          compact1
      -> java.util.concurrent.atomic                        compact1
      -> java.util.concurrent.locks                         compact1
@SethTisue
Copy link
Member

highly relevant here: scala/scala#5677 (with more such work planned before 2.13 ships, and participation highly welcome). so for example scala.sys will move to a module, which will fix the compact3 requirement you found.

I'm curious about the java.rmi dependency. what's that doing, have you looked yet...?

@SethTisue
Copy link
Member

I think the Scala.js and Scala Native projects would appreciate work in this direction as well (hello @sjrd, @densh)

@avkonst
Copy link
Author

avkonst commented Oct 19, 2017

No, I have not looked into rmi dependency, but I did look into java.beans dependency. And it turned out it is very strong dependency based on inheritance from java.beans.SimpleBeanInfo.

@gourlaysama
Copy link
Member

The java.rmi dependency is brought in by compiler support for the scala.remote annotation, but it was deprecated in 2.12, so it should be possible to cut that dependency in 2.13.

@adriaanm
Copy link
Contributor

adriaanm commented Nov 6, 2017 •
edited
Loading

I'll submit a PR that removes all dependencies on @remote for 2.13.0-M3, so that we can remove it entirely once we compile the compiler with M3.

@adriaanm adriaanm mentioned this issue Nov 6, 2017
Merged
@adriaanm
Copy link
Contributor

adriaanm commented Nov 6, 2017

Yay! Library and reflect now only depend on the compact1 profile: scala/scala#6164

@adriaanm adriaanm mentioned this issue Nov 6, 2017
Closed
@avkonst
Copy link
Author

avkonst commented Nov 7, 2017

Great! Big step forward.

smarter added a commit to dotty-staging/dotty that referenced this issue Dec 7, 2017
@smarter
Restrict dotty-compiler to the jvm compact1 profile
d060c7a 
As verify by running `jdep -s -P` on the dotty-compiler jar. See
scala/bug#10559 for why this is useful and
scala/scala#6164 for how this was implemented in
scalac.
@smarter smarter mentioned this issue Dec 7, 2017
Merged
smarter added a commit to dotty-staging/dotty that referenced this issue Dec 7, 2017
@smarter
Restrict dotty-compiler to the jvm compact1 profile
651f92f 
As verify by running `jdep -s -P` on the dotty-compiler jar. See
scala/bug#10559 for why this is useful and
scala/scala#6164 for how this was implemented in
scalac.
smarter added a commit to dotty-staging/dotty that referenced this issue Dec 7, 2017
@smarter
Restrict dotty-compiler to the jvm compact1 profile
c30de2d 
As verify by running `jdep -s -P` on the dotty-compiler jar. See
scala/bug#10559 for why this is useful and
scala/scala#6164 for how this was implemented in
scalac.
vitorsvieira pushed a commit to vitorsvieira/dotty that referenced this issue Dec 11, 2017
@smarter @vitorsvieira
Restrict dotty-compiler to the jvm compact1 profile
7913642 
As verify by running `jdep -s -P` on the dotty-compiler jar. See
scala/bug#10559 for why this is useful and
scala/scala#6164 for how this was implemented in
scalac.
@adriaanm adriaanm added this to the 2.13.0-M3 milestone Jan 31, 2018
@adriaanm adriaanm self-assigned this Jan 31, 2018
@milessabin milessabin modified the milestones: 2.13.0-M3, 2.13.0-M4 Mar 9, 2018
@lrytz lrytz modified the milestones: 2.13.0-M4, 2.13.0-M5 Apr 19, 2018
@dwijnand
Copy link
Member

What's left to do to close this ticket?

@adriaanm
Copy link
Contributor

Add the test to the build to close scala/scala-dev#437

dwijnand added a commit to dwijnand/scala that referenced this issue Jun 14, 2018
@dwijnand
Enforce library/reflect restriction to compat1 profile
3528ca6 
Enforcement of scala#6164

Tested manually by reverting the sys.process.javaVmArguments change
07ff7ac and then running testJDepsImpl:

    > testJDeps
    [info] Compiling 1 Scala source to /d/scala/build/quick/classes/library...
    [info] Packaging /d/scala/build/pack/lib/scala-library.jar ...
    [info] Packaging /d/scala/build/pack/lib/scala-reflect.jar ...
    [info] Done packaging.
    [info] Done packaging.
    [trace] Stack trace suppressed: run last root/*:testJDeps for the full output.
    [error] (root/*:testJDeps) Detected dependency outside of compact1:
    [error] scala-library.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/rt.jar (compact3)
    [error] scala-reflect.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/rt.jar (compact1)
    [error] scala-reflect.jar -> /d/scala/build/pack/lib/scala-library.jar
    [error] Total time: 4 s, completed 14-Jun-2018 14:20:02

Fixes scala/scala-dev#437
Fixes scala/bug#10559
@dwijnand dwijnand mentioned this issue Jun 14, 2018
Merged
@dwijnand dwijnand self-assigned this Jun 14, 2018
dwijnand added a commit to dwijnand/scala that referenced this issue Jun 14, 2018
@dwijnand
Enforce library/reflect restriction to compat1 profile
7ddfd3c 
Enforcement of scala#6164

Tested manually by reverting the sys.process.javaVmArguments change
07ff7ac and then running testJDepsImpl:

    > testJDeps
    [info] Compiling 1 Scala source to /d/scala/build/quick/classes/library...
    [info] Packaging /d/scala/build/pack/lib/scala-library.jar ...
    [info] Packaging /d/scala/build/pack/lib/scala-reflect.jar ...
    [info] Done packaging.
    [info] Done packaging.
    [trace] Stack trace suppressed: run last root/*:testJDeps for the full output.
    [error] (root/*:testJDeps) Detected dependency outside of compact1:
    [error] scala-library.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/rt.jar (compact3)
    [error] scala-reflect.jar -> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/rt.jar (compact1)
    [error] scala-reflect.jar -> /d/scala/build/pack/lib/scala-library.jar
    [error] Total time: 4 s, completed 14-Jun-2018 14:20:02

Fixes scala/scala-dev#437
Fixes scala/bug#10559
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adriaanm adriaanm

@dwijnand dwijnand

Labels
jdk9 [obsolete]
Projects
None yet
Milestone
2.13.0-M5
Development

No branches or pull requests
7 participants
@adriaanm @lrytz @milessabin @SethTisue @dwijnand @gourlaysama @avkonst