Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Separate compilation of module-info.java #72

Closed
tlinkowski opened this issue Mar 21, 2019 · 14 comments

Comments

Projects
None yet
6 participants
@tlinkowski
Copy link
Contributor

commented Mar 21, 2019

Overview

I'd like to request a feature for library developers who need to target JDK 6-8 but who also want to provide a module-info.class (as per Option 3 by @jodastephen).

The proposed feature has two parts:

  1. Low-level API: separate compilation of module-info.java.
  2. High-level API: easy setting of javac --release option, separately for main Java code and for module-info.java.

Justification

General

Library maintainers need to target JDK 6-8 because these JDKs (especially JDK 8) are still very popular (taking ~95% share in 2018, and predicted to take ~90% in 2019).

Still, those maintainers could make the most of JPMS by:

  • providing module-info.class for consumers who put the library on module path,
  • compiling module-info.java against the remaining classes of this module and against other modules (which provides better encapsulation and prevents split packages).

So providing a module-info.class is like saying:

Hey, I'm JPMS-compatible! You can safely put me on the module path — I guarantee to have no split packages!

To sum up, I see this feature request as great means of promoting JPMS! Would you agree, @paulbakker, @sandermak?

Examples

Here are some popular libraries that provide module-info.class while targeting JDK 6-8, e.g.:

  • Maven:
  • Ant:
    • Lombok (JDK 9 module-info.class in root dir)

There are also some libraries that:

Vavr /ˈveɪ.vɚ/

Let's have a closer look at vavr's case:

  1. It's a quite popular library (3000+ Github stars) targeting JDK 8.
  2. Its creator, @danieldietrich, is (or was?) a believer in JPMS.
  3. However, as vavr-io/vavr#2230 (comment) shows, he backed out of it (partly because the required Gradle configuration was too complex).
  4. I personally think it's a shame if such a popular library cannot be easily shipped with a module-info.class while still targeting JDK 8.

Proposed DSL

Note: Everything in this section (especially naming) is TBD.

Low-level API

Extra property on moduleOptions extension:

build.gradle
compileJava.moduleOptions.compileModuleInfoSeparately = true
build.gradle.kts
tasks {
  compileJava.moduleOptions.compileModuleInfoSeparately = true
}

High-level API

Special modularity extension with two functions:

  • for releasing to JDK 6-8:
    • mixedJavaRelease(int mainJavaRelease, int moduleInfoJavaRelease = 9)
  • for releasing to JDK 9+
    • standardJavaRelease(int mainJavaRelease)

For example, the most common configuration (JDK 8 + JDK 9 module-info.class) would be a one-liner:

build.gradle
modularity.mixedJavaRelease 8
build.gradle.kts
modularity.mixedJavaRelease(8)

Proposed behavior

module-info.java location

I propose to leave module-info.java in src/main/java (I don't see any need to put it in a separate source set directory).

module-info.class location

There are two places where module-info.class can end up:

  1. in the root output directory (natural; corresponds to module-info.java location)
  2. in META-INF/versions/9 (Multi-Release JAR, AKA MRJAR)

Having read a post on MRJARs by @melix, I'm rather suspicious of MRJARs, and so I prefer option 1.

On the other hand, @gunnarmorling claims that it's better to use option 2 because module-info.class "will be out of the way in most cases when being used on an older Java version". But I don't really know why, because as far as I understand, any tool that scans the classpath (like Guava's ClassPath) will return module-info.class no matter where it is (based on its .class extension). @gunnarmorling, would you care to give me a pointer here?

Alternatives

ModiTect

It's fair to mention ModiTect (by @gunnarmorling) and its Gradle plugin (by @siordache) here.

However, ModiTect does much more than I request here: ModiTect actually generates module-info.class from a special notation (there's even no module-info.java there edit: it can actually parse module-info.java). Personally, I find it too complex for my needs.

Badass-Jar plugin

@siordache also created a Gradle plugin that lets one "seamlessly create modular jars that target a Java release before 9".

It looks quite nice, however:

  • I'm not sure how it plays along with org.javamodularity.moduleplugin (and I'd like to use it for patching modules, testing on module-path, etc.)
  • in order to build the proper JAR and make sure your module-info.java is correct, you actually have to run the build twice (./gradlew jar and ./gradlew -PjavaCompatibility=9 jar) - too complex
  • it doesn't use javac's --release option, which (as opposed to using -source and -target) guarantees that only the right APIs are referenced (e.g. it throws when compiling optional.isEmpty() with --release 8)
  • it produces only Multi-Release JARs, and I'm somewhat skeptical about them (as I mentioned before)

Related StackOverflow questions

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Mar 21, 2019

java9-modularity#72: initial refactorings/improvements
necessary for further commits

MAIN:
1) renamed TestModuleOptions.isRunOnClasspath() to getRunOnClasspath() (otherwise won't work with Kotlin DSL) + added a Kotlin DSL example for "runOnClasspath = true" to README.md
2) introduced JavaProjectHelper and applied it to CompileTask
3) added comments for all anonymous classes that should not be removed
4) minor improvements in ModuleSystemPlugin
5) minor fixes in test-project-kotlin/README.md

TEST:
1) bumped smoke-test Gradle version to 5.0 (for improved Kotlin DSL)
2) introduced a no-op "moduleOptions" access in greeter.api (for testing DSL)
3) introduced SmokeTestHelper for ModulePluginSmokeTest
@sandermak

This comment has been minimized.

Copy link

commented Mar 21, 2019

Thanks for the great overview. Definitely a good usecase to support!

@danieldietrich

This comment has been minimized.

Copy link

commented Mar 21, 2019

Its creator, @danieldietrich, is (or was?) a believer in JPMS.

It seems no one is adopting JPMS... Some believe it has no future.

@siordache

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

@tlinkowski The badass-jar plugin is probably what you're looking for.

@tlinkowski

This comment has been minimized.

Copy link
Contributor Author

commented Mar 21, 2019

It seems no one is adopting JPMS... Some believe it has no future.

@danieldietrich I guess it's because the adoption of JDK 9+ seems very small yet. But if it was easy to bundle a module-info.class and the popular libraries started doing so, then perhaps in 5-10 years JPMS could be in common use 🙂

@tlinkowski

This comment has been minimized.

Copy link
Contributor Author

commented Mar 21, 2019

@siordache Thanks! I forgot to mention your plugin (I think I stumbled upon it before). I'll edit the overview.

@gunnarmorling

This comment has been minimized.

Copy link

commented Mar 21, 2019

Hey, thanks for the mention! On this one:

ModiTect actually generates module-info.class from a special notation (there's even no module-info.java there). Personally, I find it too complex for my needs.

This is only one mode of operation in ModiTect. You also can give the contents of a module-info.java file verbatim in your pom.xml or build.gradle file; alternatively, its contents can also be obtained from a given file in the project. This will be compiled as is to the class file, using ASM. This is also the mode that works on JDK < 9. So it seems it already pretty much does what you are after (IIUC)?

@gunnarmorling

This comment has been minimized.

Copy link

commented Mar 21, 2019

But I don't really know why, because as far as I understand, any tool that scans the classpath (like Guava's ClassPath) will return module-info.class no matter where it is (based on its .class extension).

It depends on the specific implementation of scanning. But we had definitely issues with that, e.g. with older versions of Jetty stumbling upon the file. Note that scanning often isn't done using reflection on the classpath but by "manually" reading class files using ASM or similar. Older tools doing that will usually not be aware of classfiles under META-INF/versions.

@siordache

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

I'm not sure how it plays along with org.javamodularity.moduleplugin (and I'd like to use it for patching modules, testing on module-path, etc.)

It actually uses the org.javamodularity.moduleplugin, as mentioned in the README:

"If the plugin detects that at least one of sourceCompatibility and targetCompatibility has an effective value >= 9, it automatically applies the gradle-modules-plugin, which adds support for the JPMS."

it produces only Multi-Release JARs, and I'm somewhat skeptical about them (as I mentioned before)

You can prevent the creation of a multi-release jar by setting multiRelease to false in the jar section of your build script:

jar {
    ...
    multiRelease = false
    ...
}

However, you are right about the plugin not using the --release option and the need to run the build twice to make sure that the module-info.java is correct.

@tlinkowski

This comment has been minimized.

Copy link
Contributor Author

commented Mar 21, 2019

its (=module-info.java file) contents can also be obtained from a given file in the project. This will be compiled as is to the class file, using ASM.

@gunnarmorling Thanks, I didn't know ModiTect can do it!

Still, I'm afraid that using it that way would be a bit like using a sledgehammer to crack a nut 😉 Plus, I'd feel safer to have my module-info.class generated by javac.


It depends on the specific implementation of scanning. But we had definitely issues with that, e.g. with older versions of Jetty stumbling upon the file. Note that scanning often isn't done using reflection on the classpath but by "manually" reading class files using ASM or similar. Older tools doing that will usually not be aware of classfiles under META-INF/versions.

As far as I understand, we're talking about tools that scan the classpath for .class files, but deliberately omit META-INF directory. Makes sense! On the other hand, an "older" tool like Guava's ClassPath (which doesn't use reflection - it uses URLClassLoader.getURLs) doesn't make this assumption, and breaks in both cases 😉

To sum up, if I understand correctly:

  • Jetty is already fixed to omit module-info.class
  • releasing as Multi-Release JAR becomes increasingly less useful (but might still be useful, true)

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Mar 21, 2019

java9-modularity#72: initial refactorings/improvements
necessary for further commits

MAIN:
1) renamed TestModuleOptions.isRunOnClasspath() to getRunOnClasspath() (otherwise won't work with Kotlin DSL) + added a Kotlin DSL example for "runOnClasspath = true" to README.md
2) introduced JavaProjectHelper and applied it to CompileTask
3) added comments for all anonymous classes that should not be removed
4) minor improvements in ModuleSystemPlugin
5) minor fixes in test-project-kotlin/README.md

TEST:
1) bumped smoke-test Gradle version to 5.0 (for improved Kotlin DSL)
2) introduced a no-op "moduleOptions" access in greeter.api (for testing DSL)
3) introduced SmokeTestHelper for ModulePluginSmokeTest
4) disabled stackTraceFilters in all tests
5) updated Kotlin to 1.3.20

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Mar 21, 2019

java9-modularity#72: support for "moduleOptions.compileModuleInfoSepa…
…rately"

added CompileModuleOptions and CompileModuleInfoTask

NOTE: potentially breaking change for "compileJava" in Kotlin DSL (see modified "build.gradle.kts")

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Mar 21, 2019

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Mar 21, 2019

java9-modularity#72: added test-project-mixed
tests the "modularity" extension (incl. verifying generated class file formats)
@tlinkowski

This comment has been minimized.

Copy link
Contributor Author

commented Mar 21, 2019

With #73, I provided a complete implementation of this proposal (with tests & documentation).

Of course, this issue is still open for discussion, and I'm ready to adapt the provided PR as needed.

@gunnarmorling

This comment has been minimized.

Copy link

commented Mar 21, 2019

Still, I'm afraid that using it that way would be a bit like using a sledgehammer to crack a nut 😉 Plus, I'd feel safer to have my module-info.class generated by javac.

I think there's nothing to be worried about, ASM works just fine. The reasoning for using it btw. was to enable the build itself running on older JDKs, which naturally can't be done when using javac to compile the descriptor. Another thing I'm curious about: how does the suggest approach work exactly, will the classes be compiled twice, once with module descriptor and once without?

Personally, I find [ModiTect] too complex for my needs.

That's interesting, because it's actually intended to reduce complexity. E.g. my thinking is that specifying exported packages via wildcards can help to reduce efforts, similar to "requires" clauses derived from dependencies. So I'm curious which complexity you actually perceive?

Not I'm welcoming this effort and obviously I've no stake in deciding whether this should be included here or not; I'm just curious to see what could be improved in ModiTect to further improve it.

@tlinkowski

This comment has been minimized.

Copy link
Contributor Author

commented Mar 21, 2019

Thanks for your response, Gunnar!

I think there's nothing to be worried about, ASM works just fine.

Agreed that ASM works fine 🙂 What I'm worried about is that Java compiler (javac) is a really complex tool. Here, I simply propose a thin wrapper over this tool. ModiTect, on the other hand, is another complex tool that (in this use case) duplicates what javac does.

Still, ModiTect is a great tool, and I'd use it if the benefits for me:

  • "defining dependence clauses based on your project's dependencies" 👍
  • "describing exported and opened packages with patterns" 👍
  • "auto-detecting service usages" 👍
  • "generating module-info.java descriptors" 👍
  • complex, highly configurable tool:
    • I can probably do with it what I need 👍

outweighed the costs for me:

  • no IntelliJ support for indirect module declarations (I'm spoiled about such things):
    • no instant static type safety 👎
    • no click-through to a given package/class 👎
  • complex, highly configurable tool:
    • another thing to learn & maintain its usage 👎
    • higher probability of having issues, e.g. moditect/moditect#90 👎

      ModiTect is at an early stage of development and it still has some rough edges. Use it at your own risk.

  • (strictly project-specific) I have small projects where I export, require, use, and provide little 👎

The reasoning for using it btw. was to enable the build itself running on older JDKs, which naturally can't be done when using javac to compile the descriptor.

This is certainly an advantage of ModiTect! Only, I don't need to run the build on an older JDK. I only need to target users who might run on an older JDK.

Another thing I'm curious about: how does the suggest approach work exactly, will the classes be compiled twice, once with module descriptor and once without?

A good question! Actually, every class is compiled only once:

  1. First, all classes except module-info.java are compiled with --release 6-8.
  2. Then, only module-info.java is compiled with --release 9+ while all the previously compiled classes are accessible to it on the classpath (because both compilation tasks share the same output directory).

That's interesting, because it's actually intended to reduce complexity. E.g. my thinking is that specifying exported packages via wildcards can help to reduce efforts, similar to "requires" clauses derived from dependencies. So I'm curious which complexity you actually perceive?

Yes, these are cool features that reduce complexity of usage (if we don't count lack of IDE support). The complexity I perceive is primarily the complexity of the tool itself (another thing to learn, higher risk of issues).

Not I'm welcoming this effort and obviously I've no stake in deciding whether this should be included here or not; I'm just curious to see what could be improved in ModiTect to further improve it.

ModiTect is a great tool but:

  • until Oracle picks your ideas up,
  • or at least until there's an IntelliJ plugin for it,
  • or unless I had a larger, more complex project where the benefits could be higher,

I'd probably decide against using it.

@paulbakker

This comment has been minimized.

Copy link
Collaborator

commented Mar 22, 2019

@tlinkowski This is great! Thanks for the detailed writeup and PR. I'm fully onboard with supporting this, seems like a great feature to further increase JPMS adoption.
I'll need a few days before I can give it the attention it deserves (trip coming up this weekend), but I'll get to it as soon as possible.

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Apr 3, 2019

java9-modularity#72: initial refactorings/improvements
necessary for further commits

MAIN:
1) renamed TestModuleOptions.isRunOnClasspath() to getRunOnClasspath() (otherwise won't work with Kotlin DSL) + added a Kotlin DSL example for "runOnClasspath = true" to README.md
2) introduced JavaProjectHelper and applied it to CompileTask
3) added comments for all anonymous classes that should not be removed
4) minor improvements in ModuleSystemPlugin
5) minor fixes in test-project-kotlin/README.md

TEST:
1) bumped smoke-test Gradle version to 5.0 (for improved Kotlin DSL)
2) introduced a no-op "moduleOptions" access in greeter.api (for testing DSL)
3) introduced SmokeTestHelper for ModulePluginSmokeTest
4) disabled stackTraceFilters in all tests
5) updated Kotlin to 1.3.20

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Apr 3, 2019

java9-modularity#72: support for "moduleOptions.compileModuleInfoSepa…
…rately"

added CompileModuleOptions and CompileModuleInfoTask

NOTE: potentially breaking change for "compileJava" in Kotlin DSL (see modified "build.gradle.kts")

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Apr 3, 2019

tlinkowski added a commit to tlinkowski/gradle-modules-plugin that referenced this issue Apr 3, 2019

java9-modularity#72: added test-project-mixed
tests the "modularity" extension (incl. verifying generated class file formats)

paulbakker added a commit that referenced this issue Apr 4, 2019

#72: initial refactorings/improvements
necessary for further commits

MAIN:
1) renamed TestModuleOptions.isRunOnClasspath() to getRunOnClasspath() (otherwise won't work with Kotlin DSL) + added a Kotlin DSL example for "runOnClasspath = true" to README.md
2) introduced JavaProjectHelper and applied it to CompileTask
3) added comments for all anonymous classes that should not be removed
4) minor improvements in ModuleSystemPlugin
5) minor fixes in test-project-kotlin/README.md

TEST:
1) bumped smoke-test Gradle version to 5.0 (for improved Kotlin DSL)
2) introduced a no-op "moduleOptions" access in greeter.api (for testing DSL)
3) introduced SmokeTestHelper for ModulePluginSmokeTest
4) disabled stackTraceFilters in all tests
5) updated Kotlin to 1.3.20

paulbakker added a commit that referenced this issue Apr 4, 2019

#72: support for "moduleOptions.compileModuleInfoSeparately"
added CompileModuleOptions and CompileModuleInfoTask

NOTE: potentially breaking change for "compileJava" in Kotlin DSL (see modified "build.gradle.kts")

paulbakker added a commit that referenced this issue Apr 4, 2019

#72: support for "modularity" project extension
via ModularityExtension interface

paulbakker added a commit that referenced this issue Apr 4, 2019

#72: added test-project-mixed
tests the "modularity" extension (incl. verifying generated class file formats)
@paulbakker

This comment has been minimized.

Copy link
Collaborator

commented Apr 4, 2019

Merged the work done by @tlinkowski! Will be in the next release (which I plan to publish today)

@paulbakker paulbakker closed this Apr 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.