OSGi Java 9 support: Multi-release JAR

[njbartlett]

This issue was raised against Felix (maven-bundle-plugin) but it needs to be addressed in bnd: https://issues.apache.org/jira/browse/FELIX-5592

[njbartlett]

See also https://issues.apache.org/jira/browse/FELIX-5527

[bjhargrave]

Some more detail on what "support" for multi release jars would look like in OSGi is needed here.

Bnd could certain stop objecting to classes in META-INF/versions/9 but that would not make them available at runtime IF the runtime is Java 9.

This topic was discussed in a recent OSGi expert group call and the OSGi "way" here would be to use a fragment to supply the types for a java version with the host bundle carving out a space in the Bundle-Classpath.

But if Bnd is being fed a target/classes folder with classes in META-INF/versions/9, I am not sure how Bnd can make a bundle which will work as expected for multi release jars.

[bjhargrave]

Also, if classes in META-INF/versions/9 exist, should Bnd process them for imports? The imports for the bundle may vary depending upon whether the classes in META-INF/versions/9 are "in effect". And there is no way to have imports conditional on the version of Java at runtime. This is why fragments to supply these types is more appropriate in OSGi since the fragments can supply additional imports (and exports) for the types.

[bjhargrave]

How do JPMS modules and multi-release jars interact? I assume they are mutually exclusive since JPMS modules would have the same issues with varying requires/exports depending upon the runtime version of Java. Yes, this is not a practical issue yet for JPMS since it is only Java 9 for now. But with Java 10, then what?

[njbartlett]

Unfortunately the version-specific part under META-INF/versions can override the non-version-specific classes. It seems that a multi-release JAR would have to result in at least a base bundle plus one fragment for Java 9 and a further fragment for the pre-Java-9 code. We might be able to optimise this if we detect that there is actually no overlap.

[njbartlett]

You can have module-info.class under META-INF/versions. So you could have entirely different dependencies for Java 9, 10, etc.

[bjhargrave]

We don't need a fragment for pre-Java 9 code. The host bundle would have all the pre-9 code and Bundle-Classpath: versioned, . and the fragment would have all its code in the versioned folder.

[njbartlett]

Okay nice idea with Bundle-ClassPath.

[bjhargrave]

You can have module-info.class under META-INF/versions. So you could have entirely different dependencies for Java 9, 10, etc.

Hmm, so bnd could put different manifests under META-INF/versions and the framework would pick a manifest at runtime? (sigh)

META-INF/MANIFEST.MF (java 8 manifest)
META-INF/versions/9/META-INF/MANIFEST.MF (java 9 manifest)

[karlpauls]

The problem with the bundle classpath is that it needs to be conditional. So you would need an entry for each existing java version.

[njbartlett]

Yes I think that might need to be the way forward... runtime support for fragments nested inside a bundle. That way projects like log4j can just release a single JAR, which is what they want to do.

[bjhargrave]

The problem with the bundle classpath is that it needs to be conditional. So you would need an entry for each existing java version.

No. There will be multiple fragments which only resolve for the specific java version.

[karlpauls]

No. There will be multiple fragments which only resolve for the specific java version.

But they need to override each other.

[bjhargrave]

Yes I think that might need to be the way forward... runtime support for fragments nested inside a bundle. That way projects like log4j can just release a single JAR, which is what they want to do.

I think there are many issues here. Like how does Bundle.getEntry/getEntries treat this?

[bjhargrave]

But they need to override each other.

No. At most one of the fragments is attached and it supplies classes in the versioned folder.

[karlpauls]

No. At most one of the fragments is attached and it supplies classes in the versioned folder.

So you would duplicate all classes from the other versions that are not overriden in this version in each fragment in the chain?

[rotty3000]

Could we add some new "effective"-ness' which map to Java version X? Import-Package: \ com.foo,\ com.bar;effective=resolve-java9 Plus the Bundle-ClassPath?

…

On Wed, Dec 6, 2017 at 6:09 PM, BJ Hargrave ***@***.***> wrote: You can have module-info.class under META-INF/versions. So you could have entirely different dependencies for Java 9, 10, etc. Hmm, so bnd could put different manifests under META-INF/versions and the framework would pick a manifest at runtime? (sigh) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI9TKR4CisE1GdlTDIRmODvyhf5jsqbks5s9x6ngaJpZM4Q4tuA> .

-- *Raymond Augé* <http://www.liferay.com/web/raymond.auge/profile> (@rotty3000) Senior Software Architect *Liferay, Inc.* <http://www.liferay.com> (@liferay) Board Member & EEG Co-Chair, OSGi Alliance <http://osgi.org> (@OSGiAlliance)

[karlpauls]

the thing is you can have:

/A.class
/9/B.class
/10/C.class
/11/B.class

[bjhargrave]

So you would duplicate all classes from the other versions that are not overriden in this version in each fragment in the chain?

Why would you need to do that? Does multi release jars stack up all the versions? So that on Java 11, you search 11, then 10, then 9, then the base jar for a class?

[karlpauls]

Why would you need to do that? Does multi release jars stack up all the versions? So that on Java 11, you search 11, then 10, then 9, then the base jar for a class?

Yes

[bjhargrave]

Yes

That is in insane design. Good luck building such a jar and keeping it operational in all the versions of java...

And how do you build all the module-infos for all these versions of java (if you use JPMS)?

[karlpauls]

At least that is what I think http://openjdk.java.net/jeps/238 is saying

[karlpauls]

In the example above, when running on an MRJAR-aware Java 9 JDK, it would see the 9-specific versions of A and B and the general versions of C and D; on a future MRJAR-aware Java 10 JDK, it would see the 10-specific version of A and the 9-specific version of B;

So i would say yes, they stack.

[njbartlett]

I think the structure would look like this:

A.class (pre-Java 9 class)
B.class (pre-Java 9 class)
META-INF/
\- MANIFEST.MF:
      Bundle-SymbolicName: foo
      Bundle-ClassPath: versioned, .
      Require-Capability: osgi.ee; filter:='(osgi.ee=JavaSE-1.8)'
\- versions
  \- 9
      \- A.class (java 9 class)
         B.class (java 9 class)
         META-INF/MANIFEST.MF:
             Fragment-Host: foo
             Require-Capability: osgi.ee; filter:='(osgi.ee=JavaSE-9)'
  \- 10
      \- A.class (java 10 class)
         B.class (java 10 class)
         META-INF/MANIFEST.MF:
             Fragment-Host: foo
             Require-Capability: osgi.ee; filter:='(osgi.ee=JavaSE-10)'

A smaller problem is that the fragment for Java 9 would resolve on Java 10 because EEs include every previous EE. We would have to express a dependency on the EE of "exactly Java 9".

[karlpauls]

Part of me wonders if the solution is to really make them fragments inside the bundle. I.e., if a dir on the bundle-classpath has a META-INF/MANIFEST.mf we should just handle it like a fragment.

[karlpauls]

A smaller problem is that the fragment for Java 9 would resolve on Java 10 because EEs include every previous EE. We would have to express a dependency on the EE of "exactly Java 9".

No, I think we want exactely that.

[karlpauls]

Bundle-Classpath: META-INF/versions/10,META-INF/versions/9,.

and then what you just proposed as layout.

[bjhargrave]

I think we need to stop fixing this in this bug. This is sounding more than Bnd can address alone. Changes are need in the Core spec to address this. And then Bnd can support the Core changes.

[karlpauls]

I think we need to stop fixing this in this bug. This is sounding more than Bnd can address. Changes are need in the Core spec to address this. And then Bnd can support the Core changes.

This is what I told David last week and what made him bring it up in your call (and because I had the feeling that wasn't the conclusion of the call I pinged Neil about it today :-)

[njbartlett]

Bundle-Classpath: META-INF/10,META-INF/9,.

Yes I think this works, and bnd can generate that. It becomes tricky when the user wants to use Bundle-ClassPath for something else as well, but maybe that should just be disallowed.

Agree with BJ that bnd should follow CPEG.

[laeubi]

Just want to let you know that I started adding MultiRelese support similar to @rotty3000 idea see

• Add multi-release support to aQute.bnd.osgi.Jar #5331

but going a step further:

1. It is now possible to set the desired release on a jar, similar as you can specify --release on the java compiler
2. depending on the set release, the getResource and alike return a "stacked" unique view on the thing provided as versions, so from the outside it looks like one traditional jar file without multi-release as if I have unpacked and flatten it
3. It even support versioned manifests and module-infos and one can write out the whole package as a multi-release jar

Based on this, it would be possible to have a "release" flag on the analyzer set, that then analyzes the jar as it would be viewed for the given release (everything else will be hidden, including the versions folder) one can e.g. pass in the BND file, or e.g. like it is done by the maven-compile plugin, put together one multi-release jar (this would then require one invocation for each declared release compiled). This is of course just a first step, but that way one not needs to make everything aware of the multirelease in BND (e.g. a AnalyzerPlugin will not notice anything here).

[timothyjward]

Setting a release flag in the bnd file seems wrong. The normal model is that one bnd file == one bundle, but the release flag would modify this to be one bnd file == one manifest, with multiple bnd files contributing to a single bundle.

If we ignore the layout issues (e.g. how I get stuff into META-INF/versions/X) then from a user perspective it feels like the most usable scenario for getting bnd to generate multi-release manifests would be to set:

Multi-Release: true

in the bnd file. This has the advantage of matching the normal Java header, and being minimal effort for the "happy path" scenario.

The things to clear up would then be stuff like:

• How do I get the multi-release classes from the build path into META-INF/versions/X? This is might be a new instruction, or an attribute on exisiting instructions.
• How do I customise the imports and required capabilities for the multi-release manifests? This might require additional bnd files.

[laeubi]

The normal model is that one bnd file == one bundle

I don't think that's really "true" as e.g. BND delegates to bnd instructions of the "parent" already, so there could also be one bnd file == no bundle or even one bnd file = many bundles (if I use the same for many bundles)...

it feels like the most usable scenario for getting bnd to generate multi-release manifests would be to set

But what if I don't want a multi-release jar? I just want a single manifest but for java 11 as a target release. I think this is a conceptual misunderstanding that a user want to build some Multi-Release stuff at all. Still I want BND to discover the same things that java would discover at runtime that is https://docs.oracle.com/javase/9/docs/api/java/util/jar/JarFile.html#getJarEntry-java.lang.String-

So the very first goal (for me) would be that BND could correctly handle Jars like they would be handled in a real executed JVM run, generatinga MR-Bundle is just the second step.

• This might require additional bnd files.

Now I'm confused, didn't you just said one bnd file == one bundle ;-)

[timothyjward]

The normal model is that one bnd file == one bundle

I don't think that's really "true" as e.g. BND delegates to bnd instructions of the "parent" already, so there could also be one bnd file == no bundle or even one bnd file = many bundles (if I use the same for many bundles)...

This really isn't the case. You may be misunderstanding things if you have only ever used the bnd-maven-plugin which auto-generates you a bnd file based on the pom and other inherited configuration if no bnd file exists. It doesn't matter whether you're using Maven, Gradle, or the bnd workspace model, each project that produces a bundle does so based on a bnd file in that project.

But what if I don't want a multi-release jar? I just want a single manifest but for java 11 as a target release.

If this is what you want then bnd does not need to change because you're not using multi-release. To get this result don't have a META-INF/versions/ folder in your bundle and everything will work as it does now. To get the target release either compile using java 11, or set the execution environment appropriately.

This might require additional bnd files.

Now I'm confused, didn't you just said one bnd file == one bundle ;-)

Seriously? I'm trying to help you here. It is important not to pre-judge the outcome of the discussions about how we do this. I'm open to some pretty significant changes to make this work, including having subsidiary bnd files, but the changes need to make sense and they need to be discussed more widely.

[laeubi]

You may be misunderstanding things if you have only ever used the bnd-maven-plugin which auto-generates you a bnd file based on the pom and other inherited configuration if no bnd file exists.

The bnd-maven-plugin takes a bnd file (or I can embeds one in the XML) and combines them with the "parent" configuration ... so I assume this works similar for other cases as well, but for sure this could be maven-specific.

If this is what you want then bnd does not need to change because you're not using multi-release. To get this result don't have a META-INF/versions/ folder in your bundle and everything will work as it does now.

You mean, it does not work ... whenever I use a MR-bundle as a dependency BND will currently not read it correctly (e.g see #5327) and if the dependency contains different packages annotations on the versioned classes variants, or even contains additional classes, BND will not work correctly.
Still my bundle will never be a MR jar/bundle and I still want BND to work correctly because if I compile it with target=11 i expect it to work for java 11, weather or not I enable MR or not.

Seriously? I'm trying to help you here.

I'm just pointing out that previously you said multiple BND files "seem bad" and then you say we better should use a alternative approach that might require multiple bnd files :-)

It is important not to pre-judge the outcome of the discussions about how we do this.

At least allow using multiple BND instructions (you are not forced to use them, see previous example where my bundle itself is not MR-Jar) does not limit anything here and you seem to indicate that this is even a speific thing of the bnd-maven-plugin, so why I should it be forbidden to do so, especially when one talks about the bnd-process goal where I'm not building a bundle at all (but generating meta-data)?

I'm open to some pretty significant changes to make this work, including having subsidiary bnd files, but the changes need to make sense and they need to be discussed more widely.

All this does not introduce anything "new" and it is not forbidden right now, so I really curious why this should "not make sense", given that the --release on java is also there, so what from my side makes "no sense" is that BND do not support something similar but want to force me creating a multi-release jar if my only requirement is

1. BND should at least be able to read dependency jars in a way that is compatible with javac so I don't get different result from BND and the compiler
2. At least I'm able to generate one manifest for such a release, BND can happily put the burden on me how to actually place this on the right location (either as the default or in a versioned folder or ...)

That's why I think BND needs a -release flag anyways, beside from that it might also include support for specifying a Multi-Release flag that in addition do some magic e.g generate multiple Manifest fragments in one run, check all my classpath to get an idea what versions are all there beside these in the bundle myself (as explained it might has different outcomes!), but this is simply another use-case and I would be happy with having at least the very basic first step.

[jonatan-ivanov]

@pkriens Could you please post a reference to the commit that introduced the fix?

[pkriens]

@jonatan-ivanov the current bnd snapshots contain the handling of MRJs.

[pkriens]

@laeubi I think you are ignoring the all important fact that MRJs mandate that an MRJ must not cause public API differences when run on different VM releases. (This is the main reason why I think MRJs are such a stupid idea; they require this humongous restriction to not fall flat on your face but they have not even the tiniest mechanisms in place to enforce this immense constraint. Ergo, another sun.misc they can fix in Java 42)

The public API exported by the classes in a multi-release JAR file must be exactly the same across versions, ...

No public API changes implies that the only differences that running on different releases can cause are the runtime dependencies. Ergo, bnd should be able to read the public API for any supported version and then the base release is sufficient. This is what we do today.

The spec also says:

A future release of this specification may relax the exact same API constraint to support careful evolution.

I do not think we should try to guess what they're going to do in Java 42 ... Tends to end up badly.

[jonatan-ivanov]

@pkriens

@jonatan-ivanov the current bnd snapshots contain the handling of MRJs.

I'm not sure this answers my question. I did not ask if bnd supports it or not, I asked if you could post a reference to the commit/PR that introduced the fix.

Like this one? #5346
Are there more? Can they be linked to this issue?

[pkriens]

This were the primary commits

• Issues/5346 fixes #5583
• Java Imports & Empty header #5588
• Add an example to verify maven multi-release jar build with maven #5581 (still open since the test fails)

[paulrutter]

I know this is an old issue, but i read this thread and the felix issue tracker, but cannot find if this is or isn't fixed in the maven bundle plugin.
I did find the example that uses the bnd plugin, but is there a Multi-Release instruction for the maven bundle plugin?
@pkriens @laeubi

I ran into this when upgrading to Jersey 3.1.7, which is a multi release jar now.

[chrisrueger]

Multi-Release

At least the release notes of bnd version 7.0.0. mention "Supports Multi release JARs" @paulrutter Don't know how this behaves with other plugins you mention in your question.

[laeubi]

You need felix-plugin with latest bnd if felix has not yet updated there is quite a good chance you can simply add bnd-7 to the dependency section of the plugin. Beside that bnd-process goal can often be a replacement for felix-bundle-plugin even though it has some benefits.

[paulrutter]

Thanks, would just adding the dependency be enough or do i need additional instructions?
The latest maven bundle plugin release uses bnd 6.3.1, see https://mvnrepository.com/artifact/org.apache.felix/maven-bundle-plugin/5.1.9

[laeubi]

If you don't build a MR jar no additional instructions are required.

[paulrutter]

Thanks, i will try it out and file an issue to the felix issue tracker to get bnd upgraded in the maven bundle plugin if it works.

EDIT

Adding this to the maven-bundle-plugin section indeed solves the build issue with consuming MR jars as compile dependency of the OSGi bundle.

        <dependencies>
          <!-- Required to upgrade bnd in the maven-bundle-plugin to be able to handle Multi-Release jars -->
          <dependency>
            <groupId>biz.aQute.bnd</groupId>
            <artifactId>biz.aQute.bndlib</artifactId>
            <version>7.0.0</version>
          </dependency>        
        </dependencies>

I will file a PR for the felix project to get it upgraded, although it will be a breaking change due to the java 17 requirement.