Classloader layering strategy failing with protobuf-v3

[mdedetrich]

So #742 did happen to fix the osgi corrupted jar issue however now with the jar being successfully loaded there appears to be another unrelated bug that is due to how the classes are laid out in the jar, i.e.

[error] Test suite org.apache.pekko.http.scaladsl.server.directives.RespondWithDirectivesSpec failed with java.lang.NoClassDefFoundError: org/apache/pekko/protobufv3/internal/MessageLite.
[error] This may be due to the ClassLoaderLayeringStrategy (ScalaLibrary) used by your task.
[error] To improve performance and reduce memory, sbt attempts to cache the class loaders used to load the project dependencies.
[error] The project class files are loaded in a separate class loader that is created for each test run.
[error] The test class loader accesses the project dependency classes using the cached project dependency classloader.
[error] With this approach, class loading may fail under the following conditions:
[error] 
[error]  * Dependencies use reflection to access classes in your project's classpath.
[error]    Java serialization/deserialization may cause this.
[error]  * An open package is accessed across layers. If the project's classes access or extend
[error]    jvm package private classes defined in a project dependency, it may cause an IllegalAccessError
[error]    because the jvm enforces package private at the classloader level.
[error] 
[error] These issues, along with others that were not enumerated above, may be resolved by changing the class loader layering strategy.
[error] The Flat and ScalaLibrary strategies bundle the full project classpath in the same class loader.
[error] To use one of these strategies, set the  ClassLoaderLayeringStrategy key
[error] in your configuration, for example:
[error] 
[error] set http-tests / Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
[error] set http-tests / Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
[error] 
[error] See ClassLoaderLayeringStrategy.scala for the full list of options.
[error] Test suite org.apache.pekko.http.scaladsl.server.RouteResultSpec failed with java.lang.NoClassDefFoundError: org/apache/pekko/protobufv3/internal/MessageLite.

Which is an exert from https://github.com/apache/incubator-pekko-http/actions/runs/6555330770/job/17803594748 .

Ininitial suspicions that this may be due to sbt-assembler but more work is needed to isolate. I will backport #742 onto the 1.0.x branch to confirm the theory that its no longer related to osgi.

@pjfanning @jrudolph @raboof

[mdedetrich]

So after the merge of #746 into the 1.0.x branch, I made a +publishLocal of pekko and then ran the nightly pekko http test suite against it, i.e.

sbt -Dpekko.http.build.pekko.version=1.0.1+45-21561418-SNAPSHOT "++ 2.12.18" Test/compile
sbt -Dpekko.http.parallelExecution=false -Dpekko.test.timefactor=2 -Dpekko.http.build.pekko.version=1.0.1+45-21561418-SNAPSHOT "++ 2.12.18" mimaReportBinaryIssues test

where 1.0.1+45-21561418-SNAPSHOT is that published pekko 1.0.x snapshot and it does indeed fail with the latest update to sbt-osgi so I was incorrect before and I think this is the root cause.

I believe this may be due to the jdk9 classes issue which was referenced in comments.

[mdedetrich]

So I just did some bisecting and I can confirm that the issue is not with the latest sbt-osgi jar fix (i.e. sbt/sbt-osgi#81). To confirm this I checked out revision 958c92163b of the pekko 1.0.x branch which was pointing to sbt-osgi 0.9.4 and the older sbt-assembler (i.e. the latest version of pekko 1.0.x that worked without issues) and I created a patched version of sbt-osgi that contained sbt/sbt-osgi#81.

I then ran the pekko-http nightly tests against the locally deployed pekko with that sbt-osgi 0.9.4 patch version and the tests passed.

Due to this, I think the core reason behind the regression is that https://github.com/apache/incubator-pekko/blob/69cc4eaa3f4bdef2665406b88e831aeb12f553b6/project/plugins.sbt#L16 was never fixed, so I will try looking into that as the next step. @raboof

[pjfanning]

Would a sbt-osgi 0.9.4.1 release make sense? The only change would be backporting the recent jar tool PR.

[mdedetrich]

I think that sbt-assembly update may be the root cause, ill do a little more digging

[mdedetrich]

@pjfanning So I can def confirm that there was a regression in sbt osgi from 0.9.4 to 0.9.5. I just tested it with 0.9.5 + corrupted jar patch fix and it fails where as 0.9.4 + corrupted jar patch fix works without issues.

I will make an issue on sbt-osgi to see if we can make a release of 0.9.4 with the corrupted jar patch so that it at least unblocks pekko and hence give us some breathing room to find the reason behind the regression

[mdedetrich]

Upstream issue created at sbt/sbt-osgi#93