Bug: java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF in actor / osgiBundle

[He-Pin]

Error when try to execute:

Jmh/run -i 3 -wi 3 -f1 -t1 .*ZipWithIndexBenchmark.* 

error] (actor / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error] Total time: 39 s, completed 2024-1-20 1:00:26
[IJ]last actor / osgiBundle
[info] Validating all packages are set private or exported for OSGi explicitly...
[warn] bnd: Unused -privatepackage instructions , no such package(s) on the class path: [org.apache.pekko.osgi.impl]
[warn] bnd: Unused Import-Package instructions: [scala.compat.java8.*]
[error] java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error]         at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:232)
[error]         at java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109)
[error]         at sbt.io.IO$.addFileEntry$1(IO.scala:717)
[error]         at sbt.io.IO$.$anonfun$writeZip$6(IO.scala:726)
[error]         at sbt.io.IO$.$anonfun$writeZip$6$adapted(IO.scala:726)
[error]         at scala.collection.immutable.Stream.foreach(Stream.scala:533)
[error]         at sbt.io.IO$.writeZip(IO.scala:726)
[error]         at sbt.io.IO$.$anonfun$archive$2(IO.scala:676)
[error]         at sbt.io.IO$.$anonfun$archive$2$adapted(IO.scala:673)
[error]         at sbt.io.IO$.$anonfun$withZipOutput$1(IO.scala:773)
[error]         at sbt.io.IO$.$anonfun$withZipOutput$1$adapted(IO.scala:753)
[error]         at sbt.io.Using.apply(Using.scala:28)
[error]         at sbt.io.IO$.withZipOutput(IO.scala:753)
[error]         at sbt.io.IO$.archive(IO.scala:673)
[error]         at sbt.io.IO$.jar(IO.scala:621)
[error]         at com.typesafe.sbt.osgi.Osgi$.$anonfun$bundleTask$1(Osgi.scala:187)
[error]         at scala.Option.getOrElse(Option.scala:189)
[error]         at com.typesafe.sbt.osgi.Osgi$.withCache(Osgi.scala:112)
[error]         at com.typesafe.sbt.osgi.Osgi$.bundleTask(Osgi.scala:139)
[error]         at com.typesafe.sbt.osgi.SbtOsgi$.$anonfun$defaultOsgiSettings$1(SbtOsgi.scala:57)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]         at sbt.Execute.work(Execute.scala:292)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:750)
[error] (actor / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error] (slf4j / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error] (stream / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error] (testkit / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF
[error] (coordination / osgiBundle) java.util.zip.ZipException: duplicate entry: META-INF/MANIFEST.MF

[He-Pin]

@mdedetrich Would you like to take a view?

[Roiocam]

Don't know why, but won't happen on my laptop, can you provider more information?

[He-Pin]

@Roiocam to reproduce:

1) sbt shell
2) project bench-jmh
3) Jmh/run -i 3 -wi 3 -f1 -t1 .*ZipWithIndexBenchmark.* 

[Roiocam]

@Roiocam to reproduce:

1) sbt shell
2) project bench-jmh
3) Jmh/run -i 3 -wi 3 -f1 -t1 .*ZipWithIndexBenchmark.* 

Yep, that's what I did before. Those ERROR don't appear on my laptop.

[He-Pin]

I think it may need assemblyMergeStrategy or something like that. @Roiocam I'm using Windows 11.

[He-Pin]

refs: sbt/sbt-osgi#113

This blockme to run any jmh on windows :(

[mdedetrich]

This seems to be windows related, I am also not experiencing any issues.

[Roiocam]

I think it may need assemblyMergeStrategy or something like that. @Roiocam I'm using Windows 11.

I think this issue related to #920

[Roiocam]

I think it may need assemblyMergeStrategy or something like that. @Roiocam I'm using Windows 11.

I think this issue related to #920

The issue: sbt/sbt-assembly#80 may help with this issue.

[mdedetrich]

The issue: sbt/sbt-assembly#80 may help with this issue.

I have doubts that this is the core issue, its over 11 years old and our issue is unique to windows which to me indicates the problem is elsewhere.

[mdedetrich]

@He-Pin What I would recommend is changing the version in https://github.com/apache/incubator-pekko/blob/main/project/plugins.sbt#L16 until you hit a point where you can no longer replicate the issue.

[He-Pin]

addSbtPlugin("com.typesafe.sbt" % "sbt-osgi" % "0.9.4")
addSbtPlugin("com.typesafe.sbt" % "sbt-osgi" % "0.9.6")
works.

 // Recent versions of BND create corrupted jars so use JDK jar instead, see https://github.com/sbt/sbt-osgi/pull/81
      OsgiKeys.packageWithJVMJar := true)
addSbtPlugin("com.github.sbt" % "sbt-osgi" % "0.9.7")

broken.

      // Recent versions of BND create corrupted jars so use JDK jar instead, see https://github.com/sbt/sbt-osgi/pull/81
      OsgiKeys.packageWithJVMJar := false)
addSbtPlugin("com.github.sbt" % "sbt-osgi" % "0.9.7")

works.

     // Recent versions of BND create corrupted jars so use JDK jar instead, see https://github.com/sbt/sbt-osgi/pull/81
      OsgiKeys.packageWithJVMJar := false)
addSbtPlugin("com.github.sbt" % "sbt-osgi" % "0.9.10")

works.

I think the duplicate file comes from packageWithJVMJar.

[mdedetrich]

So I believe the relevant change is sbt/sbt-osgi#81, i.e. moving from BND to JVM jar tools to re-create the final osgi jar bundle.

@He-Pin Can you change https://github.com/apache/incubator-pekko/blob/main/project/plugins.sbt#L16 to 0.9.7 and https://github.com/apache/incubator-pekko/blob/main/project/OSGi.scala#L46 to false and let me know if you still get problems (or not)?

If you are not getting issues when setting packageWithJVMJar to false with sbt-osgi 0.9.7 its highly likely the logic in https://github.com/sbt/sbt-osgi/pull/81/files#diff-2530a9c505dd4cb34217eb195d5db3cab065049f033db306f485078da72edd5dR81-R93 is not properly portable on windows.

[He-Pin]

@mdedetrich I just updated my comments. the must be another META-INF/MANIFEST.MF comes from jdkjar on windows.

[mdedetrich]

@He-Pin In that case I would have a look at https://github.com/sbt/sbt-osgi/pull/81/files#diff-2530a9c505dd4cb34217eb195d5db3cab065049f033db306f485078da72edd5dR81-R93 and make a PR against sbt-osgi with the fix. It should be quite understandable what that code is doing.

I don't know how/why Windows is bringing in its own META-INF/MANIFEST.MF, the solution might be just to just exclude the copying of that extra META-INF/MANIFEST.MF (I would inspect its contents to see what is in there to get an idea of whats going on).

[He-Pin]

@mdedetrich I can do that on weekend, but why not a merge strategy? and user can then rewrite as what they want.

[mdedetrich]

@mdedetrich I can do that on weekend, but why not a merge strategy? and user can then rewrite as what they want.

Because thats a hack that is not solving the core problem which is diverging behaviour on Windows, there shouldn't be any duplicate files in the first place and the MergeStrategy is only valid for sbt-assembly which is not even relevant for the actor project (sbt-assembly is only used for protobuf-v3 so it wouldn't even solve the issue here).

I would suggest to solve it properly which is figuring out the difference of Windows vs Linux/MacOS and making sure it behaves the same. In the meantime you can set packageWithJVMJar := false temporarily on your local Windows machine while working on Pekko.

[He-Pin]

SBT currently can't download the sbt plugin sources, so I can't debug, or you can publish a new sbt-osgi which can print the detail of the conflict files.

I don't think that's a hack, it's a fix for Windows users/contributors.

[mdedetrich]

SBT currently can't download the sbt plugin sources, so I can't debug, or you can publish a new sbt-osgi which can print the detail of the conflict files.

Everyone has this issue. The way that sbt-plugins are developed/debugged is by using manual log/println statements (see https://www.scala-sbt.org/1.x/docs/Howto-Logging.html#Log+messages+in+a+task for how to log in a setting/task) and using publishLocal to deploy the sbt plugin locally and then setting the version in https://github.com/apache/incubator-pekko/blob/main/project/OSGi.scala#L46 to see whats happening.

Alternately you can write a sbt-scripted test for packageWithJVMJar := true, one does not currently exist and since sbt-osgi has windows as part of its CI its likely it will pick it up.

I don't think that's a hack, it's a fix for Windows users/contributors.

MergeStrategy is a feature of sbt-assembly. sbt-assembly is not used in any pekko module aside from protobufV3 and the stacktrace you provided at #986 (comment) is for the root actor module and not protobufV3. This is not going to solve the problem, MergeStrategy is not applicable here at all.

I would help but I don't have a windows machine on hand. The best I can do for now is make a simple sbt-scripted test for packageWithJVMJar := true to try and easily replicate it.

[He-Pin]

A merge strategy with conflict file can let me choose which one to keep

[mdedetrich]

A merge strategy with conflict file can let me choose which one to keep

MergeStrategy is part of sbt-assembly (its not a general option), sbt-assembly is not used in pekko actor so its not a solution. As you can see the only place where sbt-assembly is used is here which is protobufV3.

If you have duplicate files that are being packaged that is a critical issue with how you are packaging a jar. The reason why MergeStrategy is unique to sbt-assembly is because sbt-assemblies whole goal is to merge multiple jar's into a single jar.

[mdedetrich]

@He-Pin Found the core problem and I have a PR for the solution (see sbt/sbt-osgi#115). Just needs to get reviewed and then I can merge + deploy the fix.