Support zip64 format executable archives

[pmdusso]

I first wrote this in the Gradle forum, and they recommended me transfer it to here.

I wasn't able to execute the war generated from bootRepackage in a project derived from jHipster framework. The topic still opened in Stackoverflow.

I created a small application and put it on Github to help you help me with this. Following these steps I believe you can reproduce the problem also. I'm first packing a simple Spring application and then running the generated war. This will create 66000 files in the resource folder. Packing the same app again will fail because in the second time there is more than 65535 files. We confirmed this by cleaning the resource folder and bootRepackaging it again successfully.

Then I set the zip64 property to true (I had to do it in the jar { } task also. In my original application however I do not have a jar task and set only in the war task). The bootRepackage task completes with success, but then when I run the war I get the no Start-Class error.

Running this for the first time will work, because total files < 65535 and zip64 = false

./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

Running this for the second time will not work, because total files > 65535 and zip64 = false

./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

Running this for the third time (equal to alternative one) will work, because total files < 65535 and zip64 = false

rm src/main/resources/*; ./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

At this point we have total files > 65535 and zip64 = false. We change zip64 property to true and try to bootPackage it again

Set zip64 = true in build.gradle in jar and war tasks.
./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

java.lang.IllegalStateException: No 'Start-Class' manifest entry specified in jar:file:../boot-repackage-test/build/libs/app-0.1-SNAPSHOT.war!/
        at org.springframework.boot.loader.archive.Archive.getMainClass(Archive.java:57)
        at org.springframework.boot.loader.ExecutableArchiveLauncher.getMainClass(ExecutableArchiveLauncher.java:69)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:61)
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:61)

To conclude, unzip app-0.1-SNAPSHOT.war && cat META-INF/MANIFEST.MF yields:

Manifest-Version: 1.0
Start-Class: org.Application
Spring-Boot-Version: 1.2.3.RELEASE
Main-Class: org.springframework.boot.loader.WarLauncher

In gradle forum they pointed the following class: spring-projects/spring-boot/blob/master/spring-boot-tools/spring-boot-loader/src/main/java/org/springframework/boot/loader/jar/CentralDirectoryEndRecord.java

Best regards,
Pedro Dusso

[wilkinsona]

This is a known limitation; Spring Boot's Loader doesn't support the zip64 format. Given the support for nested jars, why do you need more than 65535 files in your executable archive?

[pmdusso]

I don't know about the nested jars. Can you give point any documentation? I would like to try that. May scenario is a Spring web application with java in back and angular in the front. I guess the excess of files comes from the libraries installed to the frontend.

[wilkinsona]

Nested jars are documented here.

Even with a web application, I'm surprised at the need for more than 65535 files in your archive. Could you use something like minify to reduce the number of files (and speed things up a bit for your users)?

[tenstriker]

Hi @wilkinsona since it's a limitation of spring-boot classloader It wouldn't be fixed if I use latest jre 7 or jre 8. Correct me if I'm wrong there.
Interestingly it works when I run it from eclipse using same JRE! My eclipse project has maven dependencies on classpath. I assume eclipse internally uses java -cp option instead of java -jar option. SO is there a way from command-line to run spring-boot jar using java -cp option. Note: as I mentioned in case #5086 I use PropertyLauncher. May be I can try Jar launcher?

[wilkinsona]

since it's a limitation of spring-boot classloader

Just to avoid any confusion, tt's not a limitation of Spring Boot's ClassLoader. It's a limitation of Spring Boot's Loader which is used by the various launchers to read zip archives.

It wouldn't be fixed if I use latest jre 7 or jre 8. Correct me if I'm wrong there.

Correct. Changing the version of the JRE will make no difference.

Interestingly it works when I run it from eclipse using same JRE!

When you run your application's main method in Eclipse, none of Spring Boot's launchers is used so you avoid the problem.

is there a way from command-line to run spring-boot jar using java -cp option. Note: as I mentioned in case #5086 I use PropertyLauncher. May be I can try Jar launcher?

Using JarLauncher won't make any difference as all of the launchers use the same code for reading zip archives. You could work around the problem by not repackaging your application into an executable archive, building the classpath yourself, and calling your application's main method directly.

[tenstriker]

To circumvent this issue, I manually extracted all the zip64 jars and pointed that directory to spring-boot loader. However, this is a dirty solution. I think it's really worthwhile for spring-boot loader to support zip64 files. Java 1.7 and above by default uses zip64 based on file size. There is no way to convert third-party zip64 to regular zip file and it's also seems counter intuitive.

[philwebb]

@tenstriker Unfortunately supporting zip64 is quite problematic for us since it makes it much harder to detect the embedded script that we put at the front of the file. What build tool are you using? zip64 should really only be needed if there are a large number of files in the archive (the size of the files should not matter).

[tenstriker]

we use maven scala and java compiler and spring-boot-maven-plugin plugins. As I mentioned in earlier comment zip64 jar is provided by third-party and we don't have control over it. We just have to load it in our spring boot app during runtime. Probably that zip file does has large number of files.