Support Gradle's reproducible archives when building fat jars and wars

[seanjreilly]

An important feature of Gradle 3.4 is its initial support for reproducible archives.

(In short, reproducible builds mean that a given revision of source code should always build the exact same binary, byte-for-byte. See https://reproducible-builds.org/ for more info).

With the right settings as described in gradle documentation, building the same basic java project twice results in two jars with the same SHA-256 hash. However, when I tried the same thing with a project that uses the spring-boot plugin, I couldn't get it to work.

I've created a github repo with a minimal reproducible example of the issue.

Steps to reproduce the issue.

(run all commands from the root project directory of the cloned repo)

1. Clean and build the java-plugin project twice to generate two jar files. (The clean guarantees that a new jar will be built.)
   1. ./gradlew clean java-plugin:build
   2. cp java-plugin/build/libs/java-plugin-0.0.1-SNAPSHOT.jar /tmp/java-plugin-1.jar
   3. ./gradlew clean java-plugin:build
   4. cp java-plugin/build/libs/java-plugin-0.0.1-SNAPSHOT.jar /tmp/java-plugin-2.jar
2. The two jars should be the same.
   1. shasum -a 256 /tmp/java-plugin-*.jar
   2. (in fact the hash for each file should be e1358db781e755047d9a6c504481d078f15bf259df4b81c23c7260b5ecd4e6ef. If it's not, I'd be interested in hearing about that.)
3. Clean and build the spring-boot-plugin project twice to generate two uber jar files.
   1. ./gradlew clean spring-boot-plugin:build
   2. cp spring-boot-plugin/build/libs/spring-boot-plugin-0.0.1-SNAPSHOT.jar /tmp/spring-boot-plugin-1.jar
   3. ./gradlew clean spring-boot-plugin:build
   4. cp spring-boot-plugin/build/libs/spring-boot-plugin-0.0.1-SNAPSHOT.jar /tmp/spring-boot-plugin-2.jar
4. These two jars should be the same, but are not.
   1. shasum -a 256 /tmp/spring-boot-plugin-*.jar

I checked out the differences with diffoscope, and the differences I could see were related to file timestamps in the uber jar.

[seanjreilly]

A related question: SpringBootPlugin doesn't extend AbstractArchiveTask. Should it? Would doing so make supporting new features like this easier? (I honestly don't know the answer to that question at all — just throwing it out there)

[wilkinsona]

Thank you for the detailed problem report and the sample.

SpringBootPlugin doesn't extend AbstractArchiveTask. Should it? Would doing so make supporting new features like this easier?

It's RepackageTask that produces the fat jar, rather than SpringBootPlugin. RepackageTask isn't an AbstractArchiveTask which is the fundamental problem here. We'll address that as part of #5861 so I believe this should just work (famous last words) once we've done that. I'll keep this issue open as a reminder to ensure that it does.

[seanjreilly]

Oh.... This is exciting! Can't wait to check this out!

[seanjreilly]

Unfortunately, the example doesn't create a reproducible archive for me with spring boot 2.0.0.M7.

When I build with the gradle recommended options for reproducible builds

        preserveFileTimestamps = false
        reproducibleFileOrder = true

The build is still producing jar files with different sha checksums.
According to the diffoscope output, some files and directories inside the zip have different timestamps.

I've updated the repo I created before, but I've stuck gradle 2.0.0.M7 on a [different branch] (https://github.com/seanjreilly/spring-boot-gradle-plugin-issue/tree/2.0.0.M7) to make it easier to switch back and forth between 1.0 and 2.0.

Is there any way this issue can be reopened?

[wilkinsona]

Thanks for the updated sample. It works fine for me with Gradle 4.0. With 4.1 and later, it's failing because the CONSTANT_TIME_FOR_ZIP_ENTRIES constant has been removed from GUtil. This tells me two things:

1. We need to change to our own constant.
2. The integration tests (which run against versions of Gradle where this should have failed) don't cover creating a reproducible archive.

I've opened #11468.

[seanjreilly]

Thanks @wilkinsona. I'm sorry to say that I didn't test with any versions of Gradle other than the current (I probably should have done so before asking to reopen — I'll try to do better on that in the future).

We need to change to our own constant.

I agree.

I don't even see any particular value in having the same timestamps as a non-boot archive, as long as every spring boot archive has the same timestamps between runs.

The integration tests [...] don't cover creating a reproducible archive.

If the sample I've produced can help with that in any way I'll gladly donate it. I'm happy to transfer ownership, sign an agreement, move to a specific license, etc.

I appreciate the thorough attention to such a specific edge case.

[wilkinsona]

There's really no need to be better. The sample you provided was already very helpful. Thanks for the offer to donate the sample, but I've already taken care of fixing (I hope) and testing this in b545330.

[seanjreilly]

The commit looks good to me, and the integration tests are the integration tests I would have written. I'm confident that this will fix the issue, and keep it fixed. Thanks so much!

[jzampieron]

@wilkinsona This is still an issue in 2.0.0.RC1 if you are using:

springBoot {
   buildInfo()
}

Because the plug-in includes the build.time property and there isn't an obvious way to turn that off.

This is in the build/resources/main/META-INF/build-info.properties file.

[wilkinsona]

@jzampieron That’s really a separate problem. Please open a new issue and we’ll see what we can do