New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Gradle's reproducible archives when building fat jars and wars #8391

Closed
seanjreilly opened this Issue Feb 23, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@seanjreilly

seanjreilly commented Feb 23, 2017

An important feature of Gradle 3.4 is its initial support for reproducible archives.

(In short, reproducible builds mean that a given revision of source code should always build the exact same binary, byte-for-byte. See https://reproducible-builds.org/ for more info).

With the right settings as described in gradle documentation, building the same basic java project twice results in two jars with the same SHA-256 hash. However, when I tried the same thing with a project that uses the spring-boot plugin, I couldn't get it to work.

I've created a github repo with a minimal reproducible example of the issue.

Steps to reproduce the issue.

(run all commands from the root project directory of the cloned repo)

  1. Clean and build the java-plugin project twice to generate two jar files. (The clean guarantees that a new jar will be built.)
    1. ./gradlew clean java-plugin:build
    2. cp java-plugin/build/libs/java-plugin-0.0.1-SNAPSHOT.jar /tmp/java-plugin-1.jar
    3. ./gradlew clean java-plugin:build
    4. cp java-plugin/build/libs/java-plugin-0.0.1-SNAPSHOT.jar /tmp/java-plugin-2.jar
  2. The two jars should be the same.
    1. shasum -a 256 /tmp/java-plugin-*.jar
    2. (in fact the hash for each file should be e1358db781e755047d9a6c504481d078f15bf259df4b81c23c7260b5ecd4e6ef. If it's not, I'd be interested in hearing about that.)
  3. Clean and build the spring-boot-plugin project twice to generate two uber jar files.
    1. ./gradlew clean spring-boot-plugin:build
    2. cp spring-boot-plugin/build/libs/spring-boot-plugin-0.0.1-SNAPSHOT.jar /tmp/spring-boot-plugin-1.jar
    3. ./gradlew clean spring-boot-plugin:build
    4. cp spring-boot-plugin/build/libs/spring-boot-plugin-0.0.1-SNAPSHOT.jar /tmp/spring-boot-plugin-2.jar
  4. These two jars should be the same, but are not.
    1. shasum -a 256 /tmp/spring-boot-plugin-*.jar

I checked out the differences with diffoscope, and the differences I could see were related to file timestamps in the uber jar.

@seanjreilly

This comment has been minimized.

seanjreilly commented Feb 23, 2017

A related question: SpringBootPlugin doesn't extend AbstractArchiveTask. Should it? Would doing so make supporting new features like this easier? (I honestly don't know the answer to that question at all — just throwing it out there)

@wilkinsona

This comment has been minimized.

Member

wilkinsona commented Feb 23, 2017

Thank you for the detailed problem report and the sample.

SpringBootPlugin doesn't extend AbstractArchiveTask. Should it? Would doing so make supporting new features like this easier?

It's RepackageTask that produces the fat jar, rather than SpringBootPlugin. RepackageTask isn't an AbstractArchiveTask which is the fundamental problem here. We'll address that as part of #5861 so I believe this should just work (famous last words) once we've done that. I'll keep this issue open as a reminder to ensure that it does.

@wilkinsona wilkinsona changed the title from The spring-boot-gradle plugin doesn't make reproducible archives like the java-plugin to Support Gradle's reproducible archives when building fat jars and wars Feb 23, 2017

@wilkinsona wilkinsona added this to the 2.0.0.M1 milestone Feb 23, 2017

wilkinsona added a commit to wilkinsona/spring-boot that referenced this issue Apr 3, 2017

@wilkinsona wilkinsona closed this in d015714 Apr 4, 2017

@seanjreilly

This comment has been minimized.

seanjreilly commented Apr 4, 2017

Oh.... This is exciting! Can't wait to check this out!

@seanjreilly

This comment has been minimized.

seanjreilly commented Dec 29, 2017

Unfortunately, the example doesn't create a reproducible archive for me with spring boot 2.0.0.M7.

When I build with the gradle recommended options for reproducible builds

        preserveFileTimestamps = false
        reproducibleFileOrder = true

The build is still producing jar files with different sha checksums.
According to the diffoscope output, some files and directories inside the zip have different timestamps.

I've updated the repo I created before, but I've stuck gradle 2.0.0.M7 on a [different branch] (https://github.com/seanjreilly/spring-boot-gradle-plugin-issue/tree/2.0.0.M7) to make it easier to switch back and forth between 1.0 and 2.0.

Is there any way this issue can be reopened?

@wilkinsona

This comment has been minimized.

Member

wilkinsona commented Jan 2, 2018

Thanks for the updated sample. It works fine for me with Gradle 4.0. With 4.1 and later, it's failing because the CONSTANT_TIME_FOR_ZIP_ENTRIES constant has been removed from GUtil. This tells me two things:

  1. We need to change to our own constant.
  2. The integration tests (which run against versions of Gradle where this should have failed) don't cover creating a reproducible archive.

I've opened #11468.

@seanjreilly

This comment has been minimized.

seanjreilly commented Jan 3, 2018

Thanks @wilkinsona. I'm sorry to say that I didn't test with any versions of Gradle other than the current (I probably should have done so before asking to reopen — I'll try to do better on that in the future).

We need to change to our own constant.

I agree.

I don't even see any particular value in having the same timestamps as a non-boot archive, as long as every spring boot archive has the same timestamps between runs.

The integration tests [...] don't cover creating a reproducible archive.

If the sample I've produced can help with that in any way I'll gladly donate it. I'm happy to transfer ownership, sign an agreement, move to a specific license, etc.

I appreciate the thorough attention to such a specific edge case.

@wilkinsona

This comment has been minimized.

Member

wilkinsona commented Jan 3, 2018

There's really no need to be better. The sample you provided was already very helpful. Thanks for the offer to donate the sample, but I've already taken care of fixing (I hope) and testing this in b545330.

@seanjreilly

This comment has been minimized.

seanjreilly commented Jan 3, 2018

The commit looks good to me, and the integration tests are the integration tests I would have written. I'm confident that this will fix the issue, and keep it fixed. Thanks so much!

@jzampieron

This comment has been minimized.

jzampieron commented Feb 18, 2018

@wilkinsona This is still an issue in 2.0.0.RC1 if you are using:

springBoot {
   buildInfo()
}

Because the plug-in includes the build.time property and there isn't an obvious way to turn that off.

This is in the build/resources/main/META-INF/build-info.properties file.

@wilkinsona

This comment has been minimized.

Member

wilkinsona commented Feb 18, 2018

@jzampieron That’s really a separate problem. Please open a new issue and we’ll see what we can do

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment