Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

8264449: Enable reproducible builds with SOURCE_DATE_EPOCH #446

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

@jgneff
Copy link
Member

@jgneff jgneff commented Mar 30, 2021

This pull request allows for reproducible builds of JavaFX on Linux, macOS, and Windows by defining the SOURCE_DATE_EPOCH environment variable. For example, the following commands create a reproducible build:

$ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
$ bash gradlew sdk jmods javadoc
$ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod

The three commands:

  1. set the build timestamp to the date of the latest source code change,
  2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and
  3. recreate the JMOD files with stable file modification times and ordering.

The third command won't be necessary once Gradle can build the JMOD archives or the jmod tool itself has the required support. For more information on the environment variable, see the SOURCE_DATE_EPOCH page. For more information on the command to recreate the JMOD files, see the strip-nondeterminism repository. I'd like to propose that we allow for reproducible builds in JavaFX 17 and consider making them the default in JavaFX 18.

Fixes

There are at least four sources of non-determinism in the JavaFX builds:

  1. Build timestamp

    The class com.sun.javafx.runtime.VersionInfo in the JavaFX Base module stores the time of the build. Furthermore, for builds that don't run on the Hudson continuous integration tool, the class adds the build time to the system property javafx.runtime.version.

  2. Modification times

    The JAR, JMOD, and ZIP archives store the modification time of each file.

  3. File ordering

    The JAR, JMOD, and ZIP archives store their files in the order returned by the file system. The native shared libraries also store their object files in the order returned by the file system. Most file systems, though, do not guarantee the order of a directory's file listing.

  4. Build path

    The class com.sun.javafx.css.parser.Css2Bin in the JavaFX Graphics module stores the absolute path of its .css input file in the corresponding .bss output file, which is then included in the JavaFX Controls module.

This pull request modifies the Gradle and Groovy build files to fix the first three sources of non-determinism. A later pull request can modify the Java files to fix the fourth.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issues

  • JDK-8264449: Enable reproducible builds with SOURCE_DATE_EPOCH
  • JDK-8238650: Allow to override buildDate with SOURCE_DATE_EPOCH

Reviewers

Contributors

  • Bernhard M. Wiedemann <javabmw@lsmod.de>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jfx pull/446/head:pull/446
$ git checkout pull/446

Update a local copy of the PR:
$ git checkout pull/446
$ git pull https://git.openjdk.java.net/jfx pull/446/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 446

View PR using the GUI difftool:
$ git pr show -t 446

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jfx/pull/446.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Mar 30, 2021

馃憢 Welcome back jgneff! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label Mar 30, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Mar 30, 2021

@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 30, 2021

I think there might be a Skara bug. The pre-submit builds on Linux, macOS, and Windows completed immediately. I think that's because the first of the two commits in this pull request includes the Java Bug ID from another pending pull request, because this pull request is a continuation of that one. I can squash the two commits and force-push the changes, if that would help.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 30, 2021

I think there might be a Skara bug.

No, no bug. Sorry about that. I just don't know how GitHub works. 鈽癸笍 The pre-submit tests ran two days ago when I pushed the branch to GitHub, so by the time I submitted the pull request, they had finished long ago.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 31, 2021

/reviewers 2

@openjdk
Copy link

@openjdk openjdk bot commented Mar 31, 2021

@kevinrushforth
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 31, 2021

I recommend trying this with the following gradle flags, to match the settings for production builds:

-PCONF=Release -PPROMOTED_BUILD_NUMBER=NNN -PHUDSON_BUILD_NUMBER=MMM -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true

where NNN is the promoted build number that is being built (usually taken from the repo tag) and MMM is the CI build number. You can just pick any two positive numbers for your test builds.

Note that this will build the native media libraries and the native webkit library.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 31, 2021

I recommend trying this with the following gradle flags, to match the settings for production builds:

Thanks, Kevin. Good news so far. I'm posting the Linux results while I run the macOS and Windows builds.

Linux

I ran the following commands twice, moving the build directory to build1 and then build2 to save their output:

$ bash gradlew clean
$ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \
    -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx \
    -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true \
    -PBUILD_LIBAV_STUBS=true sdk jmods javadoc

Then I changed the Hudson job number with -PHUDSON_BUILD_NUMBER=102, ran the build a third time, and moved build to build3. I also ran strip-nondeterminism as shown in the first comment of this pull request.

The first result is as hoped, and the second result is as expected:

$ diff -qr build1 build2
$ diff -qr build2 build3
Files build2/jmods/javafx.base.jmod
   and build3/jmods/javafx.base.jmod
   differ
Files build2/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class
  and build3/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class
  differ
Files build2/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java
  and build3/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java
  differ
Files build2/publications/javafx.base-linux.jar
  and build3/publications/javafx.base-linux.jar
  differ
Files build2/sdk/lib/javafx.base.jar
  and build3/sdk/lib/javafx.base.jar
  differ
Files build2/sdk/lib/src.zip
  and build3/sdk/lib/src.zip
  differ

You have to appreciate the irony of adding all this information to the build 鈥 the time, the path, even the job number 鈥 so that we can uniquely identify a build by its output. Meanwhile, if we didn't add this information, our builds could be uniquely identified by a single Git tag.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 1, 2021

There's good news and bad news. Good news first.

macOS

The two comparisons of the three builds on macOS were similar to those on Linux:

$ diff -qr build1 build2
$ diff -qr build2 build3
Files build2/jmods/javafx.base.jmod
  and build3/jmods/javafx.base.jmod
  differ
Files build2/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class
  and build3/modular-sdk/modules/javafx.base/com/sun/javafx/runtime/VersionInfo.class
  differ
Files build2/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java
  and build3/modular-sdk/modules_src/javafx.base/com/sun/javafx/runtime/VersionInfo.java
  differ
Files build2/publications/javafx.base-mac.jar
  and build3/publications/javafx.base-mac.jar
  differ
Files build2/sdk/lib/javafx.base.jar
  and build3/sdk/lib/javafx.base.jar
  differ
Files build2/sdk/lib/src.zip
  and build3/sdk/lib/src.zip
  differ

Windows

The Windows build, on the other hand, failed to produce identical copies of the media and WebKit shared libraries:

$ diff -qr build1 build2
Files build1/jmods/javafx.media.jmod
  and build2/jmods/javafx.media.jmod
  differ
Files build1/jmods/javafx.web.jmod
  and build2/jmods/javafx.web.jmod
  differ
Files build1/modular-sdk/modules_libs/javafx.media/fxplugins.dll
  and build2/modular-sdk/modules_libs/javafx.media/fxplugins.dll
  differ
Files build1/modular-sdk/modules_libs/javafx.media/glib-lite.dll
  and build2/modular-sdk/modules_libs/javafx.media/glib-lite.dll
  differ
Files build1/modular-sdk/modules_libs/javafx.media/gstreamer-lite.dll
  and build2/modular-sdk/modules_libs/javafx.media/gstreamer-lite.dll
  differ
Files build1/modular-sdk/modules_libs/javafx.media/jfxmedia.dll
  and build2/modular-sdk/modules_libs/javafx.media/jfxmedia.dll
  differ
Files build1/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll
  and build2/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll
  differ
Files build1/publications/javafx.media-win.jar
  and build2/publications/javafx.media-win.jar
  differ
Files build1/publications/javafx.web-win.jar
  and build2/publications/javafx.web-win.jar
  differ
Files build1/sdk/bin/fxplugins.dll
  and build2/sdk/bin/fxplugins.dll
  differ
Files build1/sdk/bin/glib-lite.dll
  and build2/sdk/bin/glib-lite.dll
  differ
Files build1/sdk/bin/gstreamer-lite.dll
  and build2/sdk/bin/gstreamer-lite.dll
  differ
Files build1/sdk/bin/jfxmedia.dll
  and build2/sdk/bin/jfxmedia.dll
  differ
Files build1/sdk/bin/jfxwebkit.dll
  and build2/sdk/bin/jfxwebkit.dll
  differ

I didn't bother running the third build with -PHUDSON_BUILD_NUMBER=102. I assume CMake would be the same across platforms. I'm hoping it's just a missing /experimental:deterministic somewhere for the Windows linker. I'll track it down.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 1, 2021

@kevinrushforth I found the Makefiles building the media libraries for Windows. I can fix those, but there's also libxml and libxslt that invoke the Windows linker here:

modules/javafx.web/src/main/native/Source/ThirdParty/libxml/src/win32/Makefile.msvc
modules/javafx.web/src/main/native/Source/ThirdParty/libxslt/src/win32/Makefile.msvc

They both state at the top, "There should never be a need to modify anything below this line." Are those files directly from their upstream sources? If so, what's our policy on patching them while waiting for fixes?

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

The WebKit build is complicated (to say the least). All of the WebKit components, including the additional third-party dependencies in /Source/ThirdParty, are built using cmake. I think any changes would involve passing flags to cmake in build.gradle.

@arun-Joseph might be able to comment on that.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

I should add that if changes are needed to the Makefile.msvc files to accept extra link flags to be passed in, then it would be fine to modify them.

@arun-Joseph
Copy link
Member

@arun-Joseph arun-Joseph commented Apr 1, 2021

It would be better to add the flag in libxml/CMakeLists.txt or in build.gradle via cmakeArgs.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

Ideally the latter, if feasible, so that the logic to handle SOURCE_DATE_EPOCH is more localized.

Enable reproducible builds of the native media shared libraries for
Windows when SOURCE_DATE_EPOCH is defined. The libraries are:

  fxplugins.dll
  glib-lite.dll
  gstreamer-lite.dll
  jfxmedia.dll
@bmwiedemann
Copy link

@bmwiedemann bmwiedemann commented Apr 1, 2021

IMHO, it would make sense to merge any clear improvement of the status-quo and debug+fix more in a later PR. "Perfect is the enemy of good."

Regarding Webkit, I know of
https://bugs.webkit.org/show_bug.cgi?id=174540
https://bugs.webkit.org/show_bug.cgi?id=188738
https://build.opensuse.org/request/show/667729
but your version is probably not that old.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

Given the level of effort to test this, I would recommend minimizing the number of separate fixes for this enhancement. If WebKit turns out to be too big a can of worms, then it might make sense to do everything else with this fix and file a follow-on for WebKit.

Regarding Webkit...but your version is probably not that old.

Right, we aren't nearly that old. We just updated WebKit a little over 2 months ago with recent sources (we update WebKit every six months).

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 1, 2021

IMHO, it would make sense to merge any clear improvement of the status-quo and debug+fix more in a later PR.

I agree that we will have to draw the line somewhere, but right now I'm down to just one file on one operating system out of 10,633 files in the build output! (It's just jfxwebkit.dll on Windows.) It would be a shame to give up now.

I would, though, like to postpone tests of static builds and builds for Android and iOS. My priorities for this pull request are:

  1. Linux distributions building JavaFX
  2. Developers building JavaFX from the repository source with the build defaults
  3. Oracle, Gluon, and BellSoft building JavaFX for their customers and for downloading

I would be happy to satisfy just the first group, but we might be close to getting all three.

@bmwiedemann
Copy link

@bmwiedemann bmwiedemann commented Apr 1, 2021

/contributor add Bernhard M. Wiedemann javabmw@lsmod.de

@openjdk
Copy link

@openjdk openjdk bot commented Apr 1, 2021

@bmwiedemann Only the author (@jgneff) is allowed to issue the contributor command.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

@sashamatveev Can you look at the changes to the media Makefiles?

Copy link
Member

@sashamatveev sashamatveev left a comment

Media makefiles look fine.

@mlbridge
Copy link

@mlbridge mlbridge bot commented Apr 2, 2021

Mailing list message from Eric Bresie on openjfx-dev:

Silly question, is the difference with Windows just the nature of the native support on each platform (Unix based vs Windows) and libraries used as part of that?

Eric Bresie
Ebresie at gmail.com (mailto:Ebresie at gmail.com)

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 12, 2021

Thank you, Alexander and Arun, for looking over the Makefiles and verifying the builds.

I'd like to take a week to get feedback and mull over the code changes ...

After realizing how little I knew about Gradle, I did some research last week and took four courses on Groovy, Gradle, and the Gradle Java Plugin. Although I now see a couple things I could change, I'm still happy with the commits as they are. I welcome any comments or questions.

Once #422 and this pull request are integrated, I can follow up with changes to make the all task reproducible (fixing javafx-sdk-17.zip) and to remove the build path from the .bss files (fixing number 4 in the first comment).

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 14, 2021

IEEE Software just published a good article that describes the problems solved in part by this pull request. The article is called "Reproducible Builds: Increasing the Integrity of Software Supply Chains," by Chris Lamb and Stefano Zacchiroli. It's an easy read of 10 pages, available at the links below:

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 16, 2021

Pending review by @johanvos -- I bumped the number of reviewers, so he will have a change to review before it is approved.

/reviewers 3

@openjdk
Copy link

@openjdk openjdk bot commented Apr 16, 2021

@kevinrushforth
The number of required reviews for this PR is now set to 3 (with at least 1 of role reviewers).

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 16, 2021

Note that this PR is dependent on PR #422 .

@jgneff As an alternative, you could close that PR (since all of the changes from that one are currently included here), and list both issues in this PR with Skara's /issue add ... command. It's up to you.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 17, 2021

As an alternative, you could close that PR ... and list both issues in this PR ...

I prefer to keep them separate because they have distinct attributions and descriptions. Few developers will read #422 if we merge them.

They also pose distinct questions for reviewers. The implicit question in #422 is, should we define an external environment variable or just remove the build timestamp entirely? The implicit question in this pull request is, given the environment variable, should we also use it as a flag for reproducible builds? Furthermore, which sources of non-determinism should be removed unconditionally, and which should be conditional on the flag?

I chose conservative answers to those questions in my changes, but I'm open to other suggestions.

@openjdk
Copy link

@openjdk openjdk bot commented Jun 1, 2021

@jgneff this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout allow-reproducible-builds
git fetch https://git.openjdk.java.net/jfx master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

@jgneff
Copy link
Member Author

@jgneff jgneff commented Jun 14, 2021

@kevinrushforth I'll take the opportunity provided by this merge-conflict to simplify as you suggested.

I'll close pull request #422, list both issues in this pull request, add Bernhard as a contributor, and add a commit that makes the all task reproducible. That will consolidate what would have been three pull requests into one and will leave only the build path problem for a follow-up request (fix number 4 in my first comment).

@jgneff
Copy link
Member Author

@jgneff jgneff commented Jun 14, 2021

/issue add 8238650

@jgneff
Copy link
Member Author

@jgneff jgneff commented Jun 14, 2021

/contributor add Bernhard M. Wiedemann javabmw@lsmod.de

@openjdk
Copy link

@openjdk openjdk bot commented Jun 14, 2021

@jgneff
Adding additional issue to issue list: 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH.

@openjdk openjdk bot removed the merge-conflict label Jun 14, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Jun 14, 2021

@jgneff
Contributor Bernhard M. Wiedemann <javabmw@lsmod.de> successfully added.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Jun 14, 2021

I re-tested the following build commands on Linux, macOS, and Windows.

  1. Developer - the tasks I use for my JavaFX developer builds:

    gradle sdk jmods javadoc

  2. GitHub Actions - the task used by the repository to test pull requests:

    gradle all

  3. Production - the properties and tasks for the JavaFX production builds:

    gradle -PCONF=Release -PPROMOTED_BUILD_NUMBER=12 -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc

I also ran the unit test tasks on each platform: test -x :web:test for the first two and test for the third.

In each case, the files under the build directory created by two consecutive builds from the same location are identical except for the JMOD files, as explained in the description at the top of this pull request. For now, the JMOD files and their bundled artifact javafx-jmods-17.zip can be normalized using the strip-nondeterminism command.

The table below shows the number of files created under the build directory for each type of build on the three platforms:

Build Type Linux macOS Win10
Developer 17,109 17,121 17,338
GitHub Actions 12,600 12,608 12,563
Production 10,676 10,680 10,633

Copy link
Member

@kevinrushforth kevinrushforth left a comment

I did a set of full builds on all three platforms today, and can verify that the builds are reproducible, when run on the same system and build dir, for all artifacts except the jmods, which you point out as a known issue. There is one caveat related to time zones listed below.

All but two of the changes only affect reproducible builds, meaning that they only take effect when SOURCE_DATE_EPOCH is set. The following two changes affect all builds unconditionally, even when not using SOURCE_DATE_EPOCH:

  • removal of the (obsolete) jar indexing for javafx-swt.jar
  • sorting the list of files to be linked

I think it's OK to not qualify the above with a check for SOURCE_DATE_EPOCH.

When SOURCE_DATE_EPOCH is set, I see the following problems:

  • The timestamps for all files in the zip archives are set to a hard-coded "1980-02-01", rather than the date and time specified by SOURCE_DATE_EPOCH. This is both annoying, and a potential for bugs (e.g., a program looking at the time stamps of the various files in the archive could get confused). If no good solution can be found now, I want to see a follow-up bug for this. I would be reluctant to recommend the use of SOURCE_DATE_EPOCH until this can be fixed.
  • The conversion of SOURCE_DATE_EPOCH into buildTimeStamp depends on the time zone of the build machine. This means that you won't be able to compare two builds using the same time stamp on two different builds machines with different time zones. One solution would be to do the conversion such that the buildTimeStamp was always shown in UTC. This can be done as a follow-up issue, if you like.

I'm doing a CI build and will verify the results (and probably do a bit more testing) when I get a chance, but I wanted to pass on this feedback today.

build.gradle Outdated
buildDate = new java.util.Date(ms)
}
def buildTimestamp = new java.text.SimpleDateFormat("yyyy-MM-dd-HHmmss").format(buildDate)
Copy link
Member

@kevinrushforth kevinrushforth Sep 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be useful to format buildDate using UTC as the time zone when SOURCE_DATE_EPOCH is set.

Copy link
Member Author

@jgneff jgneff Nov 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. The build now uses the new java.time.Instant class to get the instant on the time-line, whether or not SOURCE_DATE_EPOCH is set, and creates the timestamp in UTC using the ISO 8601 date and time format.

cmakeArgs = "$cmakeArgs -DCMAKE_C_FLAGS='${cFlags}' -DCMAKE_CXX_FLAGS='${cFlags}'"
cmakeArgs = "$cmakeArgs -DCMAKE_SHARED_LINKER_FLAGS='${lFlags}'"
Copy link
Member

@kevinrushforth kevinrushforth Sep 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume you've tested this both with and without SOURCE_DATE_EPOCH?

Copy link
Member Author

@jgneff jgneff Nov 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, now I have! 馃槃 Thanks.

// Reproducible archives
tasks.withType(AbstractArchiveTask) {
if (sourceDateEpoch != null) {
preserveFileTimestamps = false
Copy link
Member

@kevinrushforth kevinrushforth Sep 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a problem given how gradle generates a zip archive when this is set. How hard would it be to force all time stamps to be SOURCE_DATE_EPOCH?

Copy link
Member Author

@jgneff jgneff Nov 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was harder than I had hoped, but I think I found a good solution. The only changes in the ZIP and JAR archives are shown in the diff extract below (from zipinfo -v):

29c29
<   minimum software version required to extract:   1.0
---
>   minimum software version required to extract:   2.0
33,34c33,34
<   extended local header:                          no
<   file last modified on (DOS date/time):          1980 Feb 1 00:00:00
---
>   extended local header:                          yes
>   file last modified on (DOS date/time):          2021 Nov 16 17:55:42
44c44
<   MS-DOS file attributes (10 hex):                dir 
---
>   MS-DOS file attributes (00 hex):                none

where:

  • The minimum software version required to extract is now set to the correct value of 2.0, which is the minimum for file entries compressed with DEFLATE.
  • The extended local header is defined but with length zero.
  • The timestamp is set to the local date and time in UTC of the instant of the build.
  • The MS-DOS file attribute for a directory is not set. It does not appear to be useful, as an entry in a ZIP file is a directory if and only if its name ends with a slash ("/").

These changes are due to Gradle using the Apache Ant org.apache.tools.zip.ZipEntry class, while the build now uses the java.util.zip.ZipEntry class in OpenJDK to create the archives.

def sdkTask = task("sdk$t.capital") {
group = "Basic"
dependsOn(javafxSwtIndexTask)
dependsOn(javafxSwtTask)
Copy link
Member

@kevinrushforth kevinrushforth Sep 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in the general comments, this change affects builds even when not using SOURCE_DATE_EPOCH. I think this is fine.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Sep 18, 2021

Here are couple more observations, and then I'll need to put this on the back burner for a bit:

  1. I did a CI build yesterday and again today on all three platforms and compared the sdk between the two builds for each platform. On all three platforms the results are the same: All files were identical except the native jfxwebkit library. So there is something in the WebKit build that is affected by an external input (perhaps the system date or similar).
  2. On Mac, at least, there are several differences in the dylib files between a build on my local system and on our CI system. I was using the same devkit and boot JDK on both, and both were the same version of macOS (10.15.7). Likely some difference in the software installed on the two systems matters.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Sep 18, 2021

Thanks for the great comments, Kevin. I'm looking into the issues you raised.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Sep 18, 2021

  1. On all three platforms the results are the same: All files were identical except the native jfxwebkit library.

The diffoscope tool can show you the difference between the two files. You don't even need to install it. If the files aren't too big, you can upload them to the online version.

  1. On Mac, at least, there are several differences in the dylib files between a build on my local system and on our CI system.

I would like to enable reproducible builds on ephemeral systems that create a clean and isolated build environment, like those created by GitHub Actions or the Launchpad build farm. To compare across developer systems, we would need a full system software bill of materials (SBOM) beyond what's listed in the Gradle dependency verification file. The SBOM is the next step in allowing for reproducible builds in any environment, but it's not a part of this pull request.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Sep 18, 2021

  • The timestamps for all files in the zip archives are set to a hard-coded "1980-02-01", rather than the date and time specified by SOURCE_DATE_EPOCH.

That date was chosen by the Gradle project five years ago in the commit "Add sortedFileOrder and preserveFileTimestamps for archive tasks," and not changed five months later in the follow-up commit "Move constants to classes using them." This year, they confirmed the choice by closing the issue "allow to configure timestamp used for preserveFileTimestamps.".

Also see my comment on the closed issue for more background information.

At this point, it might be safest to go with the five-year-old Gradle default when compared to the alternatives:

  1. Invoke the find and touch command-line tools from Gradle as described on the Reproducible Builds website under the "File modification times" section of the Archive metadata page.
  2. Figure out how to do the equivalent thing in Gradle, perhaps using the eachFile(closure) or eachFile(action) method of the Zip task.
  3. Add a post-processing step outside of the Gradle build that runs strip-nondeterminism to normalize the JAR and ZIP archives and then repackages the SDK, JMOD, and Javadoc bundles.

Gradle is our build system. In for a penny, in for a pound?

@kevinrushforth kevinrushforth self-requested a review Sep 23, 2021
Create the build timestamp as a zoned date and time in UTC instead
of a local date and time. If SOURCE_DATE_EPOCH is defined, set the
last modification time of ZIP and JAR entries to the local date and
time in UTC of the instant defined by SOURCE_DATE_EPOCH.

Add a comment as a reminder to make JMOD files deterministic when
the following enhancements are complete:

* Enable deterministic file content ordering for Jar and Jmod
  https://bugs.openjdk.java.net/browse/JDK-8276764
  openjdk/jdk#6395

* Enable jar and jmod to produce deterministic timestamped content
  https://bugs.openjdk.java.net/browse/JDK-8276766
  openjdk/jdk#6481
@jgneff
Copy link
Member Author

@jgneff jgneff commented Nov 22, 2021

I did a CI build yesterday and again today on all three platforms and compared the sdk between the two builds for each platform. On all three platforms the results are the same: All files were identical except the native jfxwebkit library. So there is something in the WebKit build that is affected by an external input (perhaps the system date or similar).

I can recreate this difference, but only on macOS.

Lesson learned: When testing reproducible builds on a permanent system, never use the Gradle Daemon!

I always get a different libjfxwebkit.dylib file when I run the build with bash gradlew --no-daemon. Shown below are the versions of the compilers I'm using:

john@linux:~$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

john@macos:~$ gcc --version
Apple clang version 13.0.0 (clang-1300.0.29.3)

john@win10:~$ gcc --version
gcc (GCC) 11.2.0

When I dump each of the shared libraries with the following script, I get a text file with 10,413,716 lines, yet there are only 108 lines different between the two output files.

#!/bin/bash
# Dumps macOS shared libraries
# Diffoscope fails due to missing '=' after '--arch-name' option:
#     llvm-objdump: error: unknown argument '--arch-name'
llvm-objdump --arch-name=x86_64 --section="__TEXT,__text" --macho \
    --demangle --no-leading-addr --no-show-raw-insn "$@"

The differences are shown below:

diff --git a/libjfxwebkit.5.txt b/libjfxwebkit.6.txt
index 9d6932a..63f2b70 100644
--- a/libjfxwebkit.5.txt
+++ b/libjfxwebkit.6.txt
@@ -1,4 +1,4 @@
-libjfxwebkit.5.dylib:
+libjfxwebkit.6.dylib:
 Contents of (__TEXT,__text) section
 __ZN6WebKit15StorageAreaImplD2Ev:
 	pushq	%rbp
@@ -881412,30 +881412,22 @@ __ZN7WebCore30isCSSPropertyEnabledBySettingsENS_13CSSPropertyIDEPKNS_8SettingsE:
 	movq	%rsp, %rbp
 	movb	$1, %al
 	testq	%rsi, %rsi
-	je	0x319ee0
+	je	0x319ee7
 	leal	-258(%rdi), %ecx
 	cmpw	$37, %cx
-	ja	0x319eb7
+	ja	0x319eb2
 	movzwl	%cx, %ecx
-	leaq	79(%rip), %rdx
+	leaq	75(%rip), %rdx
 	movslq	(%rdx,%rcx,4), %rcx
 	addq	%rdx, %rcx
 	jmpq	*%rcx
 	movb	466(%rsi), %al
-	andb	$4, %al
-	shrb	$2, %al
-	popq	%rbp
-	retq
+	jmp	0x319ee2
+	cmpw	$43, %di
+	je	0x319edc
 	movzwl	%di, %ecx
 	cmpl	$366, %ecx
-	je	0x319ed5
-	cmpw	$43, %di
-	jne	0x319ee0
-	movb	451(%rsi), %al
-	andb	$4, %al
-	shrb	$2, %al
-	popq	%rbp
-	retq
+	jne	0x319ee7
 	movb	454(%rsi), %al
 	andb	$32, %al
 	shrb	$5, %al
@@ -881446,46 +881438,52 @@ __ZN7WebCore30isCSSPropertyEnabledBySettingsENS_13CSSPropertyIDEPKNS_8SettingsE:
 	shrb	%al
 	popq	%rbp
 	retq
+	movb	451(%rsi), %al
+	andb	$4, %al
+	shrb	$2, %al
+	popq	%rbp
+	retq
+	nopl	(%rax)
+	.long 4294967230	@ KIND_JUMP_TABLE32
+	.long 4294967230	@ KIND_JUMP_TABLE32
+	.long 4294967230	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967255	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967291	@ KIND_JUMP_TABLE32
+	.long 4294967255	@ KIND_JUMP_TABLE32
+	.long 4294967268	@ KIND_JUMP_TABLE32
+	nopw	%cs:(%rax,%rax)
 	nop
-	.long 4294967226	@ KIND_JUMP_TABLE32
-	.long 4294967226	@ KIND_JUMP_TABLE32
-	.long 4294967226	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967269	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967280	@ KIND_JUMP_TABLE32
-	.long 4294967269	@ KIND_JUMP_TABLE32
-	.long 4294967282	@ KIND_JUMP_TABLE32
-	nopl	(%rax,%rax)
 __ZN7WebCore15getPropertyNameENS_13CSSPropertyIDE:
 	leal	-2(%rdi), %eax
 	movzwl	%ax, %eax

@bmwiedemann Bernhard, does this look familiar? It appears to be similar to the problem you fixed in WebKit (twice).

@jgneff
Copy link
Member Author

@jgneff jgneff commented Nov 22, 2021

Status

The results of my latest round of builds are shown in the table below:

System Develop Actions Release Notes
Linux Except JMODs
macOS Except JMODs & libjfxwebkit.dylib
Windows Except JMODs

where the check mark () means that the unit tests ran successfully and that the files in the build directory were identical between two builds in the same project directory on the same system, except for the libjfxwebkit.dylib file on macOS and the JMOD archives on all systems.

The build types are listed below with their Gradle options and tasks:

Develop: bash gradlew --no-daemon sdk jmods javadoc test -x :web:test

Actions: bash gradlew --no-daemon all test -x :web:test

Release: bash gradlew --no-daemon -PCONF=Release -PPROMOTED_BUILD_NUMBER=7 -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc test

I ran each set of builds as shown by the following example:

$ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
$ bash gradlew --no-daemon cleanAll
$ bash gradlew --no-daemon all
$ mv build build1
$ bash gradlew --no-daemon cleanAll
$ bash gradlew --no-daemon all
$ mv build build2
$ bash gradlew --no-daemon all test -x :web:test

I ran two more builds on each system with SOURCE_DATE_EPOCH undefined: a reference Develop build and a Release build with its unit tests.

I also confirmed that the builds are reproducible, except for the JMOD files, on all of the following Linux architectures running Ubuntu 18.04 LTS:

Architecture Develop Notes
amd64 Except JMODs
arm64 Except JMODs
armhf Except JMODs
i386 Except JMODs
ppc64el Except JMODs
s390x Except JMODs

One bonus with reproducible builds is that you can compare across systems. For example, the following JAR files are now identical across operating systems and hardware architectures, just as one would expect:

  • javafx.base.jar
  • javafx.fxml.jar
  • javafx.swing.jar
  • javafx-swt.jar
  • javafx.web.jar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 participants