Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8264010: Add Gradle dependency verification #437

Closed
wants to merge 9 commits into from

Conversation

@jgneff
Copy link
Member

@jgneff jgneff commented Mar 23, 2021

This pull request adds dependency verification to the Gradle builds of JavaFX on Linux, macOS, and Windows. It is the third of three changes that close the gaps in the JavaFX build security:

"Without dependency verification it's easy for an attacker to compromise your supply chain," warns the Gradle User Guide. All three changes come from conference talks by members of the Gradle team, available as PDF slides or on YouTube in the following two videos:

"We all run in a crazy-unsafe environment, in a way," says Louis Jacomet at the end of his talk. These three changes make it just a little less crazy-unsafe for all of us building JavaFX, regardless of our system, network, or country.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

Reviewers

Contributors

  • Kevin Rushforth <kcr@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jfx pull/437/head:pull/437
$ git checkout pull/437

Update a local copy of the PR:
$ git checkout pull/437
$ git pull https://git.openjdk.java.net/jfx pull/437/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 437

View PR using the GUI difftool:
$ git pr show -t 437

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jfx/pull/437.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Mar 23, 2021

👋 Welcome back jgneff! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label Mar 23, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Mar 23, 2021

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 23, 2021

/reviewers 2

@openjdk
Copy link

@openjdk openjdk bot commented Mar 23, 2021

@kevinrushforth
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 23, 2021

This seems like a good idea to do. I have a couple overall questions before reviewing / testing.

  1. Can you add some sort of README file that describes the how to update the checksums? Also, the instructions in UPDATING-lucene.txt should be updated accordingly.
  2. Some of the files listed are not used directly. I presume that you added them because they are used indirectly by other components? Are all of them actually needed?
@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 24, 2021

Thanks, Kevin. I added a README file and updated the Lucene instructions, as you suggested. I'm open to any other suggestions on the wording or formatting, no matter how minor.

Some of the files listed are not used directly. I presume that you added them because they are used indirectly by other components? Are all of them actually needed?

The Gradle command, now documented in the gradle/README.txt file, adds entries to the dependency verification file for all dependencies, including transitive ones. I think that's the list of everything downloaded during the builds on Linux, macOS, and Windows. I'll clear the Gradle cache and double-check it now. I'll let you know if I find anything unexpected.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 24, 2021

Thanks for providing / updating the instructions.

My internal test build failed right off the bat, since we have a supplemental closed gradle file that augments the build and downloads additional build tools for our internal CI machines.

I don't yet know to handle this, since there is a single, global validation.xml file and no way that I know of to supplement this. This validation file must contain all artifacts that gradle downloads (and their transitive dependencies). From the gradle docs:

A dependency verification configuration is global: a single file is used to configure verification of the whole build. In particular, the same file is used for both the (sub)projects and buildSrc.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 24, 2021

I don't yet know to handle this ...

Would any of the following options work?

  1. If you're using your own supplemental closed Gradle build file, create your own supplemental closed Gradle verification file, too. Before the internal build, replace the current file with your own.
  2. Remove the verification file before running your internal build. In this case, though, you'll also lose its protection against software supply-chain attacks.
  3. Add your internal dependency checksum entries to the public verification file and publish the updated file in the repository.

I think the protection from the verification file is worth having as a default in the public repository. Gluon, Oracle, BellSoft, and anyone else building JavaFX can decide, based on their own security assessment, whether or not they want to use it. The point of including the file in the repository is to make that decision explicit.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 24, 2021

The three options you listed are roughly what I had come up with as well.

Option 1 would otherwise be ideal, but it would violate the best practice of not writing to an SCM-managed file during the build. If gradle were to provide a way to specify an alternate location -- even if it were a single global file -- then that's what we'd do. I'll check gradle 7 and see if they've added something like this, since we will want to update at some point in the not-too-distant future (so we can support JDK 16 as a boot JDK).

Option 2 could be easily achieved by setting system property org.gradle.dependency.verification to lenient|off. We never download anything from outside our firewall during our CI builds -- we only download pre-verified binaries from our local server. Still, this isn't the ideal situation. What I'd really like is a mode that would verify everything that is in the validation.xml file, and warn (but not fail) if an artifact is missing. It would be better than turning it off entirely, but not as good as failing if there are missing dependencies.

Option 3 is what I was leaning towards. I'll take a look (not this week, though) at generating the checksums for the internal files and see what it looks like. It's really only cmake (for WebKit build), Ninja (for WebKit build on Windows), and the compiler toolchain "devkit". Unless I'm forgetting something.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Mar 25, 2021

Are all of them actually needed?

Just to follow up on that question, all of them are in fact downloaded during the build, at least. I removed the Gradle directory $HOME/.gradle and ran the build as follows. Then I compared the list of downloaded artifacts with the ones listed in the dependency verification file.

$ rm -r $HOME/.gradle
$ bash gradlew sdk jmods javadoc apps test -x :web:test
  ...
$ find ~/.gradle/caches/modules-2 \( -name "*.jar" -o -name "*.pom" \) \
  -exec basename {} \; | sort > downloaded.log
$ grep '<artifact' gradle/verification-metadata.xml | sed 's/.*name="\(.*\)">/\1/' \
  | sort > verified.log
$ diff downloaded.log verified.log 
31a32
> org.eclipse.swt.cocoa.macosx.x86_64_3.105.3.v20170228-0512-.jar
32a34
> org.eclipse.swt.win32.win32.x86_64_3.105.3.v20170228-0512-.jar

A total of 36 artifacts (14 JAR files and 22 POM files) are downloaded during the build. The SWT libraries for macOS and Windows were not downloaded because I ran the build on Linux.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 31, 2021

We have a few in-flight or imminent updates that will impact this PR. There is a tight deadline on one of them (an ICU data file dependency), so I'd prefer to wait on integrating this until after they are done.

It's still worth continuing the review in the mean time. I noticed that the libav bundles are missing on Linux. To ensure that you aren't missing any dependencies, can you add the following gradle flags to your build?

-PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true

This will build the native media libraries, including the libav stubs (the latter is Linux only). Eventually, you will need to include WebKit, but that's not needed for now.

@kevinrushforth kevinrushforth self-requested a review Apr 1, 2021
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

When you build media with libav stubs on Linux, you should see the following 5 new entries:

ffmpeg-3.3.3-.tar.gz
ffmpeg-4.0.2-.tar.gz
libav-11.4-.tar.gz
libav-12.1-.tar.gz
libav-9.14-.tar.gz

And I was right about the additional internal tools that I would need to add. Here is the list:

cmake-3.13.3-Darwin-x86_64.tar.gz
cmake-3.13.3-Linux-x86_64.tar.gz
cmake-3.13.3-win32-x86.zip
devkit-linux_x64-gcc10.2.0-OL6.4+1.0.tar.gz
devkit-macosx_x64-Xcode11.3.1-MacOSX10.15+1.0.tar.gz
devkit-windows_x64-VS2019-16.7.2+1.0.tar.gz
jfx-devkit-gcc-patch+1.1.tar.gz
ninja-win.zip

When you build media with libav stubs on Linux, you should see the following 5 new entries:

ffmpeg-3.3.3-.tar.gz
ffmpeg-4.0.2-.tar.gz
libav-11.4-.tar.gz
libav-12.1-.tar.gz
libav-9.14-.tar.gz

And I was right about the additional tools that I would need to add (as a bonus I found an unused tool that I will eliminate). Here is the list:

cmake-3.13.3-Darwin-x86_64.tar.gz
cmake-3.13.3-Linux-x86_64.tar.gz
cmake-3.13.3-win32-x86.zip
devkit-linux_x64-gcc10.2.0-OL6.4+1.0.tar.gz
devkit-macosx_x64-Xcode11.3.1-MacOSX10.15+1.0.tar.gz
devkit-windows_x64-VS2019-16.7.2+1.0.tar.gz
jfx-devkit-gcc-patch+1.1.tar.gz
ninja-win.zip

And here is one that will show up after PR #450 is integrated:

icudt-64l.zip

And here is one that will show up after PR #450 is integrated:

icudt-64l.zip
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 1, 2021

When you build media with libav stubs on Linux, you should see the following 5 new entries:

ffmpeg-3.3.3-.tar.gz
ffmpeg-4.0.2-.tar.gz
libav-11.4-.tar.gz
libav-12.1-.tar.gz
libav-9.14-.tar.gz

I'll let you add them.

And I was right about the additional internal tools that I would need to add. Here is the list:

cmake-3.13.3-Darwin-x86_64.tar.gz
cmake-3.13.3-Linux-x86_64.tar.gz
cmake-3.13.3-win32-x86.zip
devkit-linux_x64-gcc10.2.0-OL6.4+1.0.tar.gz
devkit-macosx_x64-Xcode11.3.1-MacOSX10.15+1.0.tar.gz
devkit-windows_x64-VS2019-16.7.2+1.0.tar.gz
jfx-devkit-gcc-patch+1.1.tar.gz
ninja-win.zip

I'll provide the sha256 sums once I've tested this (probably next week). Speaking of which, we will do a compiler update shortly after JDK 17 does theirs (should be pretty soon for Mac and Windows anyway), which will update the devkits.

(as a bonus I found an unused tool that I will eliminate)

And here is one more for WebKit that will show up after PR #450 is integrated:

icudt-64l.zip

This, and then its successor when we update ICU, are the main reason I want to hold off on this for 2-3 more weeks.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 12, 2021

Thanks for the list, Kevin. Pull request #450 was closed in favor of #456, which was integrated. I'll build with the media and WebKit libraries so Gradle can add their dependencies, and I'll update this pull request with the new verification file.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 12, 2021

Yes, there are two updates:

  1. As you noted, PR #450 was withdrawn in favor of PR #456, and the latter is now integrated. As a result, there will be no icudt-64l.zip file, but you will see a new download artifact, icu4c-68.2-data-bin-l.zip once you merge the lastest master into your branch and do a build with WebKit.

  2. With the integration of PR #460 this morning, there is a new devkit for Xcode 12.4. Here is the updated list of internal artifacts:

cmake-3.13.3-Darwin-x86_64.tar.gz
cmake-3.13.3-Linux-x86_64.tar.gz
cmake-3.13.3-win32-x86.zip
devkit-linux_x64-gcc10.2.0-OL6.4+1.0.tar.gz
devkit-macosx_x64-Xcode11.3.1-MacOSX10.15+1.0.tar.gz
devkit-macosx-Xcode12.4+1.0.tar.gz
devkit-windows_x64-VS2019-16.7.2+1.0.tar.gz
jfx-devkit-gcc-patch+1.1.tar.gz
ninja-win.zip

Since this should be settled down for now, I'll send you the checksums some time later this week (presuming you have added the media and WebKit artifacts by then).

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 14, 2021

There's an odd thing happening with some of the artifact names in the dependency file. For example:

<component group="" name="ffmpeg-3.3.3" version="">
    <artifact name="ffmpeg-3.3.3-.tar.gz">
        <sha256 value="6600...bf3c" origin="Generated by Gradle"/>
    </artifact>
</component>

Gradle creates the artifact name from the component name and version, separated by a hyphen. Because the version attribute is empty, we get an artifact name ending in -.tar.gz. It should be ffmpeg-3.3.3.tar.gz, but Gradle sees -3.3.3 as part of the name. This particular name originates from:

dependencies {
    if (IS_BUILD_LIBAV_STUBS) {
        media name: "libav-9.14", ext: "tar.gz"
        media name: "libav-11.4", ext: "tar.gz"
        media name: "libav-12.1", ext: "tar.gz"
        media name: "ffmpeg-3.3.3", ext: "tar.gz"
        media name: "ffmpeg-4.0.2", ext: "tar.gz"
    }
    compile project(":base")
    compile project(":graphics")
}

Other places in the build file seem to depend on having the version as part of the name string, and my initial attempt to separate the two failed. The build works with the names as they are, and these are in fact the names of the files in the Gradle cache. Some day, though, we might want to have better component elements in the dependency file, seeing as it functions also like a software bill of materials.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 17, 2021

Yeah, I noticed that about the names, too. Not sure it's worth worrying about, although it is a little odd.

Btw, I have the final list of internal downloads that I'll pass on to you, so you can add them to the PR. I've done a CI build with this version of verification.xml and it passes.

You can see the diffs here: kevinrushforth/jfx@ce1aefa

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 17, 2021

Thanks, Kevin. I added your list to the file and ran just the Linux build followed by tests, all successful:

$ gradle -PCONF=Release -PPROMOTED_BUILD_NUMBER=7 \
  -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx \
  -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true \
  sdk jmods javadoc test
  ...
BUILD SUCCESSFUL in 4m 26s
224 actionable tasks: 93 executed, 131 up-to-date
@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 18, 2021

/contributor add kcr

@openjdk
Copy link

@openjdk openjdk bot commented Apr 18, 2021

@jgneff
Contributor Kevin Rushforth <kcr@openjdk.org> successfully added.

Copy link
Member

@kevinrushforth kevinrushforth left a comment

Looks good, with one comment on the new README.txt file.

When upgrading an external dependency to a newer version, update the
dependency verification file as follows:

$ gradle --write-verification-metadata sha256 help

This comment has been minimized.

@kevinrushforth

kevinrushforth Apr 19, 2021
Member

This isn't sufficient for many of the dependencies. Gradle won't try to download external dependencies until the point they are used. For example: the junit dependency is downloaded only when running gradle test, the icu data dependency is downloaded only when building the sdk with -PCOMPILE_WEBKIT=true, the libav media libraries (for Linux) are downloaded only when building the sdk with -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true, etc.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 27, 2021

@jgneff Except for the README, I think this is ready. I'm about to send PRs for compiler updates on Windows and Linux, so if they go in first (which seems likely), I'll need to send you an additional entry for each.

@johanvos can you also review this?

By way of general comment, here is another article highlighting the importance of doing this:
https://wwws.nightwatchcybersecurity.com/2021/04/25/supply-chain-attacks-via-github-com-releases/

@kevinrushforth kevinrushforth requested a review from johanvos Apr 27, 2021
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 27, 2021

Here are the two additional internal downloads for the new versions of the gcc-10.3 and vs2019-16.9.3 download bundles; can add them to the PR? I've done a CI build with this version of verification.xml and it passes.

You can see the diffs here: kevinrushforth/jfx@c0db736

Thanks.

@jgneff
Copy link
Member Author

@jgneff jgneff commented Apr 29, 2021

Here are the two additional internal downloads for the new versions of the gcc-10.3 and vs2019-16.9.3 download

Thanks, Kevin. I'll assume those replace the older versions shown in the excerpt below, and I'll remove them. Let me know otherwise.

<component group="javafx" name="devkit-linux_x64-gcc10.2.0" version="OL6.4+1.0.tar">
<component group="javafx" name="devkit-linux_x64-gcc10.3.0" version="OL6.4+1.0.tar">

<component group="javafx" name="devkit-windows_x64-VS2019" version="16.7.2+1.0.tar">
<component group="javafx" name="devkit-windows_x64-VS2019" version="16.9.3+1.0.tar">
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 29, 2021

They do replace the older ones, so it would be OK to remove them. If you are going to do that, then you might also remove the Xcode 11.3 entry.

jgneff added 2 commits Apr 29, 2021
Add more details to the file 'gradle/README.txt' on how to create and
update the dependency verification file for Linux, macOS, Windows, and
the internal Oracle builds.
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 30, 2021

The changes look good. I'm doing one last CI build to double-check everything.

One thing I just thought of: Since you removed the old devkit entries, this will need to wait for PR #482. That PR needs one more reviewer and should be ready soon, so I don't think this will cause a delay.

Copy link
Member

@kevinrushforth kevinrushforth left a comment

Looks good.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 30, 2021

PR #482 is now integrated, so once you get a second review, from Johan, you can integrate this.

@johanvos
Copy link
Collaborator

@johanvos johanvos commented May 1, 2021

Looks good, I'll do a deeper inspection tomorrow.

@openjdk
Copy link

@openjdk openjdk bot commented May 3, 2021

@jgneff This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8264010: Add Gradle dependency verification

Co-authored-by: Kevin Rushforth <kcr@openjdk.org>
Reviewed-by: kcr, jvos

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 2 new commits pushed to the master branch:

  • 7ec132c: 8265399: Update to Visual Studio 2019 version 16.9.3
  • affb108: 8137323: Incorrect parsing of mnemonic in controls text

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label May 3, 2021
@jgneff
Copy link
Member Author

@jgneff jgneff commented May 3, 2021

/integrate

@openjdk openjdk bot closed this May 3, 2021
@openjdk openjdk bot added integrated and removed ready rfr labels May 3, 2021
@openjdk
Copy link

@openjdk openjdk bot commented May 3, 2021

@jgneff Since your change was applied there have been 2 commits pushed to the master branch:

  • 7ec132c: 8265399: Update to Visual Studio 2019 version 16.9.3
  • affb108: 8137323: Incorrect parsing of mnemonic in controls text

Your commit was automatically rebased without conflicts.

Pushed as commit a9f6035.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@jgneff jgneff deleted the jgneff:dependency-verification branch May 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants