Add checksum-dependency-plugin for verification of plugin/dependency checksums

[vlsi]

checksum-dependency-plugin is a superset of gradle-witness, and it enables to increase the level of security.

See https://github.com/vlsi/vlsi-release-plugins#checksum-dependency-plugin

fixes #395

Note: I have not removed gradle-witness, however it becomes obsolete.
Note: <trust-requirement pgp='MODULE' checksum='MODULE' /> could be used to verify checksum always. Current configuration (pgp=GROUP checksum=NONE) uses PGP when available, and resorts to checksum when PGP is missing.

[ligi]

Thanks - this looks really good - much appreciated! Will most likely merge - I just need to read up a bit more on the used plugin before merging. But what I see currently it looks like an improvement - thanks!

[vlsi]

Nice. Feel free to ping me or file an issue at https://github.com/vlsi/vlsi-release-plugins/issues

[ligi]

@vlsi one question: what is the process when updating dependencies with this plugin?

Also currently having this strange effect after updating kotlin to 1.3.50:

> Checksum/PGP violations detected on resolving configuration :classpath
    Trusted PGP keys for group org.jetbrains.kotlin are [98fe03a974ce0a0b], however artifact is signed by [6a0975f8b1127b83] only:
      org.jetbrains.kotlin:kotlin-android-extensions:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-annotation-processing-gradle:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-build-common:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-compiler-embeddable:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-compiler-runner:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-daemon-client:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-daemon-embeddable:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-gradle-plugin-api:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-gradle-plugin-model:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-native-utils:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-reflect:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-script-runtime:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-scripting-common:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-scripting-jvm:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-stdlib-common:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-stdlib:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-util-io:1.3.50 (pgp=[6a0975f8b1127b83], sha512=[computation skipped])

Do you know what happened there? I doubt jetbrains changed their PGP key for this release.

[vlsi]

Does https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin#updating-dependencies answer your question?

Technically speaking, the plugin saves the updated file as $rootDir/build/checksum/checksum.xml, so you can inspect it (or copy over $rootDir/checksum.xml and see the diff via git diff).
Do you think "Checksum/PGP violations detected" message should suggest that updated checksum.xml is available and/or one can use -PchecksumUpdate to update the file in place?

Frankly speaking I refrained from adding that to prevent people from blindly using -PchecksumUpdate all over the place.

[vlsi]

I think JetBrains indeed used a different key to publish 1.3.50.
I've filed an issue so they publish the set of the authorized release keys: https://youtrack.jetbrains.com/issue/KT-33781

[ligi]

Thanks so much for the insight. Facing the following problem now:

Checksum/PGP violations detected on resolving configuration :app:_internal_aapt2_binary
No PGP signature (.asc file) found for artifact:
com.android.tools.build:aapt2:3.4.1-5326820:linux (pgp=[], sha512=[2FECA0A69BAB243434B9E7586BA69CAB6ACE9D1A5DF22EF85DC4F688996983BC98E11C27FF4A1126C10FD4E35768CDA52B57959A3A3D20A19D5F15729BEE8327])

this happens on the unmodified version of this PR. Perhaps it is only a problem on linux? Unfortunately my build-servers run on linux and I cannot merge the PR without the agreement of the CI

[vlsi]

@ligi , unfortunately, there's no generic way to ask Gradle to resolve "all the dependencies the project would ever require".

For CI configuration I suggest you add -PchecksumFailOn=build_finish -PchecksumPrint (it can be configured with command line and/or with environment variables)

That would make your CI to collect all the violations and print the updated checksum.xml file to the build log.
Then you can get that contents and update checksum.xml (of course, you review the diff)

Feel free to force-push the updates to vlsi/pgp branch, or let me know what is the updated checksum.xml.
I'm using macOS.

[ligi]

Thanks! I did this - but afterwards I end up with:

> Checksum/PGP violations detected on resolving configuration :app:composer
    No PGP signature (.asc file) found for artifact:
      com.linkedin.dextestparser:parser:1.1.0 (pgp=[], sha512=[2AAD73C3FE10D2D6E0E863F8C0A90B04A52D6C70FBADE1B9F29888815BDA4C4C4B9189E64FBD2FC71C8A721A2479D9F7E1DB4CD56AF2A80096BA591731207968])
    No trusted PGP keys are configured for group com.beust:
      com.beust:jcommander:1.71 (pgp=[5ad66315fc018797], sha512=[computation skipped])
    No trusted PGP keys are configured for group com.gojuno.commander:
      com.gojuno.commander:android:0.1.7 (pgp=[1ed2576e643cdeca], sha512=[computation skipped])
      com.gojuno.commander:os:0.1.7 (pgp=[1ed2576e643cdeca], sha512=[computation skipped])
    No trusted PGP keys are configured for group com.gojuno.composer:
      com.gojuno.composer:composer:0.6.0 (pgp=[1ed2576e643cdeca], sha512=[computation skipped])
    No trusted PGP keys are configured for group io.reactivex:
      io.reactivex:rxjava:1.3.0 (pgp=[94b291aef984a085], sha512=[computation skipped])
    Trusted PGP keys for group commons-io are [86fdc7e2a11262cb], however artifact is signed by [9c4f7e9d98b1cc53] only:
      commons-io:commons-io:2.5 (pgp=[9c4f7e9d98b1cc53], sha512=[computation skipped])
    Trusted PGP keys for group org.apache.commons are [a2115ae15f6b8b72], however artifact is signed by [9daadc1c9fcc82d0] only:
      org.apache.commons:commons-lang3:3.5 (pgp=[9daadc1c9fcc82d0], sha512=[computation skipped])
    Trusted PGP keys for group org.jetbrains.kotlin are [98fe03a974ce0a0b], however artifact is signed by [b2c4d8b48a99f98a] only:
      org.jetbrains.kotlin:kotlin-reflect:1.1.1 (pgp=[b2c4d8b48a99f98a], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-runtime:1.1.1 (pgp=[b2c4d8b48a99f98a], sha512=[computation skipped])
      org.jetbrains.kotlin:kotlin-stdlib:1.1.2 (pgp=[b2c4d8b48a99f98a], sha512=[computation skipped])

[vlsi]

Thanks! I did this - but afterwards I end up with:

By default the plugin fails on the first violation. So it is expected it might fail multiple times provided you add dependencies one by one.

As build output suggests, if you are brave enough, you might add -PchecksumFailOn=build_finish. Then it would fail just once.

[ligi]

Thanks! Should not have done this on the road - but was excited about this PR - now at home on my desk everything works like a charm - the problem was the isCI flag that dragged in different dependencies.
PR will most likely be merged soon