Reproducible Builds

[jdaugherty]

ASF Requirements

To begin releasing under the ASF, the Grails project must adhere to it's release policies. The requirements of Automated Release Signing require a reproducible build.

History of Custom Transformations

Grails makes heavy use of:

1. AST transforms
2. properties files
3. custom xml metadata files

Historically, these transforms were unordered, and up until Groovy 4 it was not possible to order them. There were a set of transforms where order did matter. They were grouped by function - the grails-core framework transforms & the grails-data-mapping transforms. To work around the Groovy limitation, Grails introduced both a GlobalGrailsClassInjectorTransformation (for grails-core) and OrderedGormTransformation (for grails-data-mapping). These AST transformations would then call other transformers in their defined order.

The problem with the Grails implementation is that as each annotated node was processed by the Groovy transform, it would call all of the Grails transformers. This mean that the order of processing would often be: Class Node (apply all transformers), Method Node (apply all transformers), etc. While the Groovy 4 TransformWithPriority interface processes all nodes under a single transform, and then moves to the next highest priority transformer.

So far, we have worked towards reproducible builds by moving as many transforms out of the custom grails & data-mapping transformers. We've adopted the Groovy TransformWithPriority interface where possible. We've switched our method iterations & various caches to use collections with determined sort orders. We also have changed our xml files & properties files to write consistent dates based on the presence of a SOURCE_DATE_EPOCH environment variable that is set in CI based on the last commit date.

GroovyTransformOrder

Because the order of AST Transforms was not guaranteed in Groovy 3, we determined the Grails 7 order by extensive debugging. Once we added various combinations on a class and listed the resulting transforms, we created an order registry in the class GroovyTransformOrder. This class lists the transforms in order from highest priority (first run) to lowest priority (last run).

Future Deprecations

Debugging some of these transforms revealed that some are no longer heavy used and have limited test coverage. In some cases, use was even discouraged (i.e. the mixin transform). This ticket deprecates the Mixin transform since AST transforms should be adopted instead. We will remove in a future Grails Version.

Affect on Grails Application

In addition to the security benefits, having a reproducible build means that Grails Applications will likely be reproducible as well. We also can make better use of Gradle caching & transient recompilation issues that have been seen previously will likely cease. As part of preparing for the ASF release process, we have made almost all of the grails-core build parallel & lazy. Gradle tasks such as the views compilers & GSP compilers are now cache-able and should not always rerun when there are no changes. Grails Applications updating to Grails 7 will likely have to update their build files to be compatible with these Gradle enhancements.

Initial PR work

Initial Pull Requests of the reproducibility work are:

1. PR #1 - Working towards reproducible builds #14670
2. PR #2 - reproducible builds #14677
3. reproducible builds - consistent property file dates #14675
4. (tangentially related) Redesign of TCK #14674

The initial Pull Requests to refactor the entire Grails Gradle Build include:

1. (majority of the work) PR #2 Merge grails-data-mapping & grails-geb into grails-core #14324
2. (documentation related tasks) Fix tests from geb/mapping merge #14342

Current Differing Builds

Of the 263 jars, we have the remaining 26 differing jars:

	grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
	grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-data-hibernate5/grails-plugin/build/libs/grails-data-hibernate5-7.0.0-SNAPSHOT.jar
	grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
	grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
	grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
	grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
	grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
	grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
	grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
	grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
	grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
	grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
	grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar

Testing Changes

A script is checked in under etc/bin that can be run at the root of the project to run 2 back-to-back builds to find differences and compare the results:

      etc/bin/test-reproducible-builds.sh

After running this script, the following artifacts are produced under etc/bin/results:

1. diff.txt - the jars that have differences
2. first.txt - the jars found in the first run, including their sha256 hash
3. second.txt - the jars found in the second run, including their sha256 hash
4. first - the folder containing the jars found in the first run
5. second - the folder containing the jars found in the second run

A jar path can be copied from diff.txt and the script etc/bin/extract-build-artifact.sh can be run to pull each jar, & extract it to firstArtifact / secondArtifact. From there, the directories can be compared in IntelliJ to see why the jars are different.

Remaining Issues:

26 of 263 jars are mismatching

1. 13 of these jars are the javadoc jars. Grails uses the Gradle Groovydoc task to produce documentation for groovy & java source code. The generated groovydoc is then injected into the javadoc file in place of any javadoc. This task is currently using groovy 3 because Gradle does not have Groovy 4. We are seeing reordering differences on parent methods (i.e. Object / GroovyObject) in the javadoc.

2. 13 of these jars appear to have ordering issues in their class files. The simplest case is the ApplicationTagLib class under the artifact path grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar. This class does not appear to have any significant transforms being applied to it, but it does make use of multiple traits.

3. All of the Grails Data mapping transforms have been split from the OrderedGormTransformation except the Transaction & Tenant Transformations. These transformations appear to call each other, and calling the transaction transformation on all nodes prior to the tenant will fail with an access forbidden to a tenantId field. See BookService data service in the grails-datamapping-core-test for an example failure.

Next Steps

This ticket is to track the remaining work for reproducible builds.

[jdaugherty]

The reordering issues appear to all be trait related. It may be related to this change inheritance change that was fixed in later Groovy 4 versions (see this commit)

[jdaugherty]

As a workaround (in addition to the diff), we may be able to make use of the command javap to list method & parameter orders.

[jromero-onclick]

I'm sorry I can't help due to lack of knowledge, but I wanted to thank you for the great work you're doing. Following the discussions on the developer mailing list, I'm eagerly awaiting the new Grails 7 milestone :)

[jdaugherty]

These are the failures currently after updating profiles:

	grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
	grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-data-hibernate5/grails-plugin/build/libs/grails-data-hibernate5-7.0.0-SNAPSHOT.jar
	grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
	grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
	grails-gradle/docs-core/build/libs/grails-docs-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-gradle/model/build/libs/grails-gradle-model-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
	grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
	grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
	grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
	grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
	grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
	grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
	grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
	grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
	grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar

in #14710

[jdaugherty]

After updating to the 4.0.27 snapshot build with the groovydoc fixes by Paul, these are the remaining jar's now:

	grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
	grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
	grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
	grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
	grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
	grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
	grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
	grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
	grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
	grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
	grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
	grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar

[jdaugherty]

Minor correction to the original research: Paul noted that the ordering of AST transform application in Groovy 3 was deterministic - it just depended on how it was defined in what projects. That's why we had consistent behavior before.

[jdaugherty]

After the merge of wrapper, core, and fixes to the grails-gradle jar generation, there are a total of 290 jar files generated by the Grails build.

[jdaugherty]

Paul has been working on this and has determined the order is due to Class#getMethods not guaranteeing order. Groovy expects calls to be deterministic, but the Grails project is able to somehow cause indeterminate ordering. He is looking at how to work around this, so far some of the changes he's made reduces the above list by half, but he's still investigating for a more complete solution.