Gradle modernization #1642

rozza · 2025-03-05T17:11:43Z

This PR comprises of 3 separate commits for the purpose of aiding reviewability.

The commits themselves aren't designed to be standalone (they are not) but rather compose separate areas of work that updating the build has required.

Added library conventions and language based configurations. - Added `doclets` to `buildSrc` for the `javadocs` - Added `conventions` to centralize configuration of gradle plugins - Added `projects` to handle project level conventions JAVA-5758 JAVA-5757

Added library conventions and language based configurations. - Updated to Gradle 8 (JAVA-5756) - Removed `util:taglets` as now in `buildSrc`. - Moved `util:spock` into `driver-core` as the annotation is now shared via `testArtifacts`. - Added optional support (JAVA-5757) JAVA-5756 JAVA-5758 JAVA-5757

Added library conventions and language based configurations. - Refactored `JsonPoweredTestHelper` to handle either local and jar sources - Refactored test cases to use new `JsonPoweredTestHelper.getTestDocuments` method - Refactored test cases to normalize usage of loading test data - Moved scala integration tests into a standardized `integerationTest` directory JAVA-5758

rozza · 2025-03-12T18:11:38Z

Testing builds between `main` and `JAVA-5758` branches.

Output testing

Manual checks

javadoc - ensured taglets output correctly
scaladoc - ensured mongo-scala-driver docs include the bson-scala scaladoc

The following shows how I tested / compared outputted pom.xml files, MANIFEST.MF files and the gradle tasks run when running the check command.

Generating build output for main:

git co main
./gradlew clean publishAllPublicationsToMaven2Repository

mkdir -p ./build/checks
branch_name=$(git rev-parse --abbrev-ref HEAD)
file_prefix="./build/checks/${branch_name}"

# output pom files &> /dev/null
find ./build/repo -name "*.pom" | xargs -I {} cat {} > ${file_prefix}-poms.txt

# output manifests &> /dev/null
find ./build/repo -regex ".*[[:digit:]]\.jar" | sort | xargs -I {} unzip -p {} META-INF/MANIFEST.MF  &> ${file_prefix}-manifests.txt

# check tasks list &> /dev/null
./gradlew check --dry-run | sort -o ${file_prefix}-check-tasks.txt

Generating build output for JAVA-5758:

Note: Here we also clean the previously created build/docs directory caused by publishing the main branch. Its needed to clean up elements-list artifacts that break javadoc.

git co JAVA-5758
rm -rf build/docs
./gradlew clean publishMavenToLocalBuildRepo 

mkdir -p ./build/checks
branch_name=$(git rev-parse --abbrev-ref HEAD)
file_prefix="./build/checks/${branch_name}"

# output pom files &> /dev/null
find ./build/repo -name "*.pom" | xargs -I {} cat {} > ${file_prefix}-poms.txt

# output manifests &> /dev/null
find ./build/repo -regex ".*[[:digit:]]\.jar" | sort | xargs -I {} unzip -p {} META-INF/MANIFEST.MF  &> ${file_prefix}-manifests.txt

# check tasks list &> /dev/null
./gradlew check --dry-run | sort -o ${file_prefix}-check-tasks.txt

Comparing output

Compare pom files ✅

Compare the main branch outputted poms with that of the new gradle build:

diff ./build/checks/main-poms.txt ./build/checks/JAVA-5758-poms.txt

Differences:

All BSON jars now have the url set to bson.org (not just the bson.jar)
All BSON jars use capitalized BSON (previously it was mixed BSON and Bson)
License url updated to use https (previously it was mixed)
developerConnection updated to use https: scm:https://github.com/mongodb/mongo-java-driver.git (git@github support has been removed)
Order of some dependencies may have changed (usually slf4j)
Scala libraries bumped to latest patch.

Compare manifests ✅

Compare the main branch outputted MAINFESTS with that of the new gradle build:

diff ./build/checks/main-manifests.txt ./build/checks/JAVA-5758-manifests.txt

Differences:

Build-Version updated - expected
Bnd-LastModified updated - expected
driver-legacy now has a manifest - side effect of the single publishing config

Compare check tasks ✅

Check any tasks that existed in main but not in JAVA-5758:

diff build/checks/main-check-tasks.txt build/checks/JAVA-5758-check-tasks.txt | grep ^\<  > build/checks/check-tasks-removals.txt

codenarc / compile groovy removed from projects without spock
checkstyle removed from non java projects
util project and all tasks removed

Check any tasks added to JAVA-5758:

diff build/checks/main-check-tasks.txt build/checks/JAVA-5758-check-tasks.txt | grep ^\>  > build/checks/check-tasks-additions.txt

Spotless tasks added to all projects
integrationTest tasks for kotlin
testJar added for test artifacts
buildSrc project added

Evergreen test counts

Env & task	main	JAVA-5758		Findings
latest Sharded Cluster Auth SSL JDK8 Linux test-core	4718	4776	⬆️	Unknown: Possibly `getData` helper fetching more tests? - diff of logs wasn't instructive
latest Sharded Cluster Auth SSL JDK8 Linux test-legacy	1033	1033
latest Sharded Cluster Auth SSL JDK8 Linux test-reactive	3218	3218
latest Sharded Cluster Auth SSL JDK8 Linux test-sync	4454	4460	⬆️
Load Balancer latest Auth SSL JDK21 Ubuntu	1826	1826
Scala 2.13 JDK17 7.0 Replica Set Ubuntu	944	944
Kotlin: JDK17 7.0 Replica Set Ubuntu	1314	1227	⬇️ ⚠️	Now Fixed - Kotlin extensions had the wrong junit library
JDK8 Linux Unit test bson and crypt	3813	3813

Source Main: Evergreen main build
Source PR: Evergreen patch build

Issues found and fixed by this testing

Scala language dependencies updated to api from implementation to keep the poms the same.
slf4j dependencies updated to optionalAPI from optionalImplementation to keep poms the same.
bson-kotlinx dependencies for datetime and json serialization updated to optionalAPI from optionalImplementation to keep poms the same.
Fixed various mainfest attributes as they differed from previous versions
Fixed driver-kotlin-extensions testing and updated kotlin junit dependency in libs.versions.toml to use junit5.

…to use junit 5

NathanQingyangXu · 2025-03-13T18:01:05Z

bom/build.gradle.kts

+                        "BOM must not contain <scope> elements in dependency:\n$destination"
+                    }
+                    assert(it.getNode("optional") == null) {
+                        "BOM must not contain <scope> elements in dependency:\n$destination"


should be "BOM must not contain <optional> elements ..."?

jyemin · 2025-03-13T18:59:34Z

driver-legacy now has a manifest - side effect of the single publishing config

It was intentional that driver-legacy doesn't have a manifest (and also can never be a Java module). The reason is that OSGi requires that Java packages are segregated between modules, i.e. you can't have more than module containing types in the same Java package. And for various reasons, we were forced to do that with driver-legacy, as there are types in both core and legacy in the "com.mongodb" package (and same for "org.bson" in the bson module.

Is there a straightforward way to exclude the manifest for driver-legacy? I know there is at least one place where our module names are referenced in buildSrc. Maybe this is another one?

jyemin

I haven't reviewed the buildSrc directory yet.

jyemin · 2025-03-13T19:24:26Z

bom/build.gradle.kts

+}
+
+/*
+ * Validate the Bom file.


Suggested change

* Validate the Bom file.

* Validate the BOM file.

jyemin · 2025-03-13T19:25:35Z

bom/build.gradle.kts

+ * Modify the generated POM to include all supported versions of Scala for driver-scala or bson-scala.
+ */
+configureMavenPublication {
+    components.findByName("javaPlatform")?.let { from(it) }


What does this do?

jyemin · 2025-03-13T19:29:24Z

bson-scala/src/test/scala/org/mongodb/scala/bson/BaseSpec.scala


-@RunWith(classOf[JUnitRunner])


Why is this no longer needed?

jyemin · 2025-03-13T19:32:04Z

bson/build.gradle.kts

@@ -0,0 +1,38 @@
+/*


I don't see a deleted build.gradle in this directory, but it's not in the source either. Where did it get deleted?

jyemin · 2025-03-13T20:12:11Z

graalvm-native-image-app/src/main/com/mongodb/internal/graalvm/NativeImageApp.java

@@ -18,7 +18,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

-import javax.annotation.Nullable;
+import org.jetbrains.annotations.Nullable;


Why do we need the jetbrains one? I didn't know we depended on an artifact that contains this annotation?

NathanQingyangXu · 2025-03-14T02:28:49Z

buildSrc/src/main/kotlin/conventions/testing-integration.gradle.kts

+
+configurations[integrationTest.implementationConfigurationName].extendsFrom(configurations.testImplementation.get())
+
+configurations[integrationTest.runtimeOnlyConfigurationName].extendsFrom(configurations.testRuntimeOnly.get())


If you are tired of "integrationTestImplementation"(...), you might well replace the above two statements with

val integrationTestImplementation: Configuration by configurations.getting { extendsFrom(configurations.implementation.get()) } val integrationTestRuntimeOnly: Configuration by configurations.getting { extendsFrom(configurations.runtimeOnly.get()) }

then you can use integrationTestImplementation(...) directly

nvm. Further more configs are required. maybe not worth while.

rozza force-pushed the JAVA-5758 branch 6 times, most recently from 0144d6e to aeda926 Compare March 6, 2025 13:35

rozza changed the base branch from gradle to main March 6, 2025 13:39

rozza changed the base branch from main to gradle March 6, 2025 13:40

rozza marked this pull request as draft March 6, 2025 13:42

rozza force-pushed the JAVA-5758 branch from aeda926 to d730cc3 Compare March 6, 2025 14:46

rozza marked this pull request as ready for review March 6, 2025 14:49

rozza requested a review from katcharov March 6, 2025 14:49

rozza force-pushed the JAVA-5758 branch 4 times, most recently from 1ca551a to d80ee10 Compare March 11, 2025 14:25

rozza changed the title ~~Gradle modernization to precompile build scripts~~ Gradle modernization Mar 11, 2025

rozza removed the request for review from katcharov March 11, 2025 16:30

rozza marked this pull request as draft March 11, 2025 16:30

rozza force-pushed the JAVA-5758 branch 3 times, most recently from 5fa97ac to 3f03dbd Compare March 12, 2025 12:11

rozza added 3 commits March 12, 2025 17:51

Gradle: Added buildSrc (1/3)

266a2ef

Added library conventions and language based configurations. - Added `doclets` to `buildSrc` for the `javadocs` - Added `conventions` to centralize configuration of gradle plugins - Added `projects` to handle project level conventions JAVA-5758 JAVA-5757

rozza force-pushed the JAVA-5758 branch from 3f03dbd to 3311cc4 Compare March 12, 2025 17:52

Fix kotlin junit library version and update driver-kotlin-extensions …

ea08cde

…to use junit 5

rozza marked this pull request as ready for review March 13, 2025 14:37

jyemin self-requested a review March 13, 2025 15:18

rozza requested review from a team and vbabanin and removed request for a team March 13, 2025 16:10

NathanQingyangXu reviewed Mar 13, 2025

View reviewed changes

jyemin requested changes Mar 13, 2025

View reviewed changes

NathanQingyangXu reviewed Mar 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gradle modernization #1642

Gradle modernization #1642

rozza commented Mar 5, 2025

rozza commented Mar 12, 2025 •

edited

Loading

NathanQingyangXu Mar 13, 2025 •

edited

Loading

jyemin commented Mar 13, 2025

jyemin left a comment

jyemin Mar 13, 2025

jyemin Mar 13, 2025

jyemin Mar 13, 2025

jyemin Mar 13, 2025

jyemin Mar 13, 2025

NathanQingyangXu Mar 14, 2025

NathanQingyangXu Mar 14, 2025


		configurations[integrationTest.implementationConfigurationName].extendsFrom(configurations.testImplementation.get())

		configurations[integrationTest.runtimeOnlyConfigurationName].extendsFrom(configurations.testRuntimeOnly.get())


		@RunWith(classOf[JUnitRunner])

Gradle modernization #1642

Are you sure you want to change the base?

Gradle modernization #1642

Conversation

rozza commented Mar 5, 2025

rozza commented Mar 12, 2025 • edited Loading

Testing builds between main and JAVA-5758 branches.

Output testing

Manual checks

Generating build output for main:

Generating build output for JAVA-5758:

Comparing output

Compare pom files ✅

Compare manifests ✅

Compare check tasks ✅

Evergreen test counts

Issues found and fixed by this testing

NathanQingyangXu Mar 13, 2025 • edited Loading

Choose a reason for hiding this comment

jyemin commented Mar 13, 2025

jyemin left a comment

Choose a reason for hiding this comment

jyemin Mar 13, 2025

Choose a reason for hiding this comment

jyemin Mar 13, 2025

Choose a reason for hiding this comment

jyemin Mar 13, 2025

Choose a reason for hiding this comment

jyemin Mar 13, 2025

Choose a reason for hiding this comment

jyemin Mar 13, 2025

Choose a reason for hiding this comment

NathanQingyangXu Mar 14, 2025

Choose a reason for hiding this comment

NathanQingyangXu Mar 14, 2025

Choose a reason for hiding this comment

rozza commented Mar 12, 2025 •

edited

Loading

Testing builds between `main` and `JAVA-5758` branches.

NathanQingyangXu Mar 13, 2025 •

edited

Loading