KT-64377 Maven dev publish #3433

adam-enko · 2023-12-20T17:38:50Z

Use project-local Maven dev repos for integration testing.

Impact

Using a Maven Repo is useful for integration tests, because it verifies that the artifacts are published correctly with the expected coordinates. It is also useful for Gradle Plugins, because we can verify that the plugin marker was correctly published.

Removing the dependency on Maven Local is one of the last steps before Dokka's build logic (just the build logic, not Dokka Gradle Plugin!) can be made fully Config Cache compatible.

Verification

To test locally, you can use configuration cache:

./gradlew :dokka-integration-tests:check --configuration-cache --configuration-cache-problems=warn

(Although there will be warnings, they can be ignored - Dokka build scripts (not DGP!) will be made CC compatible in upcoming PRs.)

Summary

Local convention plugin dokkabuild.dev-maven-publish.gradle.kts publishes subprojects to a project-local directory.

Benefits:
- compatible with: config cache, build cache, isolated projects
- doesn't use Maven Local, so there's no chance of cross-contamination with outdated artifacts or other projects
Dynamically inject the local Maven dir:
- Gradle projects: find/replace /* %{PROJECT_LOCAL_MAVEN_DIR}% */ with an exclusive Maven repo
- Maven projects: provide a settings.xml with the local plugin repo.

…3418) * remove Gradle wrapper files for integration test projects * .gitignore Gradle wrapper files

…egration-test-build-config

…ity (#3420) rename 'tryK2' to 'useK2' for integration tests

# Conflicts: # dokka-integration-tests/gradle/projects/it-wasm-basic/gradle.properties

…#3428)

…egration-test-build-config

- create a test suite for each project under test (helps with Gradle caching, task avoidance) - update CI/CD tasks to run specific test-suite tasks - disable integration test tasks due to dependency on Maven Local publishing (this will be fixed in an upcoming PR)

…es files

# Conflicts: # build-logic/src/main/kotlin/dokkabuild/DokkaBuildProperties.kt # dokka-integration-tests/gradle/README.md # dokka-integration-tests/gradle/build.gradle.kts # dokka-integration-tests/utilities/src/main/kotlin/org/jetbrains/dokka/it/AbstractIntegrationTest.kt

Maven publish artifacts to project-local directory. This helps with Gradle Build & Config cache.

…suites

dokka-integration-tests/settings.gradle.kts

adam-enko · 2024-02-16T10:01:55Z

dokka-integration-tests/gradle/projects/coroutines/coroutines.diff

+-        classpath "org.jetbrains.kotlinx:kotlinx-knit:$knit_version"
+        classpath("org.jetbrains.kotlinx:kotlinx-knit:$knit_version") {
+            exclude(group: "org.jetbrains.kotlinx", module: "dokka-pathsaver-plugin")
+        }
+        classpath("org.jetbrains.kotlinx:dokka-pathsaver-plugin:$knit_version") {
+            exclude(group: "org.jetbrains.dokka", module: "templating-plugin")
+            exclude(group: "org.jetbrains.dokka", module: "dokka-base")
+        }
+        classpath("org.jetbrains.dokka:templating-plugin:${providers.gradleProperty("dokka_it_dokka_version").get()}")
+        classpath("org.jetbrains.dokka:dokka-base:${providers.gradleProperty("dokka_it_dokka_version").get()}")


Knit uses Dokka v1.8.10 dependencies. Since the local dev-repo is marked as an exclusive source for org.jetbrains.dokka dependencies, and the repo doesn't contain 1.8.10 versions, the build will fail because it can't find the 1.8.10 dependencies.

Maybe there's a better way of doing this? Or update the version of Coroutines we're testing?

Knit uses Dokka v1.8.10 dependencies. Since the local dev-repo is marked as an exclusive source for org.jetbrains.dokka dependencies, and the repo doesn't contain 1.8.10 versions, the build will fail because it can't find the 1.8.10 dependencies.

Maybe there's a better way of doing this? Or update the version of Coroutines we're testing?

Hmm, that's both good and bad that we found it, I didn't realize it would be a problem...

One of the things that these tests test is the compatibility with Dokka plugins, such as knit. Especially for cases such as when AGP started shipping an older versions of Dokka, and it produced build errors if you had a newer version of Dokka in one of the subprojects (not in the root, which had older Dokka from AGP), and all sorts of things ended up on the classpath. So it would help to keep it working roughly the same way as it would work for the users, meaning coroutines will be running with an explicit new version of Dokka, and with knit that depends on the older version of Dokka.

Or update the version of Coroutines we're testing?

Will it help? Coroutines use the published versions of knit that depend on published versions of Dokka, so if we update coroutines - knit would depend not on Dokka 1.8.10, but on Dokka 1.9.10, which doesn't seem to help us that much?

Maybe there's a better way of doing this?

Just a thought: is there a reason for why the local dev repo should be the exclusive source for org.jetbrains.dokka? Can it be resolved with the order of the repositories? Meaning, use the local dev-repo as the first repository, so that it resolves all snapshot dependencies first, and then mavenCentral / mavenLocal after it so that it resolves other versions of Dokka and other dependencies if requested? I understand it will slow down the build (Gradle's lookup will be longer), but it's fine in this particular case, won't make much of a difference - integration tests are expected to be rather long and not really expected to be run locally often anyway.

Maybe there's a better way of doing this?

Just a thought: is there a reason for why the local dev repo should be the exclusive source for org.jetbrains.dokka?

The exclusivity filter is there to ensure the integration tests only test the current Dokka version.

I wanted to prevent the scenario where 1. publishing to the local test-repo silently fails, and 2. the external projects pick up some other Dokka version (for example, from Maven Local).

If you think that risk is acceptable then we could rely on ordering, as you suggested.

I wanted to prevent the scenario where 1. publishing to the local test-repo silently fails, and 2. the external projects pick up some other Dokka version (for example, from Maven Local).

An idea: can the value of dokka_it_dokka_version be something specific to the integration tests? A long time ago it was set to for-integration-tests-SNAPSHOT, I don't know what for exactly, but I can assume now that it was done to avoid the problem you mentioned. The chances of having a for-integration-tests-SNAPSHOT Dokka version in Maven Local seem to be slim, so I think it'd be OK

It doesn't have to be for-integration-tests-SNAPSHOT, but something that is not 2.0.0-SNAPSHOT, not a release, -dev or -test version, which could indeed be available elsewhere

I see that the tests are passing now - were you able to solve it in some other way?

An idea: can the value of dokka_it_dokka_version be something specific to the integration tests?

Unfortunately, it's a little more difficult than that :(

The version is set in multiple places (e.g. there's a generated .properties file in the plugin JAR that contains the version, the version is set into all of the Maven/Gradle metadata.) So in order to set it consistently, it needs to be set from the start. And since the integration-test project is 'just' a consumer of the Dokka subprojects, it's hard for it to influence the version.

Maybe it could be possible if the whole Dokka project just set its version to 1.9.20-DEV by default, but never publishing any -DEV version (we could add some checks in the Gradle tasks to prevent this). Or with some Composite Build hackery.

I see that the tests are passing now - were you able to solve it in some other way?

Oh, the code I highlighted fixed the problem, so the code ran, but I was highlighting to ask if there was a better idea.

IgnatBeresnev · 2024-02-22T07:52:46Z

Tremendous work with this and all the prerequisites, thanks!

Unlike the standard Gradle publication, snapshot versions will not constantly be re-published. (publishToMavenLocal will also not constantly republish snapshot versions, but it's not easily possible to change the location of Maven Local)

Isn't that the whole point of -SNAPSHOT versions, that they are re-published every single time regardless of everything? Or perhaps I misunderstood you about when it doesn't do re-publishing compared to how it works now?

Say, I have 2.0.0-SNAPSHOT now, I run integration tests and I now have a local Maven repo with Dokka 2.0.0-SNAPSHOT. Then I introduce a breaking change in dokka-base (something obvious such as throwing an exception on start), and run integration tests again without doing clean - will a new version be published, or will the tests use the version published previously and thus pass?

gradle/libs.versions.toml

dokka-integration-tests/settings.gradle.kts

dokka-integration-tests/maven/build.gradle.kts

dokka-integration-tests/gradle/projects/serialization/kotlinx-serialization

IgnatBeresnev · 2024-02-22T08:34:39Z

dokka-integration-tests/gradle/projects/coroutines/coroutines.diff

+-        classpath "org.jetbrains.kotlinx:kotlinx-knit:$knit_version"
+        classpath("org.jetbrains.kotlinx:kotlinx-knit:$knit_version") {
+            exclude(group: "org.jetbrains.kotlinx", module: "dokka-pathsaver-plugin")
+        }
+        classpath("org.jetbrains.kotlinx:dokka-pathsaver-plugin:$knit_version") {
+            exclude(group: "org.jetbrains.dokka", module: "templating-plugin")
+            exclude(group: "org.jetbrains.dokka", module: "dokka-base")
+        }
+        classpath("org.jetbrains.dokka:templating-plugin:${providers.gradleProperty("dokka_it_dokka_version").get()}")
+        classpath("org.jetbrains.dokka:dokka-base:${providers.gradleProperty("dokka_it_dokka_version").get()}")


Knit uses Dokka v1.8.10 dependencies. Since the local dev-repo is marked as an exclusive source for org.jetbrains.dokka dependencies, and the repo doesn't contain 1.8.10 versions, the build will fail because it can't find the 1.8.10 dependencies.

Maybe there's a better way of doing this? Or update the version of Coroutines we're testing?

Hmm, that's both good and bad that we found it, I didn't realize it would be a problem...

One of the things that these tests test is the compatibility with Dokka plugins, such as knit. Especially for cases such as when AGP started shipping an older versions of Dokka, and it produced build errors if you had a newer version of Dokka in one of the subprojects (not in the root, which had older Dokka from AGP), and all sorts of things ended up on the classpath. So it would help to keep it working roughly the same way as it would work for the users, meaning coroutines will be running with an explicit new version of Dokka, and with knit that depends on the older version of Dokka.

Or update the version of Coroutines we're testing?

Will it help? Coroutines use the published versions of knit that depend on published versions of Dokka, so if we update coroutines - knit would depend not on Dokka 1.8.10, but on Dokka 1.9.10, which doesn't seem to help us that much?

Maybe there's a better way of doing this?

Just a thought: is there a reason for why the local dev repo should be the exclusive source for org.jetbrains.dokka? Can it be resolved with the order of the repositories? Meaning, use the local dev-repo as the first repository, so that it resolves all snapshot dependencies first, and then mavenCentral / mavenLocal after it so that it resolves other versions of Dokka and other dependencies if requested? I understand it will slow down the build (Gradle's lookup will be longer), but it's fine in this particular case, won't make much of a difference - integration tests are expected to be rather long and not really expected to be run locally often anyway.

IgnatBeresnev · 2024-02-22T08:42:27Z

dokka-integration-tests/gradle/projects/coroutines/coroutines.diff

@@ -1,72 +1,139 @@
 diff --git a/build.gradle b/build.gradle
-index e7d405e12..236310823 100644
+index e7d405e12..db5dcec66 100644


Slightly concerned with how the diff grew, and some additional changes in here.

Ideally, these integration tests (which are e2e really) should represent the real-life application of Dokka in user projects, so I would expect the diff to contain minimal changes to the build logic. If we start excluding dependencies and overriding them explicltly in a user project - it looks like there's a problem with Dokka itself that needs to be addressed, and we might miss actual bugs

Keeping changes to the absolute bare minimum also helps with updating the versions of the tested projects. I mentioned it that updating the diffs is tedious, because people update their build configs and we then have to resolve merge conflicts and figure out their whole setup to understand where to add things. So far it's been simple: add mavenLocal, bump the version of Dokka, done. Now there's some additional repositories and plugins, dependency manipulation and so on. So it'd be more difficult for us to update it now compared to before, but I think I'm fine with it if you agree to help us update the diffs on request

Yeah, modifying the projects via git diffs is not nice. Testing the external Gradle projects would be easier if they used idiomatic Gradle (for example, if they used Version Catalogs then there would be just one place where the Dokka version should be changed). The build config of Serialization and Coroutines has improved in recent versions, so maybe it would be easier.

We did discuss a while ago about alternatives, and one option is to create a fork of the external projects in Space, and modify it to use the latest Dokka version. That would be easier to update and maintain.

In the meantime, yes, I can help out with updating the external projects as required.

… `maven.repo.local`

dokka-integration-tests/settings.gradle.kts

adam-enko · 2024-03-06T18:11:54Z

Isn't that the whole point of -SNAPSHOT versions, that they are re-published every single time regardless of everything? Or perhaps I misunderstood you about when it doesn't do re-publishing compared to how it works now?

When publishing to repos, yes, Gradle will always publish a new version.

This is okay for testing: because we're using a local repo for testing, publishing is effectively a 'zip & copy files' operation, which is cheap and quick to do, and isn't worth caching.

Say, I have 2.0.0-SNAPSHOT now, I run integration tests and I now have a local Maven repo with Dokka 2.0.0-SNAPSHOT. Then I introduce a breaking change in dokka-base (something obvious such as throwing an exception on start), and run integration tests again without doing clean - will a new version be published, or will the tests use the version published previously and thus pass?

Yes, Gradle will detected that the source files changed, so it will re-run the compilation tasks. The resulting files will be packaged and re-published.

- only register deterministic repo files as a task input - cache the output of Gradle integration test tasks

…publish

build-logic/src/main/kotlin/dokkabuild.dev-maven-publish.gradle.kts

dokka-runners/runner-gradle-plugin-classic/build.gradle.kts

…publish

adam-enko added 5 commits December 19, 2023 10:33

KT-64377 - remove gradle wrapper files for integration test projects (#…

c31dd23

…3418) * remove Gradle wrapper files for integration test projects * .gitignore Gradle wrapper files

Merge branch 'master' into KT-64377/update-integration-test-build-config

61889af

Merge remote-tracking branch 'origin/master' into KT-64377/update-int…

74a1b20

…egration-test-build-config

refactor integration test properties to use DokkaBuildProperties util…

20e0984

…ity (#3420) rename 'tryK2' to 'useK2' for integration tests

Merge branch 'master' into KT-64377/update-integration-test-build-config

66287db

# Conflicts: # dokka-integration-tests/gradle/projects/it-wasm-basic/gradle.properties

adam-enko mentioned this pull request Jan 12, 2024

KT-64377 use Java Test Suites for Gradle integration tests #3427

Merged

adam-enko added 6 commits January 12, 2024 17:30

disable explicit API mode in non-published integration-test utilities (…

28f4d7a

…#3428)

Merge remote-tracking branch 'origin/master' into KT-64377/update-int…

7af0394

…egration-test-build-config

revert GitHub smoke test Gradle task back to 'test'

5f35a73

auto-discover and set ANDROID_SDK, and also search for local.properti…

0babb2c

…es files

add TODO link for integration tests - doNotTrackState()

9c6c80e

adam-enko mentioned this pull request Jan 16, 2024

KT-64377 - Preparation for Gradle integration test build config refactoring #3422

Merged

adam-enko changed the base branch from KT-64377/update-integration-test-build-config to master January 19, 2024 14:07

adam-enko added 2 commits January 22, 2024 11:02

tidy up after merge

28a1c74

adam-enko force-pushed the KT-64377/maven-dev-publish branch from 3de37c5 to d8c86ac Compare January 22, 2024 10:56

Use Maven Dev Publish plugin

70fd4f8

Maven publish artifacts to project-local directory. This helps with Gradle Build & Config cache.

adam-enko force-pushed the KT-64377/maven-dev-publish branch from d8c86ac to 70fd4f8 Compare January 22, 2024 11:28

adam-enko added 2 commits January 22, 2024 12:28

Merge remote-tracking branch 'origin/master' into KT-64377/java-test-…

8a24b4d

…suites

merge cleanup

57b93cf

adam-enko self-assigned this Jan 22, 2024

adam-enko added runner: Gradle plugin An issue/PR related to Dokka's Gradle plugin infrastructure Everything related to builds tools, CI configurations and project tooling and removed runner: Gradle plugin An issue/PR related to Dokka's Gradle plugin labels Jan 22, 2024

adam-enko added 5 commits January 23, 2024 16:55

change plugin test-suite-base -> jvm-test-suite

a974d9b

move registerTestProjectSuite() to top-level function

852fc7b

move gradle-specific test utils to gradle integration test

ba0c484

move unused systemVariableProviders.kt back

6d5ab63

move multimodule-inter-module-links to test-suite

14364ea

adam-enko commented Feb 16, 2024

View reviewed changes

dokka-integration-tests/settings.gradle.kts Outdated Show resolved Hide resolved

adam-enko commented Feb 16, 2024

View reviewed changes

adam-enko added 2 commits February 16, 2024 12:09

bump dev-publish-plugin version

e39fa37

restore deleted it-multimodule-inter-module-links/gradle.properties

67ba564

adam-enko marked this pull request as ready for review February 16, 2024 13:45

adam-enko requested review from dkrasnoff and IgnatBeresnev February 16, 2024 13:45

dkrasnoff requested a review from Tapchicoma February 21, 2024 16:36

IgnatBeresnev reviewed Feb 22, 2024

View reviewed changes

adam-enko added 2 commits February 23, 2024 17:50

replace dev.adamko.dev-publish with custom implementation that uses…

b29f542

… `maven.repo.local`

Merge branch 'master' into KT-64377/maven-dev-publish

b963d07

adam-enko commented Feb 23, 2024

View reviewed changes

dokka-integration-tests/settings.gradle.kts Outdated Show resolved Hide resolved

adam-enko added 2 commits February 23, 2024 18:01

remove old kotlinx-serialization git submodule

537d5e0

fix CC compatibility

4164cdf

adam-enko added 3 commits March 8, 2024 14:09

fixin' build caching

53a1704

- only register deterministic repo files as a task input - cache the output of Gradle integration test tasks

Merge remote-tracking branch 'origin/master' into KT-64377/maven-dev-…

72a2206

…publish

Merge branch 'master' into KT-64377/maven-dev-publish

fa79d74

dkrasnoff reviewed Mar 18, 2024

View reviewed changes

build-logic/src/main/kotlin/dokkabuild.dev-maven-publish.gradle.kts Outdated Show resolved Hide resolved

dokka-runners/runner-gradle-plugin-classic/build.gradle.kts Show resolved Hide resolved

adam-enko added 2 commits March 18, 2024 15:08

Merge remote-tracking branch 'origin/master' into KT-64377/maven-dev-…

1655ee0

…publish

use tasks.check {} dsl accessor

d6e55de

dkrasnoff approved these changes Mar 19, 2024

View reviewed changes

adam-enko requested a review from IgnatBeresnev March 19, 2024 12:59

IgnatBeresnev approved these changes Mar 21, 2024

View reviewed changes

adam-enko added 3 commits March 22, 2024 18:21

update IB comment

5ee709d

Merge remote-tracking branch 'origin/master' into KT-64377/maven-dev-…

75e43fe

…publish

update comment regarding Maven Metadata exclusions

aa226fa

adam-enko merged commit ad9b822 into master Mar 27, 2024
13 checks passed

adam-enko deleted the KT-64377/maven-dev-publish branch March 27, 2024 11:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KT-64377 Maven dev publish #3433

KT-64377 Maven dev publish #3433

adam-enko commented Dec 20, 2023 •

edited

Loading

adam-enko Feb 16, 2024

IgnatBeresnev Feb 22, 2024

adam-enko Mar 6, 2024

IgnatBeresnev Mar 21, 2024 •

edited

Loading

adam-enko Mar 27, 2024

IgnatBeresnev commented Feb 22, 2024

IgnatBeresnev Feb 22, 2024

IgnatBeresnev Feb 22, 2024

adam-enko Mar 27, 2024

adam-enko commented Mar 6, 2024

KT-64377 Maven dev publish #3433

KT-64377 Maven dev publish #3433

Conversation

adam-enko commented Dec 20, 2023 • edited Loading

Impact

Verification

Summary

adam-enko Feb 16, 2024

Choose a reason for hiding this comment

IgnatBeresnev Feb 22, 2024

Choose a reason for hiding this comment

adam-enko Mar 6, 2024

Choose a reason for hiding this comment

IgnatBeresnev Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

adam-enko Mar 27, 2024

Choose a reason for hiding this comment

IgnatBeresnev commented Feb 22, 2024

IgnatBeresnev Feb 22, 2024

Choose a reason for hiding this comment

IgnatBeresnev Feb 22, 2024

Choose a reason for hiding this comment

adam-enko Mar 27, 2024

Choose a reason for hiding this comment

adam-enko commented Mar 6, 2024

adam-enko commented Dec 20, 2023 •

edited

Loading

IgnatBeresnev Mar 21, 2024 •

edited

Loading