Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JetBrains cache-redirector #3650

Merged
merged 10 commits into from
Jun 27, 2024
106 changes: 68 additions & 38 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,20 @@ Bug reports, feature requests and questions are welcome. Submit issues [here](ht

* Search for existing issues to avoid reporting duplicates.
* When submitting a bug report:
* Test it against the most recently released version. It might have been fixed already.
* Include code that reproduces the problem. Provide a complete reproducer, yet minimize it as much as
possible. A separate project that can be cloned is ideal.
* If the bug is in behavior, explain what behavior you expected and what the actual result is.
* Test it against the most recently released version. It might have been fixed already.
* Include code that reproduces the problem. Provide a complete reproducer, yet minimize it as much as
possible. A separate project that can be cloned is ideal.
* If the bug is in behavior, explain what behavior you expected and what the actual result is.
* When submitting a feature request:
* Explain why you need the feature.
* Explaining the problem you face is more important than suggesting a solution.
Report your problem even if you have no proposed solution.
* Explain why you need the feature.
* Explaining the problem you face is more important than suggesting a solution.
Report your problem even if you have no proposed solution.

## Submitting PRs

Dokka has extensive [Developer Guides](https://kotlin.github.io/dokka/1.9.20/developer_guide/introduction/) documentation
which goes over the development [Workflow](https://kotlin.github.io/dokka/1.9.20/developer_guide/workflow/) and
Dokka has extensive [Developer Guides](https://kotlin.github.io/dokka/1.9.20/developer_guide/introduction/)
documentation
which goes over the development [Workflow](https://kotlin.github.io/dokka/1.9.20/developer_guide/workflow/) and
[Dokka's architecture](https://kotlin.github.io/dokka/1.9.20/developer_guide/architecture/architecture_overview/),
which can help you understand how to achieve what you want and where to look.

Expand All @@ -39,7 +40,7 @@ For any code changes:
* Write tests that reproduce the bug or test new features.
* PRs without tests are accepted only in exceptional circumstances if it is evident that writing the
corresponding test is too hard or otherwise impractical.
* If you add new or change old public API, [update public API dump](#updating-public-api-dump), otherwise it will fail
* If you add new or change old public API, [update public API dump](#updating-public-api-dump), otherwise it will fail
the build.

Please [contact maintainers](#contacting-maintainers) in advance to coordinate any big piece of work.
Expand All @@ -54,15 +55,15 @@ Building Dokka is pretty straightforward:
./gradlew build
```

This will build all subprojects and trigger `build` in all composite builds. If you are working on a
This will build all subprojects and trigger `build` in all composite builds. If you are working on a
[runner](dokka-runners), you can build it independently.

Checks that are performed as part of `build` do not require any special configuration or environment, and should
Checks that are performed as part of `build` do not require any special configuration or environment, and should
not take much time (~2-5 minutes), so please make sure they pass before submitting a pull request.

### Use/test locally built Dokka

Below you will find a bare-bones instruction on how to use and test locally built Dokka. For more details and examples,
Below you will find a bare-bones instruction on how to use and test locally built Dokka. For more details and examples,
visit [Workflow](https://kotlin.github.io/dokka/1.9.20/developer_guide/workflow/) topic.

1. Publish a custom version of Dokka to Maven Local: `./gradlew publishToMavenLocal -Pversion=1.9.20-my-fix-SNAPSHOT`
Expand All @@ -78,22 +79,47 @@ plugins {

There is an automation script for this routine, see [testDokka.sh.md](scripts/testDokka.sh.md) for details.

### Updating Gradle
adam-enko marked this conversation as resolved.
Show resolved Hide resolved

To update Gradle, follow these steps:

1. Open https://gradle.org/release-checksums/ and copy the "Binary-only (-bin) ZIP"
checksum corresponding to the desired Gradle version.
2. Update the Gradle version and checksum by either:
1. Using the wrapper task:

Run `./gradlew wrapper --gradle-version=<new version> --gradle-distribution-sha256-sum=<new checksum>`

2. OR update
[`./gradle/wrapper/gradle-wrapper.properties`](gradle/wrapper/gradle-wrapper.properties):

```diff
- distributionSha256Sum=<old checksum>
+ distributionSha256Sum=<new checksum>
- distributionUrl=[...]/distributions/gradle-<old version>-bin.zip
+ distributionUrl=[...]/distributions/gradle-<new version>-bin.zip
```
3. ⚠️ Run `./gradlew wrapper` to update the Gradle wrapper files.

(Don't forget to run the `wrapper` task after updating the versions,
otherwise [Gradle won't update its wrapper files](https://github.com/gradle/gradle/issues/884).)

### Updating public API dump

[Binary Compatibility Validator](https://github.com/Kotlin/binary-compatibility-validator/blob/master/README.md)
[Binary Compatibility Validator](https://github.com/Kotlin/binary-compatibility-validator/blob/master/README.md)
is used to keep track of public API changes.

Run `./gradlew apiDump` to update API index files after introducing new or changing old public API. Commit updated
Run `./gradlew apiDump` to update API index files after introducing new or changing old public API. Commit updated
API indexes together with other changes.

### Run integration tests

Dokka's [integration tests](dokka-integration-tests) help check compatibility with various versions of Kotlin, Android,
Gradle and Java. They apply Dokka to real user-like projects and invoke Gradle / Maven / CLI tasks to generate the
Dokka's [integration tests](dokka-integration-tests) help check compatibility with various versions of Kotlin, Android,
Gradle and Java. They apply Dokka to real user-like projects and invoke Gradle / Maven / CLI tasks to generate the
documentation.

Integration tests require a significant amount of available RAM (~20-30GB), take 1+ hour and may require additional
environment configuration to run. For these reasons, it's not expected that you run all integration tests locally
Integration tests require a significant amount of available RAM (~20-30GB), take 1+ hour and may require additional
environment configuration to run. For these reasons, it's not expected that you run all integration tests locally
as part of the everyday development process, they will be run on CI once you submit a PR.

However, if you need to run all integration tests locally, you can use the `integrationTest` task:
Expand All @@ -109,7 +135,7 @@ It's possible to run integration tests with a custom Dokka version published to
[MavenCentral](https://central.sonatype.com),
[dev](https://maven.pkg.jetbrains.space/kotlin/p/dokka/dev),
[test](https://maven.pkg.jetbrains.space/kotlin/p/dokka/test) or
`mavenLocal`
`mavenLocal`
via `org.jetbrains.dokka.integration_test.dokkaVersionOverride` Gradle property:

```bash
Expand All @@ -120,11 +146,12 @@ via `org.jetbrains.dokka.integration_test.dokkaVersionOverride` Gradle property:

### Java version

To minimize compatibility problems, [Gradle's Java toolchains](https://docs.gradle.org/current/userguide/toolchains.html)
To minimize compatibility
problems, [Gradle's Java toolchains](https://docs.gradle.org/current/userguide/toolchains.html)
are [used](build-logic/src/main/kotlin/org/jetbrains/conventions/base-java.gradle.kts) to build and test the project.

When run, Gradle tries to auto-detect the required JRE/JDK installation locally, but it may fail if you don't have
that version of Java installed or if it's installed in an unusual location. Please refer to
that version of Java installed or if it's installed in an unusual location. Please refer to
[Gradle's documentation on toolchains](https://docs.gradle.org/current/userguide/toolchains.html#sec:auto_detection)
for customization and problem resolution questions.

Expand All @@ -137,62 +164,65 @@ Dokka with a different version of Java:
| `org.jetbrains.dokka.javaToolchain.testLauncher` | `8` | The version used run unit and integration tests. |

Separating the compiler and test versions is needed to check Dokka's compatibility with various Java versions.
For example, the GitHub Actions based unit tests are
For example, the GitHub Actions based unit tests are
[run under multiple versions of Java](.github/workflows/tests-thorough.yml)
at the same time.

### GitHub Actions

The majority of automated checks and builds are run as
The majority of automated checks and builds are run as
[GitHub Actions workflows](https://docs.github.com/en/actions/using-workflows/about-workflows).

The configuration for Dokka's workflows can be found in [`.github/workflows`](.github/workflows).
The configuration for Dokka's workflows can be found in [`.github/workflows`](.github/workflows).

For your first PR, a maintainer will need to approve the workflow runs manually. For your subsequent PRs, the workflows
will be triggered and run automatically on every PR commit.

While GitHub Actions checks can expose real problems, [TeamCity-based integration tests](#teamcity) need to be run for
any significant changes as they have more thorough compatibility checks.
While GitHub Actions checks can expose real problems, [TeamCity-based integration tests](#teamcity) need to be run for
any significant changes as they have more thorough compatibility checks.

Notable workflows:

* [Publish preview to GitHub Actions Artifacts](https://github.com/Kotlin/dokka/actions/workflows/preview-publish-ga.yml)
builds the HTML API reference of several libraries like `kotlinx.coroutines`, and publishes the results as `zip` archive
builds the HTML API reference of several libraries like `kotlinx.coroutines`, and publishes the results as `zip`
archive
artifacts. You can use it to preview your changes. The workflow is triggered for all commits.
* [Publish preview to web (S3)](https://github.com/Kotlin/dokka/actions/workflows/preview-publish-web-s3.yml)
does the same thing as `Publish examples to GitHub Actions Artifacts`, but publishes the generated documentation
to S3, so it can be accessed from any browser without the need to download anything. The web link will be printed
under the `Print link` job step. This workflow is triggered by maintainer commits only as it requires encrypted
under the `Print link` job step. This workflow is triggered by maintainer commits only as it requires encrypted
secrets to be run.

**Notes**:

* Some workflow job runs are flaky, but if more than a couple are failing at the same time or repeatedly - it indicates
* Some workflow job runs are flaky, but if more than a couple are failing at the same time or repeatedly - it indicates
a problem in the PR that needs to be addressed.
* While a Java version can be configured in the [`setup-java`](https://github.com/actions/setup-java#basic-configuration)
action, it is not necessarily the version of Java that will be used to build/test the project. See the
* While a Java version can be configured in
the [`setup-java`](https://github.com/actions/setup-java#basic-configuration)
action, it is not necessarily the version of Java that will be used to build/test the project. See the
[Java version](#java-version) section for more details.

### TeamCity

TeamCity is used for a subset of important / longer builds and checks, including artifact publication.
TeamCity is used for a subset of important / longer builds and checks, including artifact publication.

The runs are triggered automatically by maintainer commits, or can be started manually by maintainers. External
The runs are triggered automatically by maintainer commits, or can be started manually by maintainers. External
contributors can view the results in guest mode.

Notable builds:

* [Dokka Integration Tests](https://teamcity.jetbrains.com/buildConfiguration/KotlinTools_Dokka_IntegrationTests)
runs Dokka's integration tests, which are designed to test compatibility with different Kotlin versions, with different
runs Dokka's integration tests, which are designed to test compatibility with different Kotlin versions, with
different
multiplatform targets and with various user scenarios.

### Gradle Build Scans

[Gradle Build Scans](https://scans.gradle.com/) can provide insights into a Dokka Build.
[Gradle Build Scans](https://scans.gradle.com/) can provide insights into a Dokka Build.
JetBrains runs a [Gradle Develocity server](https://ge.jetbrains.com/scans?search.rootProjectNames=dokka).
that can be used to automatically upload reports.

To automatically opt in add the following to `$GRADLE_USER_HOME/gradle.properties`.
To automatically opt in add the following to `$GRADLE_USER_HOME/gradle.properties`.

```properties
org.jetbrains.dokka.build.scan.enabled=true
Expand All @@ -205,5 +235,5 @@ A Build Scan may contain identifiable information. See the Terms of Use https://
## Contacting maintainers

* If something cannot be done, not convenient, or does not work &mdash; submit an [issue](#submitting-issues).
* Discussions and general inquiries &mdash; use `#dokka` channel in
* Discussions and general inquiries &mdash; use `#dokka` channel in
[Kotlin Community Slack](https://surveys.jetbrains.com/s3/kotlin-slack-sign-up).
20 changes: 15 additions & 5 deletions build-logic/settings.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,28 @@ rootProject.name = "build-logic"

pluginManagement {
repositories {
mavenCentral()
gradlePluginPortal()
maven("https://cache-redirector.jetbrains.com/repo.maven.apache.org/maven2") {
name = "MavenCentral-JBCache"
}
maven("https://cache-redirector.jetbrains.com/plugins.gradle.org/m2") {
name = "GradlePluginPortal-JBCache"
}
}
includeBuild("../build-settings-logic")
}

dependencyResolutionManagement {
@Suppress("UnstableApiUsage")
repositories {
mavenCentral()
google()
gradlePluginPortal()
maven("https://cache-redirector.jetbrains.com/repo.maven.apache.org/maven2") {
name = "MavenCentral-JBCache"
}
maven("https://cache-redirector.jetbrains.com/dl.google.com.android.maven2") {
name = "Google-JBCache"
}
maven("https://cache-redirector.jetbrains.com/plugins.gradle.org/m2") {
name = "GradlePluginPortal-JBCache"
}
}

versionCatalogs {
Expand Down
16 changes: 12 additions & 4 deletions build-settings-logic/settings.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,25 @@ rootProject.name = "build-settings-logic"

pluginManagement {
repositories {
mavenCentral()
gradlePluginPortal()
maven("https://cache-redirector.jetbrains.com/repo.maven.apache.org/maven2") {
name = "MavenCentral-JBCache"
}
maven("https://cache-redirector.jetbrains.com/plugins.gradle.org/m2") {
name = "GradlePluginPortal-JBCache"
}
}
}

@Suppress("UnstableApiUsage")
dependencyResolutionManagement {
repositoriesMode = PREFER_SETTINGS
repositories {
mavenCentral()
gradlePluginPortal()
maven("https://cache-redirector.jetbrains.com/repo.maven.apache.org/maven2") {
name = "MavenCentral-JBCache"
}
maven("https://cache-redirector.jetbrains.com/plugins.gradle.org/m2") {
name = "GradlePluginPortal-JBCache"
}
}

versionCatalogs {
Expand Down
63 changes: 43 additions & 20 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -11,35 +11,43 @@ val gradlePluginIncludedBuilds = listOf("runner-gradle-plugin-classic")

addDependencyOnSameTasksOfIncludedBuilds("assemble", "build", "clean", "check")

registerParentGroupTasks("publishing", taskNames = listOf(
"publishAllPublicationsToMavenCentralRepository",
"publishAllPublicationsToProjectLocalRepository",
"publishAllPublicationsToSnapshotRepository",
"publishAllPublicationsToSpaceDevRepository",
"publishAllPublicationsToSpaceTestRepository",
"publishToMavenLocal"
)) {
registerParentGroupTasks(
"publishing", taskNames = listOf(
"publishAllPublicationsToMavenCentralRepository",
"publishAllPublicationsToProjectLocalRepository",
"publishAllPublicationsToSnapshotRepository",
"publishAllPublicationsToSpaceDevRepository",
"publishAllPublicationsToSpaceTestRepository",
"publishToMavenLocal"
)
) {
it.name in publishedIncludedBuilds
}

registerParentGroupTasks("gradle plugin", taskNames = listOf(
"publishPlugins",
"validatePlugins"
)) {
registerParentGroupTasks(
"gradle plugin", taskNames = listOf(
"publishPlugins",
"validatePlugins"
)
) {
it.name in gradlePluginIncludedBuilds
}

registerParentGroupTasks("bcv", taskNames = listOf(
"apiDump",
"apiCheck",
"apiBuild"
)) {
registerParentGroupTasks(
"bcv", taskNames = listOf(
"apiDump",
"apiCheck",
"apiBuild"
)
) {
it.name in publishedIncludedBuilds
}

registerParentGroupTasks("verification", taskNames = listOf(
"test"
))
registerParentGroupTasks(
"verification", taskNames = listOf(
"test"
)
)

tasks.register("integrationTest") {
group = "verification"
Expand Down Expand Up @@ -82,3 +90,18 @@ fun includedBuildTasks(taskName: String, filter: (IncludedBuild) -> Boolean = {
.filter { it.name != "build-logic" }
.filter(filter)
.mapNotNull { it.task(":$taskName") }


tasks.wrapper {
doLast {
// Manually update the distribution URL to use cache-redirector.
// (Workaround for https://github.com/gradle/gradle/issues/17515)
propertiesFile.writeText(
propertiesFile.readText()
.replace(
"https\\://services.gradle.org/",
"https\\://cache-redirector.jetbrains.com/services.gradle.org/",
)
)
}
}
Loading
Loading