Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Versions Catalogs gradle/libs.versions.toml #509

Merged
merged 106 commits into from Aug 10, 2022
Merged

Conversation

jmfayard
Copy link
Member

@jmfayard jmfayard commented Feb 25, 2022

Hello & bonjour,

What?

Resolves #333

Add support for Versions Catalogs gradle/libs.versions.toml in all our tasks (refreshVersions, refreshVersionsMigrate, refreshVersionsCleanup, refreshVersionsCatalog)

Why?

First class support for Gradle versions catalogs, removing fear of plugin lock-in for our users.

Testing?

You can test it in your own project, because I published a snapshot version on Sonatype

Setup refreshVersions "0.41.0-SNAPSHOT" as in https://jmfayard.github.io/refreshVersions/setup/#if-you-want-to-use-a-snapshot-version

Then:

# Versions Catalog are stable in Gradle 7.4
$ ./gradlew wrapper --gradle-version 7.4

# Generate the versions catalog
$ ./gradlew -refresh-dependencies refreshVersionsCatalog --versions --all
        modified:   versions.properties
        modified:   gradle/libs.versions.toml

# Migrate your build.gradle(.kts)
$ ./gradlew refreshVersionsMigrate --toml
        modified:   /Users/jmfayard/tignum/backend-tignum-x/build.gradle.kts

# Show dependency updates in gradle/libs.versions.toml
$ ./gradlew refreshVersions
        modified:   versions.properties
        modified:   gradle/libs.versions.toml

How?

  • ./gradlew refreshVersionsCatalog generate gradle/libs.versions.toml
    • only for Gradle >= 7.4
    • with placeholder version "_"
    • with hardcoded version when --versions option is used
    • only for dependencies without a built-in dependency notation
    • except if option --all is used
    • adds plugins in the [plugins] section
    • should be incremental: don't overwrite an existing entry
    • should support refreshVersions dependency rules, create a versions entry and use version.ref for the dependency
    • the versions section should be on top
    • should support configuration cache Add support for Versions Catalogs gradle/libs.versions.toml #509 (comment)
  • ./gradlew refreshVersionsMigrate
    • should support dependencies from gradle/libs.versions.toml
    • option --toml should migrate to dependencies from libs.versions.toml before built in dependencies
    • don't migrate build.gradle files inside a resources or build directory
  • ./gradlew refreshVersions
    • should ignore entries from the versions catalog using placeholder version "_"
    • should update entries with a hardcoded version
    • should update entries using versions.ref = "xxx"
    • should be idempotent
    • add unit tests
    • should not warn of hardcoded dependency if it comes from gradle/libs.versions.toml
    • supports custom `projet.buildFileName=xxx", see Add support for Versions Catalogs gradle/libs.versions.toml #509 (comment)
  • ./gradlew refreshVersionsCleanup should remove comments in gradle/libs.versions.toml
  • Update sample-kotlin
  • Documentation website should be updated
  • Publish a version "0.41.0-SNAPSHOT"
  • Ask for feedback on the issue and on Slack
  • Write blog post about it
  • @LouisCad to review it when it's not draft anymore

If your project is using Gradle 7.0, you can use instead [Gradle versions catalog](https://docs.gradle.org/current/userguide/platforms.html)

```bash
$ ./gradlew refreshVersionsCatalog

> Task :refreshVersionsCatalog
        new file:   gradle/libs.versions.toml
        modified:   settings.gradle.kts
        modified:   versions.properties
```

The generated versions catalog looks like this:

```toml
## Generated by $ ./gradlew refreshVersionsCatalog

[libraries]

browser = "androidx.browser:browser:_"

cardview = "androidx.cardview:cardview:_"

okhttp = "com.squareup.okhttp3:okhttp:_"

junit = "junit:junit:_"
```

Like with buildSrcLibs, you get type-safe accessors available in the IDE:

<img width="949" alt="refreshVersions_–_build_gradle_kts__sample-kotlin_" src="https://user-images.githubusercontent.com/459464/128611606-4ce90b01-475d-402a-b223-1757a1c51deb.png">
# Conflicts:
#	sample-kotlin/build.gradle.kts
#	sample-kotlin/gradle/wrapper/gradle-wrapper.properties
#	sample-kotlin/versions.properties
@yogurtearl
Copy link

Cool! Tried out the snapshot. :)

it inlines all the versions into [libraries], but also leaves the old versions unmodified under [versions]

[versions]
slf4j = "1.7.30"
log4j2 = "2.17.1"
kotlinLogging = "2.0.10"

[libraries]
slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }
slf4j-log4j12 = { module = "org.apache.logging.log4j:log4j-slf4j-impl", version.ref = "log4j2" }
log4j2 = { module = "org.apache.logging.log4j:log4j-core", version.ref = "log4j2" }
kotlin-logging = { module = "io.github.microutils:kotlin-logging-jvm", version.ref = "kotlinLogging" }

[bundles]
kotlinSlf4jLog4j2 = ["slf4j-api", "slf4j-log4j12", "log4j2", "kotlin-logging"]

if I run ./gradlew refreshVersionsCatalog --versions --all --no-configuration-cache

it outputs something like this:

[versions]
slf4j = "1.7.29"
log4j2 = "2.17.1"
kotlinLogging = "2.0.9"

[libraries]

kotlin-logging-jvm = "io.github.microutils:kotlin-logging-jvm:2.0.10"

log4j-core = "org.apache.logging.log4j:log4j-core:2.17.1"

log4j-slf4j-impl = "org.apache.logging.log4j:log4j-slf4j-impl:2.17.1"

slf4j-api = "org.slf4j:slf4j-api:1.7.30"

[bundles]
kotlinSlf4jLog4j2 = ["slf4j-api", "slf4j-log4j12", "log4j2", "kotlin-logging"]

Now we have duplicated versions and the kotlinSlf4jLog4j2 bundle reference a slf4j-log4j12 library that no longer exists.
Also, the next build fails with:

* What went wrong:
org.gradle.api.InvalidUserDataException: Invalid catalog definition:
  - Problem: In version catalog libs, a bundle with name 'kotlinSlf4jLog4j2' declares a dependency on 'slf4j.log4j12' which doesn't exist.
    
    Reason: Bundles can only contain references to existing library aliases.
    
    Possible solutions:
      1. Make sure that the library alias 'slf4j.log4j12' is declared.
      2. Remove 'slf4j.log4j12' from bundle 'kotlinSlf4jLog4j2'.
    
    Please refer to https://docs.gradle.org/7.4/userguide/version_catalog_problems.html#undefined_alias_reference for more details about this problem.

Maybe it should remove the [versions] section and update the bundles with the new names?

@yogurtearl
Copy link

with something like this:

[versions]
kotlin = "1.6.0"

[plugins]
kotlin-gradle = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }

it seems to just ignore the [plugins] block.

@yogurtearl
Copy link

Also, it doesn't seem to work nicely with Configuration cache.
https://docs.gradle.org/7.4/userguide/configuration_cache.html#config_cache:requirements:use_project_during_execution

invocation of Task.project📋 at execution time is unsupported.[ ?](https://docs.gradle.org/7.4/userguide/configuration_cache.html#config_cache:requirements:use_project_during_execution)
⌄task:refreshVersionsCatalog📋 of type de.fayard.refreshVersions.RefreshVersionsCatalogTask📋
⌄exception stack trace 📋
org.gradle.api.InvalidUserCodeException: Invocation of 'Task.project' by task ':refreshVersionsCatalog' at execution time is unsupported.
	at org.gradle.configurationcache.initialization.DefaultConfigurationCacheProblemsListener.onTaskExecutionAccessProblem(ConfigurationCacheProblemsListener.kt:74)
	at org.gradle.configurationcache.initialization.DefaultConfigurationCacheProblemsListener.onProjectAccess(ConfigurationCacheProblemsListener.kt:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:464)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:446)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:61)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:434)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast.dispatch(DefaultListenerManager.java:221)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast.dispatch(DefaultListenerManager.java:192)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy170.onProjectAccess(Unknown Source)
	at org.gradle.api.internal.AbstractTask.notifyProjectAccess(AbstractTask.java:1053)
	at org.gradle.api.internal.AbstractTask.getProject(AbstractTask.java:237)
	at org.gradle.api.DefaultTask.getProject(DefaultTask.java:59)
	at de.fayard.refreshVersions.RefreshVersionsCatalogTask.refreshVersionsCatalogAction(RefreshVersionsCatalogTask.kt:46)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)

@yogurtearl
Copy link

If I have something like this, it seems to just remove these libraries altogether.
These are only used in my buildSrc dir.

[libraries]
kotlin-gradle = { module = "org.jetbrains.kotlin:kotlin-gradle-plugin", version.ref = "kotlin" }
detekt-gradle = { module = "io.gitlab.arturbosch.detekt:detekt-gradle-plugin", version.ref = "detekt" }

@yogurtearl
Copy link

I have this in buildSrc/settings.gradle.kts to enable using the same version catalog in buildSrc and in my build:

dependencyResolutionManagement {
    versionCatalogs {
        create("libs") {
            from(files("../gradle/libs.versions.toml"))
        }
    }
}

and I use some of the [libraries] in buildSrc/build.gradle.kts

after I run
./gradlew refreshVersionsCatalog --versions --all --no-configuration-cache

when I run:
./gradlew refreshVersionsMigrate --no-configuration-cache

if fails because buildSrc/build.gradle.kts is referencing the old library names.

@jmfayard
Copy link
Member Author

jmfayard commented Feb 26, 2022

@yogurtearl thanks a lot for trying out the snapshot!

I initially thought the task ./gradlew refreshVersionsCatalog as a one-off thing that you use when you don't have versions catalog and then never use again. So yes, currently the libraries section is rewriting from scratch and the other are ignored. Which indeed create a build error next time if you already use the versions catalog. I will improve this.

Thanks for your reports on the configuration cache, I will have a look.

@jmfayard jmfayard changed the base branch from refreshVersionsCatalog to main February 26, 2022 17:42
Now, we are testing with realistic data when it
comes to the versions retrieved from the repos,
and this will help ensure we are going in a code
path that does the proper filtering.

Also, we are now testing for the migration into
the version catalog of dependencies that don't
have a version specified (e.g. because specified
by a BoM).

Yes, the tests are currently failing.
The freshly introduced overload will be
testable without having to perform network calls,
which means we will be able to use static, and
fake data.

Most importantly, the new overload doesn't
depend on global mutable state, but only from
the arguments it has been passed.
We no longer allocate on each call.

This commit also removes the unused parameter of
the hasHardcodedVersion function which always had
the default value of emptySet().
@LouisCAD
Copy link
Member

LouisCAD commented Aug 8, 2022

Quick update:

I am a bit late on the ETA I provided, but we're almost done. Here's the proof that we're making progress:
IMAGE 2022-08-08 12:12:51

As you can see, we're now filtering the available versions properly, and we support usage of versions.properties and versions catalog together.

Note that we currently don't support non-default catalogs yet.

Two things left blocking this PR:

  1. Migration of version-less dependencies
  2. Disabling the feature by default while we continue testing remaining edge cases thoroughly

Just one line to fix it, unbelievable!
This avoids unnecessary calculations
The problem was that the RefreshVersionsExtension was
directly touching RefreshVersionsConfigHolder, whose instance is
reused by Gradle across runs.

Now, the rejectVersionIf predicate defaults to null, is taken
each time the plugin is applied to be set on the
RefreshVersionsConfigHolder, so if it's been removed, it's set back
to null as it should.
In case someone relies on the reset behavior later (though current code
doesn't), the behavior will be correct and will avoid potential memory
leaks.
@LouisCAD
Copy link
Member

LouisCAD commented Aug 9, 2022

I did further testing, and after my latest commits, it doesn't seem to introduce regressions, so I think we will keep the flag enabled by default.

…introduce test to ensure it's never empty, and properly formatted.
This optimization will make upgrading refreshVersions slightly faster
when there's no potential dependency notations that have been removed
since last run.
@LouisCAD
Copy link
Member

LouisCAD commented Aug 9, 2022

I don't know why the CI is failing, since it's giving 404 Not Found errors when I try to look…

I reported the issue to GitHub support.

On my machine the check task runs fine.

I was ready to merge this.

@LouisCAD
Copy link
Member

GitHub Actions got fixed, and I added the missing opt-in that is not required as the Gradle 7.5.1 update updated Kotlin.

Now is time to merge!

@LouisCAD LouisCAD merged commit 0f0d63f into main Aug 10, 2022
Priorities 📝 automation moved this from Louis review priority list to Done Aug 10, 2022
@LouisCAD LouisCAD deleted the update-toml branch August 10, 2022 03:04
@imashnake0
Copy link
Contributor

pog

@LouisCAD LouisCAD restored the update-toml branch August 11, 2022 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
10 participants