Introduce a settings DSL for central dependency declaration

[melix]

This commit introduces a new DSL, available on Settings via the
dependency resolution management block, which allows declaring aliases
for dependencies and bundles of dependencies.

dependencyResolutionManagement {
    dependenciesModel {
        alias("groovy", "org.codehaus.groovy", "groovy") {
            strictly("3.0.2")
        }
    }
}

Doing this, Gradle will automatically generate type-safe accessors
which are exposed as extensions to all projects. This effectively
allows sharing dependency declarations between projects.

Said differently, this is an officially supported pattern for the
various "dependency version sharing" patterns that we found in the
wild:

• declaring dependency versions in gradle.properties, then referencing
  the version via project.someLibraryVersion
• creating a new class in buildSrc which contains dependency coordinates
  and then can be accessed in the different projects
• creating extra properties for dependency versions at the root project
• using external plugins like the "refresh dependencies" plugin
• ...

In addition to this DSL, we introduce a TOML file, which, if found under
the gradle directory, will be used to source dependency aliases.

For example, giving the following file:

[dependencies]
guava = { group = "com.google.guava", name = "guava", version = "27.0-jre" }
groovy = { group = "org.codehaus.groovy", name = "groovy", version.strictly = "3.0.2" }

a dependency can be added in a project using the type-safe libs extension:

dependencies {
    api(libs.guava)
}

This file also supports bundles of dependencies, in case more than one
dependency needs to be added:

[bundles]
groovy = ["groovy", "groovy-json"]

then dependencies can be added via:

dependencies {
   implementation(libs.groovyBundle)
}

The file is configuration-cache safe: any change to the file will invalidate the
cache. But more importantly, it's build cache safe: if the aliases don't change,
there's no need to rebuild the dependent scripts. This means that changing this
file by changing, for example, the dependency versions, will not trigger a
recompilation of build scripts like some of the approaches described above.

The TOML file also lets you declare plugin versions:

[plugins]
my.awesome.plugin="1.0.2"

which allows application of the plugin without version in build scripts:

plugins {
   id 'my.awesome.plugin'
}

In addition, we also generate type-safe accessors for projects. For example:

dependencies {
    api(project(":commons:core"))
}

can be replaced with:

dependencies {
    api(projects.commons.core)
}

Therefore any change to project coordinates, or removals of subprojects would
trigger a build script compilation error, avoiding tedious search and replace.

Fixes #?

Context

Contributor Checklist

• Review Contribution Guidelines
• Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
• Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
• Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
• Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
• Provide unit tests (under <subproject>/src/test) to verify logic
• Update User Guide, DSL Reference, and Javadoc for public-facing changes
• Ensure that tests pass locally: ./gradlew <changed-subproject>:check

Gradle Core Team Checklist

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation
• Recognize contributor in release notes

[vlsi]

In addition, we also generate type-safe accessors for projects. For example:

That is awesome.

In addition to this DSL, we introduce a TOML file, which, if found under the gradle directory

Do you think it could support overriding the version with a command-line flag?

What is the approach for file splitting?
What is the approach for plugin versions?

[melix]

Do you think it could support overriding the version with a command-line flag?

Technically speaking we can. Now I'd have to talk to the larger team to say if we should :) Note that if you override a dependency version, it would probably only override the default model, so if you have something like:

dependencies {
    implementation(libs.guava) {
        version {
           // this wins over whatever is overridden
        }
    }
}

What is the approach for file splitting?

For now I'd say KISS. We could provide alternatives if there's a legitimate interest in making this more complicated. Alternatively settings plugins can provide this feature. Last but not least, because we do code generation we could hit the limit of the number of methods in a class...

What is the approach for plugin versions?

This PR lets you declare this:

[plugins]
my.awesome.plugin="1.4"

then you don't have to specify a version when applying in build scripts.

[vlsi]

Technically speaking we can. Now I'd have to talk to the larger team to say if we should :)

Let me clarify the use-case.

I want to verify if my project works with a newer version of a third-party dependency.
For instance, I might need to verify if a newer Checkstyle version works for my project, or a newer Guava works for me.
The important point is I don't want to modify the source files because, well, editing the source files is prone to errors.

Note: I can't use composite builds for that, because:

1. Third-party is not always Gradle based (yet :'( )
2. composite build differs from the case when dependency is fetched from Maven repository. For instance, composite build ignores Gradle Metatada, so the only way to verify if Gradle Metatada is OK is to install it to a repository (e.g. ~/.m2) and build against it.

Here's the case where I install Apache Calcite Avatica to the local maven repository and build Apache Calcite with that dependency: https://github.com/apache/calcite/blob/cd922deff8ae3f25546b1d77fb147c3098eb177b/.github/workflows/main.yml#L88-L103
I use a hand-crafted version like 1.0.0-dev-master-SNAPSHOT to ensure it does not accidentally fetch an old snapshot from somewhere.

Here's the case when Checkstyle CI job verifies the new Checkstyle works for verifying of pgjdbc sources: https://github.com/checkstyle/checkstyle/blob/3c2249b239cbfe002af3649f1f7c1f9ae61df3b1/.ci/wercker.sh#L60-L71

[vlsi]

Alternatively settings plugins can provide this feature

I wonder if TOML could be separate from the code generator, so out-of-core plugins could leverage the same generator.

[melix]

I wonder if TOML could be separate from the code generator, so out-of-core plugins could leverage the same generator.

That's exactly the case: the TOML file is just, if you will, a standard plugin hooking into the code generator. In the end, there's a single code generator and settings plugins can contribute entries. But in the end we're going to be limited by the number of methods which can be defined in a class file.

[melix]

Let me clarify the use-case.

That is certainly a legitimate use case.

[martinbonnin]

That looks seriously awesome 🤩 . Thanks for addressing this !!

Will the TOML file support dependencies "groups" that need to be all the same version so that it can be changed in a single place? For an example (that obviously won't work since I don't think TOML supports variables) but just to demonstrate the goal:

[versions]
sqldelight = "1.7.0"

[dependencies]
sqlDelightPlugin = { group = "com.squareup.sqldelight", name = "gradle-plugin", version = "${versions.sqldelight}" }
sqlDelightDriverAndroid = { group = "com.squareup.sqldelight", name = "android-driver", version.strictly = "${versions.sqldelight}" }
sqlDelightDriverAndroid = { group = "com.squareup.sqldelight", name = "native-driver", version.strictly = "${versions.sqldelight}" }

Also would it be possible to copy/paste the maven coordinates in one go? That's a minor change but not having to separate group/artifactId/version would be a nice addition as most of the documentations use that representation:

[dependencies]
guava = { gav = "com.google.guava:guava:27.0-jre" }

Which leads me to a wider question: did you consider a Kotlin script for this that would expose a strongly typed way to register dependencies and could be self documented? Something like dependencies.gradle.kts:

dependencies {
  create("guava") {
    group = "com.google.guava"
    artifact = "guava"
    version= "27.0-jre"
  }
  // or the shorthand gav version
  create("guava") {
    gav = "com.google.guava:guava:27.0-jre"
  }
}

bundles {
  create("groovy") {
    from("groovy", "groovy-json")
  }
}

Could that work?

[melix]

Will the TOML file support dependencies "groups" that need to be all the same version so that it can be changed in a single place?

Not out of the box. If we do this I'd rather add a [versions] section and have the other modules reference it via a versionRef="someId" rather than inventing a substitution mechanism.

Also would it be possible to copy/paste the maven coordinates in one go? That's a minor change but not having to separate group/artifactId/version would be a nice addition as most of the documentations use that representation:

That's already supported by this PR :)

Which leads me to a wider question: did you consider a Kotlin script for this that would expose a strongly typed way to register dependencies and could be self documented?

There's already the settings DSL for this. Note, however, that using Kotlin or the settings DSL would invalidate all build scripts classpath and trigger recompilation of all scripts if any version or coordinate changes, whereas using the TOML this would only happen if you add/remove an alias, not if you change GAV coordinates.

[martinbonnin]

Not out of the box. If we do this I'd rather add a [versions] section and have the other modules reference it via a versionRef="someId" rather than inventing a substitution mechanism.

Yup, makes sense 👍

That's already supported by this PR :)

Neat ! \o/

There's already the settings DSL for this.

Cool! Is there an exemple somewhere?

using Kotlin or the settings DSL would invalidate all build scripts classpath and trigger recompilation of all scripts if any version or coordinate changes

That's a pretty big downside and certainly one of the main reason not a lot of people use this? (I think?)

[melix]

Cool! Is there an exemple somewhere?

It's in the issue description ;)

[martinbonnin]

It's in the issue description ;)

Got it, Thanks 👍 I was looking in the current production docs 🤦 . What about moving that DSL to a separate dependencies.gradle.kts file that wouldn't trigger recompilation of all scripts when changed?

[vlsi]

melix: using Kotlin or the settings DSL would invalidate all build scripts classpath and trigger recompilation of all scripts if any version or coordinate changes

AFAIK settings.gradle.kts does not serve as a classpath for other scripts, so why recompile?

I just checked a couple of modifications to settings.gradle.kts, and file edits do not seem to trigger the recompilation of other build.gradle.kts scripts.

I tried to edit a string literal inside settings.gradle.kts, and I tried to declare a function fun hello() = "42". The changes went OK, and only settings compilation was necessary.

[melix]

no you're right, using the settings API this would be fine, I was mistaken with the current buildSrc approach.

[melix]

I was looking in the current production docs

There are no docs because we didn't decide to go with this proposal yet.

[martinbonnin]

This PR got me thinking... Would it make sense for the TOML file to be generic and not only about dependencies? A kind of supercharged type-safe gradle.properties? As far as I can tell, people started using buildSrc and such other solutions mostly for type safety and autocomplete so I can see them switching to something like:

# properties.toml
[libs]
guava = "com.google.guava:guava:27.0-jre"

// build.gradle.kts
dependencies {
   // can't think of a better name than `typesafeProperties` but something shorter would be nice
   implementation(typesafeProperties.libs.guava)
}

Of course this is less expressive than having the TOML file know about dependencies but I think specifying the versions constraints, etc can be left to a platform project?

Having a generic toml format would allow for other properties to be used in a type safe manner. For an exemple for publishing:

# properties.toml
[publishing]
url = "https://github.com/me/my-library"
group = "com.example"
version = "1.0"

// build.gradle.kts
group = typesafeProperties.publishing.group
version = typesafeProperties.publishing.version
publishing {
    publications {
        create<MavenPublication>("mavenJava") {
            pom {
                name.set("My Library")
                description.set("A concise description of my library")
                url.set(version = typesafeProperties.publishing.url)
            }
        }
    }
}

This could also be used to share properties between multiple projects, inject credentials, or whatever people use gradle.properties today? I understand that deviates quite a bunch from the original intent of the PR but that could maybe address most of the use cases in a more generic way? Any thoughts?

[melix]

@martinbonnin I don't think we should do this. The big advantage of this DSL and TOML file is that we know what the properties are used for. It allows for some optimizations and it allows better modeling. We can discuss its extension to other use cases but we shouldn't mix all things together IMO.

[melix]

@bot-gradle test this

[bot-gradle]

OK, I've already triggered ReadyForMerge build for you.

[melix]

@bot-gradle test this

[bot-gradle]

OK, I've already triggered ReadyForMerge build for you.

[melix]

I'm not sure I expressed my expectation correctly, so here a little example:

No, that's exactly about this. Currently the versions are not exposed to precompiled script plugins (or production code in buildSrc). One reason is that if we use the same mechanism, say, that you define a dependencies model in buildSrc and that you make it available in the precompiled script plugins, then a plugin could have:

jacoco {
    toolVersion = libs.jacocoVersion.get()
}

The libs that you see here is a generated class which "belongs to" buildSrc code. It has a package name and class name which are generated for the buildSrc context. However, in a precompiled script plugin execution phase, the context is the project itself.

So the libs object that you would see would at best be the same because you imported the same model in both main build and buildSrc, but it could be totally different. Worse there are some name clashes.

So sharing with production code, because of class shadowing, is a bit tricky. We can certainly use a different namespace to make it explicit, maybe use a different prefix to access the extensions, ... But I think we should take time to think about the solution and not rush here.

[tprochazka]

I really want some official solution for this dependency mess.
I was thinking that the platform module is the official way, until now.
I'm quite confused that it is completely independent and partially redundant to platform modules.
I know that the platform module is just about versions, but it can be extended to generating aliases or not?
And I'm even more confused with creating a new file format TOML, instead of a primary focus on Kotlin DSL with existing IDE support. There is DSL support but it looks weird, but maybe the example in the first post just doesn't show everything. I would expect something more similar to this TOML file.

[melix]

I was thinking that the platform module is the official way, until now.
I'm quite confused that it is completely independent and partially redundant to platform modules.

It's expected because there is an overlap between the two, and you might want to use both at the same time. A platform is a component in a dependency graph, which can be upgraded on its own (for example if you have the Spring BOM in a dependency graph, it can be upgraded to a different version because of transitives), participates into dependency resolution (constraints declared in a platform will apply even if you don't use them), is inherited by consumers (a platform is a node in the graph so consumers will see it). Anything declared in a platform will influence any graph which uses this platform, including transitive dependencies. A catalog, like proposed here, is very lean: it’s only used for first level dependencies, in order to pick versions. It has no influence on the consumer and any “unused” dependency in a catalog will not influence the resolution. The consumer of a catalog is Gradle itself, while the consumer of a platform is any library, Gradle, Maven, etc.

This proposal is about fixing the "redundant declaration" use case, which isn't really addressed today and that people implement in various ways.

I know that the platform module is just about versions, but it can be extended to generating aliases or not?

This follow-up PR actually does something like that: it lets you declare a version catalog from a platform definition. The choice of how to consume the platform, either as a platform, or as a catalog, or both, is a consumer choice.

Catalogs can also be composed from different sources easily, without "leaking" to consumers, which is one of the drawbacks of platforms today.

[tprochazka]

I started to use it and it works great, in TOML file is possible to generate dependencies in three different ways

groovy = { group = "org.codehaus.groovy", name = "groovy", version = "3.0.2" }
groovy1 = { module = "org.codehaus.groovy:groovy", version = "3.0.2" }
groovy2 = "org.codehaus.groovy:groovy:3.0.2"

The disadvantages of TOML are missing code completion and error messages which not only tell the line with error but it also not tell that this error is related to TOML file parsing until you use --stacktrace

DSL currently allows just this

dependencyResolutionManagement {
    dependenciesModel("libs") {
        alias("framework-core").to("com.company.android:android-framework-core:1.5.5")
        alias("framework-core").to("com.company.android","android-framework-core").version("1.5.5")
    }
}

There is no way how to use org.codehaus.groovy:groovy as in TOML file

If you want to create a bundle from multiple library with the same version it is possible in this way via DSL

dependencyResolutionManagement {
    dependenciesModel("libs") {
        version("lifecycle", "1.5.5")
        alias("androixLifecycleRuntime").to("androidx.lifecycle","lifecycle-runtime-ktx").versionRef("lifecycle")
        alias("androixLifecycleViewmodel").to("androidx.lifecycle","lifecycle-viewmodel-ktx").versionRef("lifecycle")
        bundle("androixLifecycle", listOf("androixLifecycleRuntime", "androixLifecycleViewmodel"))
    }
}

I would maybe expect some more structuralized way like

        bundle("androixLifecycle").version("1.5.5") {
            alias("androixLifecycleRuntime").to("androidx.lifecycle","lifecycle-runtime-ktx")
            alias("androixLifecycleViewmodel").to("androidx.lifecycle","lifecycle-viewmodel-ktx")
            alias("someDifferentLibInBundle").to("xx","yy").version("1.0.0")
        }

or just without bundle

        version("1.5.5") {
            alias("androixLifecycleRuntime").to("androidx.lifecycle","lifecycle-runtime-ktx")
            alias("androixLifecycleViewmodel").to("androidx.lifecycle","lifecycle-viewmodel-ktx")
        }

or

       bundle("basicLibs") {
           version("1.5.5") {
               alias("androixLifecycleRuntime").to("androidx.lifecycle","lifecycle-runtime-ktx")
               alias("androixLifecycleViewmodel").to("androidx.lifecycle","lifecycle-viewmodel-ktx")
           }
           alias("someDifferentLibInBundle").to("xx","yy").version("1.0.0")
       }

It is visually much more clear that it is a group that belong together.

I would prefer DSL way over TOML, but it is quite verbose and it must be inside of settings.gradle.kts file, separated will would be better.

And I miss a possibility to create a more structuralized libs hierarchy by using "." inside of alias name, which will create a new packages inside of libs object like

libs.androidx.<library>

[tprochazka]

And what about maven dependencies with classifier? It looks that it is completely unsupported :-(

[gildor]

I'm playing with new versionCatalog DSL, works great, but I found big general issue for my common use case: convention plugins.

So if I have dependencies.toml, I can use it any any module and in buildSrc (if include it), but I cannot use dependencies versions in my precompiled convention plugin: buildSrc/main/kotlin/some-convention.gradle.kts

I understand why it happening, but I think it's a big limitation in general, because I apply many dependencies using different convention plugins, so even if I able to read toml file, I will not get any generated accessor.

Should it be possible somehow to generate libs accessors also for precompiled script plugins (which already support Kotlin DSL accessors generation)?

I may create a feature request, if it makes any sense

[melix]

We have ideas how to make this possible, but probably not for the first release.

[gabrielittner]

@melix Is there an issue to follow for this?

[ljacomet]

And what about maven dependencies with classifier? It looks that it is completely unsupported :-(

The dependency catalog focuses on dependency coordinates, namely the Group, Artifact and Version.
We discussed the idea of including classifiers or even variant information and decided against it. Indeed these have a context linked to their application and thus it is better expressed in the build file rather than in the catalog.

[tprochazka]

Do you know why this doesn't work for me in gradle-6.8-milestone-3?

dependencies {
    api(projects.commons.core)
}

It was removed meantime?

[melix]

Yes, this was removed from 6.8. You can use master instead, this is still under development.

[tprochazka]

Thanks! So it will be in 6.9, right? I will keep it for now without it, I want to put into production with 6.8, 6.9 is too far.

[tprochazka]

There are some cases when it fails like this one

val ccaApi by configurations
ccaApi(libs.uiSkeleton) {
    because("Some comment why why need this dependency")
}

ends with

e: ...lib_ui_skeleton\build.gradle.kts:34:12: Type mismatch: inferred type is Provider<MinimalExternalModuleDependency!>! but String was expected

[melix]

There are some cases when it fails like this one

Can you please file issues for those cases?

[tprochazka]

It looks that in gradle-6.8-rc-1 it completely stops working :-( Or it is possible to enable it somehow in gradle.properties?
I was expecting that only project() support was removed.
@melix If you mean to report it here https://discuss.gradle.org/c/bugs/11 I have disabled New Topic button there.

[melix]

I was expecting that only propejct() support was removed.

No the whole central dependencies declaration has been removed.

And I mean creating issues with the reproducers for what you've found. This ticket is closed, there have been many follow ups and if you don't want your bug reports to be lost it's better to track them with proper issues.

[JavierSegoviaCordoba]

@melix should be great there is a place where check all info about this feature. I always visit this issue.

[tprochazka]

@melix Done. I hope that I did it in the right way. ;-)

[tprochazka]

@melix should be great there is a place where check all info about this feature. I always visit this issue.

Yes. At least slack channel for this would be great.

[tprochazka]

I just wrote a script that can generate a TOML file for a multimodule project, including versions ref extraction, and replace all old dependencies with libs.xyz in the next step. Except for the lack of IDE support, it works great. I also already implemented it on my project, because I was expected that it will be in 6.8 :-( At least as an experimental feature.

[melix]

Please refer to this issue for tracking progress: #15352

[tprochazka]

Weird that I can see, that this #15352 issue has "0 of 12 tasks complete", but I'm unable to find which tasks they are :-(

[melix]

@tprochazka Yeah I noticed that. We're using ZenHub to track epics and unfortunately you can't see anything. This is what you should see: