withDependencies allows project dependencies to be added to the DependencySet after the task graph is ready

[DanielThomas]

One of our teams building a plugin got taken surprise when they didn't realise they'd race the task graph by using withDependencies to add project dependencies.

Expected Behavior

Project dependencies should not be able to be added to configurations after configuration time.

Current Behavior

The configuration resolves successfully but without the dependency does not have the artifacts from the dependent project. When using incremental builds, you could accidentally be using an artifact from a previous build.

Context

This feels rare and we were able to identify the problem for them pretty easily, but feels like a gotcha Gradle should avoid. May be your existing plans for configurations avoids this, but wanted to document it as a potential gotcha.

Steps to Reproduce

1. Wire two Java projects w/ an implementation project dependency added in withDependencies

configurations.implementation.withDependencies { dependencies ->
	def projectB = project.dependencies.project(path: ':b')
	dependencies.add(projectB)
}

2. Note that the configuration resolves with the dependencies task
3. Run :a:compileJava and note that :b isn't considered a dependency

Your Environment

Gradle 7.5.1

[jbartok]

Sorry that you're having trouble with Gradle!

Your issue lacks information about how to reproduce the problem you're having. A reproducer project can really help us track down and fix your problem quicker. We may also be able to suggest workarounds or ways to avoid the problem if we can reproduce it.

You can use the following as a base for your reproducer: https://github.com/gradle/gradle-issue-reproducer

[DanielThomas]

Reproducer is here https://github.com/DanielThomas/gradle-issue-21490. The workaround is to configure regular project dependencies so they exist when the task graph is created. withDependencies should fail on any attempt of mutation that would not have the desired effect.

[jjohannes]

I hit this today. I think the problem is not the project dependency, but the fact that the resolution is skipped in certain cases and that the "withDependeny" actions are not triggered then:
https://github.com/jjohannes/gradle/blob/56dce2c661311fa60efb68cb9cff945f631eac6c/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/configurations/DefaultConfiguration.java#L880

This is a "micro-optimization" in Gradle for Java compilation which skips the resolution of dependencies during task graph calculation. And instead checks the direct dependencies directly for projects to compile.

I did a bit of debug, and I think the problem is that Configuration.withDependencyActions() has to be explicitly called in this specific optimization case. There are already some other places where this is done. For example, I noticed that if you add any other dependency, it also works. This seems to be coincidental, because the Configuration.withDependencyActions() is called through DefaultLocalComponentMetadata.realizeDependencies() (which in turn is triggered through addDefinedExcludes...).

Workarounds:

1. Make sure to use included builds for other components (not only in pluginManagement). The substitutions registered by the included builds turn off the optimization - add includeBuild(".") to settings.gradle
2. (a bit stupid) add something like this to all projects:

dependencies {
    implementation(files("dummy"))
}

3. Only call Gradle through the tolling API. 😄 😢 Then it works, because then the optimization also seems to be turned off. Which is also what was driving me insane, because I sometimes started through IntelliJ and sometimes from the command line, without even thinking about it.

Fix

I think this inconsistent behavior is concerning and this should be fixed.
I think the right fix would be to call Configuration.withDependencyActions() in Configuration.resolveGraphForBuildDependenciesIfRequired() to make sure it is done even if resolve graph is not required.

[jjohannes]

Another reproducer:
https://github.com/jjohannes/understanding-gradle/tree/e6a8505c25cfbd4c15b581caf3faddaf5b3151a4/31_The_Module_Path

(Make sure to checkout the indicated commit ID - withDependencies is used in the org.gradlex.java-module-dependencies plugin applied in that sample )

[jjohannes]

This is still an issue in Gradle 8.5.