Dependencies with reference to module metadata but missing metadata should be seen as broken and ignored

[Vampire]

Following this issue on the Kotlin slack: https://kotlinlang.slack.com/archives/C19FD9681/p1610547344113500 there is a potential for really strange and hard to grasp problem that looks like flaky Gradle behavior, even if it is not really.

The symptom was, that someone wanted to upate io.ktor:ktor-client-cio:1.3.1 to io.ktor:ktor-client-cio:1.5.0 and suddenly the build started to break, sometimes being able to add the ktor-client-cio-jvm variant and sometimes not being able to within the same build and in subsequent builds continuously until the dependency was delete from local maven repository.

1.3.1 was published without Gradle Module Metadata, 1.5.0 adde Gradle Module Metadata with variants for different platforms.

Roughly summarized:

• have a Gradle build that has mavenLocal() as first repository, that uses something with Gradle Module Metadata (e. g. io.ktor:ktor-client-cio:1.5.0)
• have a Maven build that also uses the same dependency
• clear your local maven repository
• run the Gradle build
  • the dependency is not found in mavenLocal()
  • it is found on maven central
  • metadata is used and thus ktor-client-cio-jvm added to the dependency tree
• run the Maven build
  • Maven puts the library to the local maven repository,
  • but of course without the Gradle Module Metadata
• run the Gradle build
  • the dependency is found in mavenLocal() but without the metadata and so
  • ktor-client-cio-jvm is not going to be added to the dependency graph

Solutions on the build author side are of course to move mavenLocal() to last position (needed for integration tests with snapshot versions) or to use repository content filters to only allow snapshot versions from mavenLocal() for example.

But this results in strange behavior that is hard to diagnose if you don't know where to look at and looks like flaky Gradle behavior.

One way of solving this on Gradle side to prevent build authors from shooting their own feet would be to see that the pom or ivy file has a reference to the module metadata but find it missing, then consider the dependency as broken and try the next repository until all repositories are exhausted.

Imho to not have a situation where something could be used from Maven but not from Gradle due to this, I'd suggest to have a second pass (or remember the previous discarded candidates) and use them even with missing module metadata if none of the defined repositories has the missing metadata present. But maybe I'm not taking something into account here.

[jjohannes]

I think the solution is to not use mavenLocal() and to make sure you only use reliable repositories - i.e. repositories which do always contain the complete components with all metadata.

Maybe Gradle should print a big warning if you use mavenLocal() as a source for dependencies.

[Vampire]

In that case the local Maven repository was used for integration tests to test with snapshot versions that code from a repository as usual.

Of course it was more a misconfiguration, than a Gradle bug, but the suggested solution would mainly implement a fail-fast with a meaningful error message instead of looking like Gradle is flaky.

The could also be other bad repos, like mirrors that so not mirror all files and so on, that would also be catched by this logic.

[ljacomet]

I understand the pain this can cause but I am dubious we would find a mitigation strategy that did not have its own corner cases.
For example, Gradle leverages a marker inside POM files to know about a potential module metadata files. But I am pretty sure there are libraries out there that have the marker but decided to not publish the Gradle Module Metadata, or simply used a plugin that still does not publish this file. In such a case, the strategy you propose would effectively break the consumption of that module.

I think the best approach we could have is to warn on resolution failure from mavenLocal() whenever there is no content filtering.

[Vampire]

In such a case, the strategy you propose would effectively break the consumption of that module.

Not if the last paragraph is taken into account, using the previously discarded result if no repository contains the metadata.

I think the best approach we could have is to warn on resolution failure from mavenLocal() whenever there is no content filtering.

My approach was meant for all repos, not only maven local, but this would at least improve the situation as maven local that is filled by Maven builds is most likely to break in such a way, while maybe the "whenever there is no content filtering" should be lifted imho. :-)