Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

Closed
lkoe opened this issue Nov 12, 2019 · 25 comments
Closed

Unable to properly resolve dynamic dependencies from mavenLocal() repo #11321

lkoe opened this issue Nov 12, 2019 · 25 comments

Comments

@lkoe
Copy link

@lkoe lkoe commented Nov 12, 2019

Gradle 6.0 fails to resolve artifacts from the mavenLocal() repository.
The artifact was published using the maven-publish plugin.
In the local maven repository metadata was published as maven-metadata-local.xml (as before).

However Gradle 6.0 seems to be exclusively looking for a file named maven-metadata.xml - which isn't there.

A problem occurred configuring root project 'common'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not find any matches for de.xxx.xxx.gradle.plugins:xxx-gradle-plugins:2.0.+ as no versions of de.xxx.xxx.gradle.plugins:geco-gradle-plugins are available.
     Searched in the following locations:
       - file:/C:/Users/xxx/.m2/repository/de/xxx/xxx/gradle/plugins/xxx-gradle-plugins/maven-metadata.xml
     Required by:
         project :

If the maven-metadata-local.xml files are renamed to maven-metadata.xml, resolution works.

Expected Behavior

The dependency should be properly resolved like with Gradle 5 and before.

Current Behavior

The dependency was not resolved from the local Maven repository.

Context

For inhouse gradle plugin development I publish the plugin locally first in order to test it with other projects.
Publishing to our artifactory repo would not be feasible as faulty plugin versions might get picked up by others.

Steps to Reproduce

Your Environment

Gradle 6.0
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
Windows 10
Build scan URL:

@jjohannes
Copy link
Contributor

@jjohannes jjohannes commented Nov 12, 2019

Thanks for reporting @lkoe. Looks like Gradle never looked for maven-metadata-local.xml but always did a fall back to listing files. So we broke this by "not looking for artifacts by default" anymore if metadata is missing.
We will work on fixing this. For now you can work around it with this:

repositories {
    mavenLocal() {
        metadataSources {
            mavenPom()
            artifact()
        }
    }
}

@Yg0R2
Copy link

@Yg0R2 Yg0R2 commented Nov 13, 2019

Hi @jjohannes ,

I wanted to mention, this behavior appears only if you use dynamic versioning.
In that case, if you define a fixed version like 2.0.1-SNAPSHOT it works properly.

@jjohannes
Copy link
Contributor

@jjohannes jjohannes commented Nov 13, 2019

Good point @Yg0R2. If only the maven-metadata.xml is "missing", but pom.xml files all exist, a repository without artifact() still works fine if no dynamic versions are used. Because then Gradle directly checks the selected versions by checking if the corresponding pom.xml exists.

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Nov 13, 2019

So doing a bit of research on this topic.

First of all, the presence of a maven-metadata-*.xml file is not guaranteed in a local repository.

  • Recent Maven versions will download the one from the remote repository, naming it maven-metadata-<repo-id>.xml.
  • A maven-metadata-local.xml only exists if you install that module from a local build.

Given all of that, my take is we should ignore the metadata sources configuration for Maven local version listing and always do file listing.

@jjohannes
Copy link
Contributor

@jjohannes jjohannes commented Nov 13, 2019

We can also restore the "old" behavior for Maven local. I.e have it default to

metadataSources {
  mavenPom()
  artifact()
}

It's probably less "special case", easier to implement, and with that less risky, as we will restore the 5.x behavior.

So something like adding this

        mavenRepository.metadataSources(new Action<MavenArtifactRepository.MetadataSources>() {
            @Override
            public void execute(MavenArtifactRepository.MetadataSources metadataSources) {
                metadataSources.mavenPom();
                metadataSources.artifact();
            }
        });

here:

public MavenArtifactRepository createMavenLocalRepository() {
MavenArtifactRepository mavenRepository = instantiator.newInstance(DefaultMavenLocalArtifactRepository.class, fileResolver, transportFactory, locallyAvailableResourceFinder, instantiatorFactory, artifactFileStore, pomParser, metadataParser, createAuthenticationContainer(), fileResourceRepository, mavenMetadataFactory, isolatableFactory, objectFactory, urlArtifactRepositoryFactory);
File localMavenRepository = localMavenRepositoryLocator.getLocalMavenRepository();
mavenRepository.setUrl(localMavenRepository);
return mavenRepository;

@melix
Copy link
Contributor

@melix melix commented Nov 13, 2019

OR, the 3rd way. Currently "listing" and "metadata sources" have been bound together, but we could decide to have a different kind of lister for Maven local, which doesn't care about the presence of artifact() or not in metadata sources.

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Nov 13, 2019

That's the current idea I am looking at, have a specialized MavenVersionLister for Maven local which would not even attempt a maven-metadata.xml but directly do the basic version listing.

Effectively changing the configuration is a simpler change, but it also makes it harder to explain the difference. And finally, the current MavenVersionLister is just useless code for Maven local, as it will never find a file.

ljacomet added a commit that referenced this issue Nov 14, 2019
There is no way to rely on a `maven-metadata.xml` for Maven local
version listing.
Since Gradle 6.0 removed the default `artifact()` metadata source, this
causes all dynamic version resolution to fail with Maven local.
This commit changes the way Maven local is handled to _always_ do
version listing through directory listing.

Fixes #11321
ljacomet added a commit that referenced this issue Nov 14, 2019
ljacomet added a commit that referenced this issue Nov 14, 2019
ljacomet added a commit that referenced this issue Nov 15, 2019
ljacomet added a commit that referenced this issue Nov 15, 2019
@jjohannes
Copy link
Contributor

@jjohannes jjohannes commented Nov 15, 2019

@lkoe we have fixed the issue for a 6.0.1 patch release. Here is a nightly containing the fix:
https://services.gradle.org/distributions-snapshots/gradle-6.0.1-20191115103811+0000-bin.zip (link updated)
It would be very much appreciated if you can acknowledge that it works for you.

@lkoe
Copy link
Author

@lkoe lkoe commented Nov 15, 2019

Thanks @jjohannes, unfortunately due to a business trip I can only manage to test next wednesday. Hope that's still inside the 6.0.1 timeframe...

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Nov 15, 2019

There was a copy/paste mistake in the nightly version. Please use 6.0.1-20191115103811+0000 as the version, or https\://services.gradle.org/distributions-snapshots/gradle-6.0.1-20191115103811+0000-bin.zip for the gradle-wrapper.properties

We hope to have 6.0.1 out before next wednesday, but no worry if you cannot validate before then. We will have to trust our own testing 😉

@ljacomet ljacomet changed the title Gradle 6.0 does not properly resolve dependencies from mavenLocal() repo Gradle 6.0 does not properly resolve dynamic dependencies from mavenLocal() repo Nov 15, 2019
@big-guy big-guy changed the title Gradle 6.0 does not properly resolve dynamic dependencies from mavenLocal() repo Unable to properly resolve dynamic dependencies from mavenLocal() repo Nov 18, 2019
@lkoe
Copy link
Author

@lkoe lkoe commented Nov 20, 2019

Confirmed, works for me with 6.0.1. Thx!

@kristian

This comment has been minimized.

@kristian
Copy link
Contributor

@kristian kristian commented Dec 1, 2019

Sorry just found out, "dynamic plugin versions are not supported". Thanks anyways!

@lkoe
Copy link
Author

@lkoe lkoe commented Dec 1, 2019

We're using a dynamic version as well for our inhouse plugins. This worked fine in the past and - with this issue fixed - also with 6.0.1

@kristian
Copy link
Contributor

@kristian kristian commented Dec 1, 2019

We're using a dynamic version as well for our inhouse plugins. This worked fine in the past and - with this issue fixed - also with 6.0.1

Hey @lkoe, interesting! Can you tell me how you did it? Maybe just to clarify, I was speaking about referencing the plugin versions dynamically like:

plugins {
    id 'anyplugin' version '0.+'
}

Not referencing libraries used by the plugin dynamically using a dependency block.

@lkoe
Copy link
Author

@lkoe lkoe commented Dec 1, 2019

We're defining our plugin as dependency in the buildscript block in ´init.gradle´

allprojects {
    buildscript {

        repositories {
            mavenLocal()
            maven {
                url System.properties["artifactoryUrl"] + "libs/"
                credentials {
                    username = System.properties["artifactoryUser"]
                    password = System.properties["artifactoryPassword"]
                }
            }
        }

        def pluginVersion = (gradle.gradleVersion[0] as int) > 4 ? '2.0.+' : '1.0.+'
         
        dependencies {
            classpath "xxx.xxx.xxx.gradle.plugins:xxx-gradle-plugins:$pluginVersion"
        }
    }
}

In the projects we then just apply our plugins without version.

@kristian
Copy link
Contributor

@kristian kristian commented Dec 1, 2019

Thanks so much @lkoe and sorry for spamming this issue with a "help request". I didn't came to the idea of using the legacy buildscript block, but it worked flawelessly! By adding the buildscript block to the settings.gradle, we not even had to use the allprojects closure block! Thanks again Lars!

@mkhludnev
Copy link

@mkhludnev mkhludnev commented Dec 4, 2019

`

Could not resolve all files for configuration ':int-test:testCompileClasspath'.
Could not find org.apache.solr:solr-test-framework:8.4.0-SNAPSHOT.
Searched in the following locations:
- file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/maven-metadata.xml
- file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/solr-test-framework-8.4.0-SNAPSHOT.pom
- https://jcenter.bintray.com/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/maven-metadata.xml
- https://jcenter.bintray.com/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT/solr-test-framework-8.4.0-SNAPSHOT.pom
Required by:
project :int-test

..
See https://docs.gradle.org/6.0.1-20191115103811+0000/userguide/command_line_interface.html#sec:command_line_warnings
`
I have maven-metadata-local.xml at that location, also notice gradle version int the url.

how I can resolve it?

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Dec 5, 2019

6.0.1 has been released. Can you try with that version?

If that does not work, can you share:

  • The way you declare the dependency
  • the content of file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework and file:/C:/Users/noob/.m2/repository/org/apache/solr/solr-test-framework/8.4.0-SNAPSHOT

@mkhludnev
Copy link

@mkhludnev mkhludnev commented Dec 5, 2019

@ljacomet thanks for your response. Originally SNAPSHOTs were installed by Apache Solr ant script. Turns out, after I manually invoked $mvn install-file for all snapshot artefacts gradle successfully find local dependencies. I also noticed that mvn insall-file put some _remote.repositories which is starts from line: #NOTE: This is an Aether internal implementation file,. So, my issue is about outdated install routine, and not worth to worry about.
fwiw: there's a movement for migrating Solr to gradle.

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Dec 5, 2019

Good to know the issue is solved.
And of course it is exciting to learn about a project considering Gradle!

@henryju
Copy link
Contributor

@henryju henryju commented May 11, 2021

Ithink I'm still affected by this issue. Let me know if you prefer that I create a separate issue.

./gradlew --version

------------------------------------------------------------
Gradle 6.8.2
------------------------------------------------------------

Build time:   2021-02-05 12:53:00 UTC
Revision:     b9bd4a5c6026ac52f690eaf2829ee26563cad426

Kotlin:       1.4.20
Groovy:       2.5.12
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          11.0.11 (Red Hat, Inc. 11.0.11+9)
OS:           Linux 5.11.18-300.fc34.x86_64 amd64

This is when using custom configurations:

allprojects {
  repositories {
        mavenLocal()
  }
}
configurations {
    create("sqplugins") { isTransitive = false }
}
dependencies {
    "sqplugins"("org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT@jar")
}
tasks.prepareSandbox {
    doLast {
        copy {
            from(project.configurations.get("sqplugins"))
            into(file("$destinationDir/$pluginName/plugins"))
        }
    }
}

Running the build fails to find the dependency:

./gradlew buildPlugin
> Task :prepareSandbox FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':prepareSandbox'.
> Could not resolve all files for configuration ':sqplugins'.
   > Could not find org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT.
     Searched in the following locations:
       - file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/maven-metadata.xml
       - file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/sonar-csharp-plugin-8.23-SNAPSHOT.pom
       - ...

but the SNAPSHOT is there:

ll -R ~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/
~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/:
total 8
drwxrwxr-x. 2 julien julien 4096 11 mai   15:45 8.23-SNAPSHOT
-rw-rw-r--. 1 julien julien  300 11 mai   16:30 maven-metadata-local.xml

~/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT:
total 6812
-rw-rw-r--. 1 julien julien     928 11 mai   16:30 maven-metadata-local.xml
-rw-rw-r--. 1 julien julien     269 11 mai   16:30 _remote.repositories
-rw-rw-r--. 1 julien julien 5163463 11 mai   16:30 sonar-csharp-plugin-8.23-SNAPSHOT.jar
-rw-rw-r--. 1 julien julien   15923 11 mai   14:09 sonar-csharp-plugin-8.23-SNAPSHOT.pom
-rw-rw-r--. 1 julien julien 1785415 11 mai   16:30 sonar-csharp-plugin-8.23-SNAPSHOT-sources.jar

The workaround worked:

        mavenLocal() {
            metadataSources {
                mavenPom()
                artifact()
            }
        }

I can't understand why the log says that the pom (file:/home/julien/.m2/repository/org/sonarsource/dotnet/sonar-csharp-plugin/8.23-SNAPSHOT/sonar-csharp-plugin-8.23-SNAPSHOT.pom) is not there, while it obviously is.

@devmattrick
Copy link

@devmattrick devmattrick commented Jun 17, 2021

I'm in the same boat as @henryju. My gradle info:

------------------------------------------------------------
Gradle 7.0
------------------------------------------------------------

Build time:   2021-04-09 22:27:31 UTC
Revision:     d5661e3f0e07a8caff705f1badf79fb5df8022c4

Kotlin:       1.4.31
Groovy:       3.0.7
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          16.0.1 (AdoptOpenJDK 16.0.1+9)
OS:           Windows 10 10.0 amd64

Workaround also works for me.

@ljacomet
Copy link
Member

@ljacomet ljacomet commented Jul 16, 2021

@henryju If you can craft a reproducer, please file an issue.
org.sonarsource.dotnet:sonar-csharp-plugin:8.23-SNAPSHOT@jar is actually an artifact dependency that should not even look at metadata.

@henryju
Copy link
Contributor

@henryju henryju commented Jul 16, 2021

I created a reproducer here: https://github.com/henryju/repro_gradle_11321
Simply install the Maven project:

cd repro-gradle-dep
mvn install

and then try to build the Gradle project:

cd repro-gradle-main
./gradlew build
./gradlew build

FAILURE: Build failed with an exception.

* Where:
Build file '/home/julien/tmp/repro_gradle_11321/repro-gradle-main/build.gradle' line: 25

* What went wrong:
Could not determine the dependencies of task ':jar'.
> Could not resolve all files for configuration ':extraLibs'.
   > Could not find org.sonarsource:repro-gradle-dep:1.0-SNAPSHOT.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom
       - file:/home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml
       - file:/home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s

I don't have file /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata.xml but I have /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/maven-metadata-local.xml:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>org.sonarsource</groupId>
  <artifactId>repro-gradle-dep</artifactId>
  <version>1.0-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <localCopy>true</localCopy>
    </snapshot>
    <lastUpdated>20210716135756</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>1.0-SNAPSHOT</value>
        <updated>20210716135756</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>1.0-SNAPSHOT</value>
        <updated>20210716135756</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

File /home/julien/.m2/repository/org/sonarsource/repro-gradle-dep/1.0-SNAPSHOT/repro-gradle-dep-1.0-SNAPSHOT.pom is there, contrary to what the log is saying.

The issue seems to come from the Maven packaging type that is sonar-plugin and not jar. I'm not sure what is different in produced metadata, but Gradle seems to be unable to locate such artifacts in the local repo. Maybe there is a confusion between packaging type and artifact extension?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants