[maven] Find snapshot metadata in nested pom

[jgarec]

What would you like Renovate to be able to do?
Being able to update to latest snapshot which could be newer than the latest release.

Describe the solution you'd like
By default, update to latest release but for some dependencies that are explictly described in config file, update to latest snapshot if available.

Describe alternatives you've considered

Additional context
In debug logs, i can see that snapshots are identified as "newer versions" but when renovate try to download it, there is an error because snapshot and release repositories have not the same structure (multiple artifacts for the same snapshot versions). This download fails so renovate considers that this version is unavailable :

DEBUG: Url not found http://xxxxx/repository/xxxxxx/yyyyyyy/artifact-name/5.1.22-SNAPSHOT/artifact-name-5.1.22-SNAPSHOT.pom

Release repository structure :

Snapshot repository structure :

In this case, i think renovate should download another maven-metadata and identify the right artifact to download.

[rarkins]

I don’t fully understand your request, sorry. Can you provide more details including examples and logs if possible? What type of Maven registry is it?

[jgarec]

Our repositories are hosted on nexus3 (i don't if the same problem occurs with artifactory).

We have 3 maven repositories :

• one for snapshots (hosted / maven 2 format / allow redeploy)
• one for private (hosted / maven 2 format / disable redeploy)
• one containing both repositories (group / maven 2 format), called : group-repository

Use case :
I have an application called myapp, with a dependency called mylib.
myapp use latest release : 5.1.21

here is an extract of maven-metadata.xml for mylib :

<metadata modelVersion="1.1.0">
  <groupId>groupid</groupId>
  <artifactId>mylib</artifactId>
  <versioning>
    <latest>5.1.22-SNAPSHOT</latest>
    <release>5.1.21</release>
    <versions>
      <version>5.0.0</version>
      <version>5.0.1</version>
      <version>5.0.2</version>
      <version>5.0.3</version>
      <version>5.0.4</version>
      <version>5.0.5</version>
      <version>5.1.20</version>
      <version>5.1.21</version>
      <version>5.1.22-SNAPSHOT</version>
    </versions>
    <lastUpdated>20190517040006</lastUpdated>
  </versioning>
</metadata>

Log file :

DEBUG: Found 1 repositories for groupid.mylib (repository=gitgroup/myapp)
DEBUG: Looking up groupid.mylib in repository #0 - http://foo/nexus3/repository/group-repository/ (repository=gitgroup/myapp)
DEBUG: Found 40 new versions for groupid.mylib in repository http://foo/nexus3/repository/group-repository/ (repository=gitgroup/myapp)
DEBUG: Found 40 versions for groupid.mylib (repository=gitgroup/myapp)
DEBUG: Url not found http://foo/nexus3/repository/group-repository/groupid/mylib/5.1.22-SNAPSHOT/mylib-5.1.22-SNAPSHOT.pom (repository=gitgroup/myapp)
DEBUG: groupid.mylib not found in repository http://foo/nexus3/repository/group-repository/ (repository=gitgroup/myapp)

when renovate try to download mylib-5.1.22-SNAPSHOT.pom, a 404 error occurs because it doesn't exist (the structure of a snapshot directory is different).

[jgarec]

Versions Maven plugin works well in this case :

mvn versions:display-dependency-updates shows no update available
mvn versions:display-dependency-updates -DallowSnapshots=true finds one update :

[INFO] The following dependencies in Dependencies have newer versions:
[INFO]   groupid:mylib ............ 5.1.21 -> 5.1.22-SNAPSHOT

but renovate is not able to find this update.

[rarkins]

Have you set "ignoreUnstable": false in your config (or with a package rule just for that one package)? It looks like it's failing earlier than that though, but you would need that setting as it's like the "allowSnapshots" equivalent.

[jgarec]

Sorry i didn't know this flag but it's only a part of the solution.

I've made a simple test : force the result of pomContent if pomContent is undefined when calling downloadMavenXml :

async function getDependencyInfo(dependency, repoUrl, version) {
    const result = {};
    const path = `${version}/${dependency.name}-${version}.pom`;
    const pomContent = await downloadMavenXml(dependency, repoUrl, path);
    if (!pomContent)
        return {homepage:'foo',sourceUrl:'bar'};
//        return result;

    const homepage = pomContent.valueWithPath('url');
    if (homepage && !containsPlaceholder(homepage)) {
        result.homepage = homepage;
    }
    const sourceUrl = pomContent.valueWithPath('scm.url');
    if (sourceUrl && !containsPlaceholder(sourceUrl)) {
        result.sourceUrl = sourceUrl;
    }
    return result;
}

renovate/lib/datasource/maven/index.js

Line 193 in b701e15

async function getDependencyInfo(dependency, repoUrl, version) {

• With this modification + ignoreUnstable: true => merge request is not created
• With this modification + ignoreUnstable: false => merge request is created

Package	Update	Change	References
groupid:mylib	patch	5.1.21 -> 5.1.22-SNAPSHOT	homepage, source

[rarkins]

So right now, if we cannot find homepage or sourc eurl then we return {} and then the version is actually ignored?

[jgarec]

Yes, if the pom file is unavailable (to extract homepage / sourceurl) then it fails.

i think it should :

• download maven-metadata.xml : http://foo/nexus3/repository/group-repository/groupid/mylib/maven-metadata.xml and get last version (existing behaviour)

• try to download the latest pom file : http://foo/nexus3/repository/group-repository/groupid/mylib/5.1.22-SNAPSHOT/mylib-5.1.22-SNAPSHOT.pom (existing behaviour)

  • if the file is available then consider this version as valid (existing behaviour)
  • if not, try to download this file http://foo/nexus3/repository/group-repository/groupid/mylib/5.1.22-SNAPSHOT/maven-metadata.xml

This new maven-metadata.xml looks like :

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>groupid</groupId>
  <artifactId>mylib</artifactId>
  <version>5.1.22-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <timestamp>20190516.193217</timestamp>
      <buildNumber>145</buildNumber>
    </snapshot>
    <lastUpdated>20190517040006</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>pom</extension>
        <value>5.1.22-20190516.193217-145</value>
        <updated>20190516193217</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

• extract value in content tag and finally download pom file : http://foo/nexus3/repository/group-repository/groupid/mylib/5.1.22-SNAPSHOT/mylib-5.1.22-20190516.193217-145.pom

[rarkins]

Sounds good, thanks.

[jgarec]

Finally, it seems that switching from ignoreUnstable: true to ignoreUnstable: false was the solution.
But i forgot to check the box "" to update the current MR ...
so .... it works perfectly ...

The only bug is that it can't find the sourceurl and homepage but i'm not sure it's necessary to add more complexity just for snapshots MR.

So if you're ok with that, we can close this "issue"

[rarkins]

I would still like to keep this open but will rename it so that we learn to capture the pom for the snapshot too. Thanks.

[jgarec]

Hi @rarkins @zharinov, FYI :
Since #5614 and because of this bug, It's no longer possible to get snapshots (with ignoreUnstable=false flag).

The current workaround is to use RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK flag.

[rarkins]

@jgarec can you summarize for us:

1. What files/data are published for snapshots vs stable releases?
2. Do you know if doing so is valid according to any spec, or would it be considered incomplete?

[jgarec]

1. Release vs Snapshot :

• Release repository structure :
  

• Snapshot repository structure :
  
  Requires to download another maven-metadata and identify the right snapshot to download (because it could contains snapshot version 142 143 144 ; this screenshot only show one) .

When this issue was opened, it was necessary to download the pom file of the snapshot to get the url of the project / the homepage ... but now, renovate is able to find this with the latest release published :

renovate/lib/datasource/maven/index.ts

Lines 274 to 277 in 5701164

	const latestVersion = getLatestStableVersion(releases);
	if (latestVersion) {
	repoForVersions[latestVersion] = repoUrl;
	}

So i wonder if we could just consider that using unstable versions doesn't require to check that the pom is valid and available ?

[rarkins]

If the pom file is there, then what leads you to need to skip the pom check altogether?

[bpfoster]

I believe I'm seeing the same issue as @jgarec. The maven datasource checks for presence of the pom via a URL like http://repo.com/groupId/myLib/5.1.22-SNAPSHOT/myLib-5.1.22-SNAPSHOT.pom in filterMissingArtifacts().

As you can see from his screenshots, the repository doesn't store a -SNAPSHOT file, instead only the timestamped files (5.1.22-SNAPSHOT/myLib-5.1.22-20190515.193717-144.pom).

So the isHttpResourceExists() in filterMissingArtifacts() returns false, and the version is filtered out.

[olegkrivtsov]

I'd like to take this one.

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

Actually this one needs a reproduction first

[github-actions]

When a bug has been marked as needing a reproduction, it means nobody can work on it until one is provided. In cases where no reproduction is possible, or the issue creator does not have the time to reproduce, we unfortunately need to close such issues as they are non-actionable and serve no benefit by remaining open. This issue will be closed after 7 days of inactivity.

[catostrophe]

It seems that #11327 doesn't fix this issue. Still can't update -SNAPSHOT deps without setting RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK.

[rarkins]

Needs reproduction

[catostrophe]

It turned out even RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK didn't work out.

[github-actions]

When a bug has been marked as needing a reproduction, it means nobody can work on it until one is provided. In cases where no reproduction is possible, or the issue creator does not have the time to reproduce, we unfortunately need to close such issues as they are non-actionable and serve no benefit by remaining open. This issue will be closed after 7 days of inactivity.

[github-actions]

This bug report has been closed as we need a reproduction to work on this. If the original poster or anybody else with the same problem discovers that they can reproduce it, please create a new issue, and reference this issue.