Qualifier not replaced in Maven target location generated feature

[lorenzodallavecchia]

The Maven location in target definition files supports the automatic generation of a feature that includes all bundles picked up from Maven. I am using that feature for shipping the dependencies as part of my product/update-site.

As shown in M2E documentation, this is the relevant piece of XML in the .targetfile.

<location includeDependencyDepth="infinite" includeDependencyScopes="compile" includeSource="true" missingManifest="generate" type="Maven">
    <feature id="test.feature.deps" label="Maven Dependencies" version="1.0.0.qualifier">
    </feature>
    <!-- ... dependencies -->
</location>

The issue is that the qualifier token is not replaced with a timestamp in the generated P2 artifact: it is still versioned 1.0.0.qualifier and not something like 1.0.0.202601280000.
This is a problem when Maven dependencies change and people use the update site get the new version of my product. Since the P2 installable unit has the same version, the software installer will skip installing the new feature, leaving the user with the old Maven dependencies.

Steps to reproduce

1. Clone the repro repository.
2. Build the project with mvn clean verify (using Maven 3.9.12).
3. Look at the generated artifacts in test-repository/target/repository/features: the file is named test.feature.deps_1.0.0.qualifier.jar.
   The same problem is also present in the test-repository/target/repository/content.jar (i.e. the problem is with P2 metadata, not just the file name).

[laeubi]

@lorenzodallavecchia this is actual expected behavior.

If you specify a feature in your target you take full control (and responsibility) for the id, version and contents and we can't do any qualifier replacements for the following reasons:

1. Target resolution happens very early possibly before even projects are read
2. Targets can come from anywhere, a GAV, a file, even an arbitrary URL so there is not always a context to resolve a qualifier against (e.g. build or git timestamps)
3. This has to work in the IDE as well and there is no concept of a qualifier resolution as well

Most important for your specific case consider not to mention external dependencies in a feature at all. P2 will pull in required dependencies anyways so you don't need to list them explicitly.

If you still want this (for whatever reasons) you need to make sure that whenever you update a dependency you also update the version of your feature (and usually this wont be the qualifier but rather a minor or major part of the version).

[lorenzodallavecchia]

Thanks for the quick reply @laeubi.

I think I quite grasp the technical challenge about the missing context and early resolution.
The problem is that the generated feature is never a real source file, so it is not affected by the Maven plugins that would publish it. It is considered a third-party dependency originating from the target platform.

I'm not fully convinced about the IDE part. The IDE (and the PDE launcher) never considers qualifiers AFAIK. I can modify plugins/features and see the effect of those changes, even if the launched product still contains the qualifier tag.
So, after changing Maven dependencies, the developers would think their changes worked; only after the build (and a manual test update) would they discover that the update did not work.

IIRC, the reason for including that special feature is non-OSGi dependencies that are automatically wrapped by BND. When a BND creates an automatic MANIFEST.MF marks all Import-Package entries as optional, so the Maven libraries will not be included in the built product/site.

At the moment, we are updating the version manually, which risks us forgetting it. The main issue is snapshot builds of a new release, during which the major.minor.micro part does not change, and we rely on SNAPSHOT/qualifier for deploying the changes.

[merks]

For what it’s worth, Orbit makes use of such features, and it automates everything.

https://github.com/eclipse-orbit/orbit-simrel

That includes updating the target files and updating the feature versions. With automation there’s no risk to forget something.

[laeubi]

I'm not fully convinced about the IDE part. The IDE (and the PDE launcher) never considers qualifiers

Exactly... it just reads what ever is there on disk. Tycho replaces the qualifier during the package phase (and before it is published to p2) but a target platform does not has such phase. As that, the version of the feature is simply literal.

Independent of that, it does not makes any sense for your case either. A Qualifier change indicates that there is no functional change, it simply do not matters. A micro change would indicate that there is at least something changed (e.g. you upgraded a plugin from version x.y.1 > x.y.2) and so on ... so depending on the qualifier for an update is broken by design anyways.

For the dynamic imports, this is simply the default and you can change it. Also this feature was never meant as a permanent function but to allow rapid prototyping until proper OSGi metadata is provided at the source and publishing such bundles has its own risk. Still if none of your client code uses anything you maybe not need it, so using a feature for that is actually the worst option maybe as it has many pitfalls.

[lorenzodallavecchia]

I second your point about updates: in fact, the problem happened because the QA folks often test snapshot builds by updating from the previous one. End users always see a bump in the "micro" component, so this is no issue for them.

Our use case is admittedly quite messy, and we were hesitant to include all optional dependencies (including non-Maven ones) immediately, so we looked at the "generated feature" as a temporary solution to ensure that all dependencies originating from Maven are shipping. I do plan to stop using the special feature eventually.

Given that this is Tycho's intentional behavior and not an oversight, this can be closed.
Thanks for helping out!

[laeubi]

If you are using a CI pipeline it might be worth replace the .qualifier in the target with the build id using a pre-step before you start the build.

Then in a release build one likely want to replace the .qualifier with an empty string.

[lorenzodallavecchia]

I looked into why we are using the auto-generated feature that includes all Maven dependencies and tried removing that.
Turns out that there are two problems that emerge when not using that feature.

Optional dependencies are not included

Even if I set includeAllDependencies to true in the tycho-p2-repository plugin, the generated P2 repository does not include transitive optional dependencies.
The problem here is not optional dependencies I control (they have a non-optional requirement elsewhere), but the "fake" optional dependencies created by bnd when wrapping non-OSGi artifacts. Basically, there are some non-OSGi artifacts that do have real dependencies, but bnd translates them to optional Import-Package entries, and includeAllDependencies still leaves out optionals.
The final repository would be perfectly installable, but will fail at runtime because of the missing transitive dependency.

How to declare the minimum required versions of dependencies

With the generated feature, my product ends up including and requiring an exact version of all transitive dependencies. This is of course not ideal (the client cannot even use newer dependencies because all versions are fixed) and I would rather not use the feature as you suggested.

However, once I remove the auto-generated feature, the P2 site no longer contains any information about minimum required versions. Since my MANIFEST.MF files do not specify any version; all the requirements of P2 IUs are 0.0.0, which means I can no longer guarantee that minimum versions are met.
I feel that the correct thing to do here is to include version requirements in MANIFEST.MF, but I wonder what is the best way to maintain them aligned. For example, if I update a dependency in the target platform, I have to change the version of all Import-Package directives across all plugins. Is this the only way? Can Tycho (or other tools) "fix" minimum versions automatically according to the target platform?