The target resolvement should never invalidate/error out, leave that up to the build.

[jcompagner]

It should later fail when the feature or plugin is actual build
so we have more information (like stuff are optional)

see for a discussion (and a test case) here: #380

[mickaelistria]

That's a good start. See inline comments about how to make it less risky.

[mickaelistria]

Looking at the implementation of Slicer, there is a chance this adds into the resolution context some undesired units (ie units that were not present nor expected, and will now be present unexpectedly).
I believe the responsibility of "removing" not applicable requirement would belong to the caller here, ie the "availableUnits" should be filtered beforehand according to the context.

[jcompagner]

so you are saying that the fix should be in AbstractSlicerResolutionStrategy.slice(Map<String, String> properties, IProgressMonitor monitor) ?

and then:

IQueryable slice = slicer.slice(seedIUs.toArray(EMPTY_IU_ARRAY), monitor);

we need to make sure that seedIUs don't have the stuff that can result in a false?

[laeubi]

But that's the whole purpose of the thing, to filter out invalid items, if we let in items that are not applicable the whole "resolve for a target environment" does not makes any sense for me as it only adds overhead to the resolution time.
Instead I would instead use a different approach were we accept all units that satisfy any of the provided target environments...

[jcompagner]

i kind of agree with that, i think that whole resolvement is not really needed, as i said in the other discussion
For me a target file when installing (so how it is used in Tycho) is just a bunch of Installable units from the given locations.

For me nothing has to be really resolved, it must just supply everything that is needed and then the actual builds (plugins,features and product) will do the actual resolvement and reporting of errors.

[laeubi]

This is the case and what we are seeing here is actually the part of targets called "Software Sites" where there are some strange requirements (possible due to the eclipse concept of singleton bundles) that only allows the "Planner mode" to resolve for exactly one environment: the checkbox: include all environments is grayed out if you select "include required software".

[jcompagner]

i also first thought about that one and i wanted to really overwrite validateInput:

but that is private :(
(then it would have a much lower impact really only that slice() call)

The thing is that ProjectorResolutionStrategy.java also used in scenario's where you would have that isApplicable() to be able to return false (and yes then something that we want does happen)

[laeubi]

but that is private :(

masters are open for 4.23 so you can open a gerrit review to make this protected :-)

[jcompagner]

so i did a refactor now i just overwrite the slice() method and filter out first what would not be applicable (so validateInput() would bomb out)

[mickaelistria]

The SlicerResolutionStrategy has a ignoreFilters flag, that is derived from includeAllPlatforms in the .target. IMO, this is exactly the kind of settings you'd like to see honored here and that is not honoured as expected.
I think your latest patch is interesting, but the behavior of custom vs regular slicer should be conditioned by the value of ignoreFilters (and that would mean you'd need to set includeAllPlatforms to true in your .target).

[laeubi]

The SlicerResolutionStrategy has a ignoreFilters flag, that is derived from includeAllPlatforms in the .target. IMO, this is exactly the kind of settings you'd like to see honored here and that is not honoured as expected. I think your latest patch is interesting, but the behavior of custom vs regular slicer should be conditioned by the value of ignoreFilters (and that would mean you'd need to set includeAllPlatforms to true in your .target).

As mentioned above includeAllPlatforms=true is incompatible with mode=planner

[mickaelistria]

As mentioned above includeAllPlatforms=true is incompatible with mode=planner

But here, @jcompagner wants to use slicer, so it's fine.

[laeubi]

Nope, the example target uses planner mode:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?pde version="3.8"?>
<target includeMode="feature" name="eclipse_sources" sequenceNumber="60">
    <locations>
	    <location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
		    <repository location="https://download.eclipse.org/eclipse/updates/4.21/"/>
    	    <unit id="org.eclipse.equinox.executable.feature.group" version="3.8.1300.v20210828-0802"/>
		    <unit id="org.eclipse.sdk.ide" version="4.21.0.I20210906-0500"/>
	    </location>
        <location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
            <repository location="https://dl.equoplatform.com/chromium-swt-ee/servoy/p2/80"/>
            <unit id="com.equo.chromium.cef.feature.feature.group" version="80.0.9.202110280424"/>
            <unit id="com.equo.chromium.feature.feature.group" version="80.0.14.202111290455"/>
        </location>
    </locations>
	<targetJRE path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-11"/>
	<environment>
		<arch>aarch64</arch>
	</environment>
</target>

[laeubi]

BTW as tycho supports multiple targets, one could think of using a "regular" target file that includes everything except the offending feature and on with that feature in slicer mode and include both in the build.

[mickaelistria]

Nope, the example target uses planner mode

My bad! Should we try to support planner+includeAllPlatforms in Tycho, despite it's not in PDE?

[jcompagner]

i have to say that i don't know what "planner" or includeAllPlatforms="false" or includeConfigurePhase="true" all really does
because in the target editor that is as far as i know that really a UI for that (that also explains this)

But are you saying that if i break up the target and one have it alll (just as the current one with the same config as above) and 1 with just the chromium stuff but then includeAllPlatforms="true" and i add both targets in my platform that it then works?
or do we then still need a patch?

I just wonder are there scenario's when we really want an exception to be given from that validateInput() when resolving the target? When is the target really kind of illegal and we should really stop?

[laeubi]

Nope, the example target uses planner mode

My bad! Should we try to support planner+includeAllPlatforms in Tycho, despite it's not in PDE?

I think this should only be done if its supported upstream (PDE+P2) otherwise it is too confusing and leads to bad user experience. As mentioned earlier I think the whole configuration here is a bit wrecked and making it "just work" has a bad smell. If one really likes to have such a behavior its better to fix the offending feature (and fork that for example) instead of putting the burden on the build tool.

[laeubi]

have to say that i don't know what "planner" or includeAllPlatforms="false" or includeConfigurePhase="true" all really does

You are probably not alone ;-)
I can only quote from the help page here:

The Included Software options at the bottom of the wizard affect how the set of plug-ins to add to your target is determined. By default, all required software is included. This means that we find all plug-ins that are required by the software you selected. All of those plug-ins must be available at the site or sites you have selected. If any piece is missing, the target will fail to resolve and display an error explaining what requirement is missing. This is the recommended setting for most users because it will ensure that you get the complete set of plug-ins in your target. The environment settings of your target definition will be used when determining required plug-ins so that platform specific fragments will be included.

The Include required software setting will search the site for any dependencies the selected items require. It is recommended that you leave this option on. If you are building a target that will not contain all dependencies or if the site is missing some required plug-ins, this option can be turned off. When off, any required plug-ins found at the site will be added, but no errors will be reported for missing items.

If the option to include required software is turned off, the Include All Environments option will be available. Turning it on will result in all available environment specific plug-ins to be added to the target, rather than just the plug-ins that apply to your target's environment settings.

[github-actions]

Unit Test Results

142 files  142 suites   41m 16s ⏱️
223 tests 220 ✔️ 3 💤 0 ❌

Results for commit f5f5ae3.

[laeubi]

and i add both targets in my platform that it then works?

You can add multiple <target> items to the tycho target-platform configuration. I'm currently trying to get a similar feature into PDE itself (you can vote for it or comment on the gerrit).

If it solves your problem depend a bit, I noticed that often projects don't mind to creating self-containing complete features what makes slicer mode a mess to use.

[mickaelistria]

How does PDE like this .target (or not)? Does it send an error, or the feature is silently included?... We could have Tycho mimicking that.

[laeubi]

How does PDE like this .target (or not)?

As expected:

org.eclipse.equinox.p2.iu; com.equo.chromium.feature.feature.group [80.0.14.202111290455,80.0.14.202111290455] cannot be installed in this environment because its filter is not applicable.

[mickaelistria]

Thanks @laeubi . So I don't think this should be changed in Tycho now; unless it's happening in a coordinated effort to have PDE also supporting this pattern without failing.
@jcompagner What you can try in the meantime, since listing this feature in the .target makes it invalid and unusable both for PDE and Tycho, is to add the https://dl.equoplatform.com/chromium-swt-ee/servoy/p2/80 repo as a <repository> in the pom.xml. I just tried it and it seems fine, it's making all the content available for resolution, without adding any requirement.

[jcompagner]

yes that seems to work,
and i can live with that because only a small maintenance price to pay here (i need to have a duplicate target one having everything that is used by our developers that generate there a local target from that where they then develop and run against)

and one target without the chromium urls (what our build system uses)

But i can just when i update the full target with the latest urls and IU versions i can just copy that over the "without chromium" and remove only those urls from it.

And then also update the pom file. because using has 1 annoying thing that you can't fix a version, and 1 product build should always results in the same artifacts, it can't be if a repository updated its thing that suddenly the product has different things. But happily chromium is provided with sub urls:

        <chromium>https://dl.equoplatform.com/chromium-swt-ee/servoy/p2/swt/80.0.14/repository/</chromium>
        <chromium.cef>https://dl.equoplatform.com/chromium-swt-ee/servoy/p2/cef/80.0.9/repository/</chromium.cef>
	</properties>

    <repositories>
        <repository>
            <id>chromium</id>
            <url>${chromium}</url>
            <layout>p2</layout>
        </repository>
        <repository>
            <id>chromium.cef</id>
            <url>${chromium.cef}</url>
            <layout>p2</layout>
        </repository>        
    </repositories>

so i can still pinpoint to a version. I just need to maintain this pom file when i update chromium in the main target, which is fine by me.

I do still think that whole target resolvement is to rigid it does stuff that it should not do at that time, its up to the system to decided on top of that if the target is ok or not.
(for me the target is just a "classpath" and the thing you build on top of that will give me "compile" errors, so not already the "classpath" itself)

[laeubi]

I do still think that whole target resolvement is to rigid it does stuff that it should not do at that time, its up to the system to decided on top of that if the target is ok or not.

As noted above, it is not the target handling but the Update site location that is supposed to work that way, that interferes here with a too restrictive feature design of the Chromium Product... If you pay for that you should better ask them to fix their feature handling to be more adaptive...

[laeubi]

I think it has proven that there is no immediate need to change tycho here, so can we close this PR?

[jcompagner]

i guess so, i have now a workaround that works ok for me

It is for me still weird if i patch (this pull) it then i have "input" and i have my "output" and everything is working just fine..

But for some reason there is in the "input" a validation that just shouldn't happen, we can discuss that the input is kind of wrong, but that is for me besides the point, still i can get a valid output, noting wrong with the output i have..

But where is the problem then exactly? is it P2 or Tycho?

[laeubi]

i guess so, i have now a workaround that works ok for me

Great!

It is for me still weird if i patch (this pull) it then i have "input" and i have my "output" and everything is working just fine..

Well its always possible to still have enough content in the target that it fulfills the requirement of a given setup. But the general contract of planner mode is that P2 has to calculate a satiable solution where all requirements are meet or else fail.

But where is [...] then exactly? is it P2 or Tycho?

P2 target content has two modes:

1. Slicer: Only declared IUs are included
2. Planner: Declared IUs and all their requirements are included

for sure one might suggest a third mode called Available or something similar that includes as much as possible but never fails on missing requirements. That should then first be suggested to P2 then PDE, and if thats avaiable we can add support for Tycho as well.

[mickaelistria]

That should then first be suggested to P2 then PDE

Directly PDE I think; p2 has nothing about Plugin Development in it, and p2 is already capable of consuming a whole repo as "available".

[laeubi]

Directly PDE I think; p2 has nothing about Plugin Development in it,

PDE (and tycho) uses P2 under the hood to resolve those target locations, but I haven't checked yet if that's just a matter of using existing P2 features (this PR seems to indicate this) or if there are any changes required.

Anyways I think this first should be implemented upstream.