Replace Orbit recipes with M2E's 'Maven' target locations fetching from Maven-Central

[HannesWell]

In order to ease the transition to use M2E's Maven target locations in projects directly and to simplify the update of existing artifacts in Orbit and reduce the technology stack, I want to propose to use Maven type target locations in Orbit.

When migrating a Orbit recipe to a dependency in a Maven target location one has to distinguished if the artifact at Maven-Central already has a OSGi compliant manifest or not:

• OSGi compliant artifacts can be used as they are from Maven-Central as simple dependency
• For not OSGi compliant artifacts, use instructions element of a Maven-Target. The instructions should probably be similar to the previous Orbit recipe (osgi.bnd).

In some cases this can leads to subtle differences in OSGi metadata and the artifact content. In Orbit some artifacts were split into two (e.g. logback 1.2) or mutliple into one (e.g. guava and its failureaccess dependency).
The generated artifact will also not contain the Eclipse license files. But if they are a requirement the build could be adjusted to add them to the artifacts with generated manifest.

To ensure license compliance of all artifacts in Orbit the reusable eclipse.dash-licenses license-check workflow can be used:

• https://github.com/eclipse/dash-licenses/blob/master/.github/workflows/mavenLicenseCheck.yml
• Usage example: https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/.github/workflows/licensecheck.yml

Ed's new SimRel-Maven service (which is great :) ) would then mainly serve the purpose of providing updates for Maven-targets used in SimRel projects until dependabot can do it (dependabot/dependabot-core#6913). At the moment I have the impression that it is also used as extended Orbit with more recent Maven-artifacts from Maven-targets (merks/simrel-maven#3).

If you are interested I can make an initial migration for a simple case, but migrating all recipes will be a greater task and will probably also need some adjustments in consuming projects because of the subtle differences mentioned above.

At the same time this could also be a good opportunity to clean up unused artifacts. But it is probably not sufficient to only look into SimRel is it?

[jonahgraham]

I want to propose to use Maven type target locations in Orbit.

A while ago I thought that we would eventually have Orbit have m2e target locations, however with Ed's work I don't think we should host them in Orbit, rather in the new stuff of Ed's.

Regardless of where it is hosted, here are some other comments/background:

has a OSGi compliant manifest or not

There are two other cases to consider, but we can handle them when we get there:

• There are some bundles modified in Orbit because of IP reasons where parts of bundles have been excluded
• Bundles that don't come from maven central (either hosted only on repo.eclipse.org) or ancient (but still consumed) bundles that we built from source and don't even have the source anymore (it was in CVS).

OSGi compliant artifacts can be used as they are from Maven-Central as simple dependency

These are the bundles that are in Ed's repo already. What is needed is a plan to remove the now redundant ones from Orbit.

For not OSGi compliant artifacts, use instructions element of a Maven-Target. [...]

Sounds fine to me, although I don't actually get the advantage of using a target files over a osgi.bnd file to store these items in. Because I am used to osgi.bnd I like them better because each output is its own file/directory. When developing an OSGi bundle this is very convenient because I can iterate quickly by just running maven in that directory and the /target dir contains all the intermediary files so I can quickly examine input, output and compare to previous version (where applicable).

That said, I don't do this often enough to stop the transition.

In some cases this can leads to subtle differences in OSGi metadata and the artifact content.

For cases where the maven bundle can be used unmodified I think that is fine, we need to make that transition. It would be easier to do it on new versions of those bundles then changing the difference in more subtle ways. That said, we have often had metadata changes with the only version change being in the qualifier as bug fixes to the metadata have been applied.

Apache Ant is one of the most complicated bundles in this category.

migrating all receipts will be a greater task

I don't think it is worth the time to blanket migrate. Instead I think we should focus on putting new versions in the new flow, but at the same time delete the corresponding Orbit bundle. It is the latter part that hasn't happened already for lots of bundles.

The retention policy for Orbit means if people want to consume older versions they can use older R-builds.

By removing all the out of date recipes we can get down to a more manageable number of things to maintain going forward.

Final thought/reminder - Orbit is used by some non-SimRel projects. example

receipt

PS it is an orbit recipe rather than receipt.

[guw]

A while ago I thought that we would eventually have Orbit have m2e target locations, however with Ed's work I don't think we should host them in Orbit, rather in the new stuff of Ed's.

@jonahgraham I think it's important to have a central location hosting (signed) Maven artifacts for simrel projects. This makes it easier to align projects on a single version. The reason I believe Orbit should (continue to) be this central instance is community involvement. If we create a second thing next to Orbit than people continue to blame Orbit for not being active and abstain from getting involved.

If it becomes super ease for people to update (flip the version) and consume the new version, then the engagement in Orbit could potentially be higher.

I think the goal should be:

• PR with version update (just a single line change)
• wait for PR checks (IP clearance)
• merge PR
• wait for CD build
• consume artifact

This is already a lot. There will still be people complaining that this is too excessive because they could do the single line change within their own .target file and be done.

[merks]

There continue to be so many conversations about this topic, but when there are 3 week gaps in the discussions, there isn't much progress to be made.

With the pressures to get Mylyn into m2, this does exist already:

https://git.eclipse.org/c/oomph/org.eclipse.oomph.incubator.git/tree/maven-bnd/tp/MavenBND.target

Of course this will be moved to GitHub, hopefully right after the 2022-06 release...

I announced the concerns and associated issue on cross projects, but almost no one was interested. So in general engagement seems fully lacking. It seems to me that unless people see the direct and immediate impact on them personally, they don't engage at all. Though later, when impacted, then there are unexpressed opinions.

[merks]

@guw

I strongly suggest you read the details of this issue:

eclipse-mylyn/org.eclipse.mylyn#103

You'll note the strong attacks against the concept of doing anything centralized, i.e., against Orbit, and against the people working on aligning projects on a single version. It feels to me like I'm the only one engaged while weathering a storm.

[guw]

@merks I hear you. I decided to stay out of these conversations.

I do think there is an opportunity, though. Maybe it requires a policy change at the simrel (planning council) level for how projects contribute.

For example, lets say there would be a decision that projects:

1. must use Import-Package for anything not originating from eclipse.org and
2. must only add bundles to their contributed feature.xml, which are developed at eclipse.org

Then simrel can have it's own repository (Orbit, yours, Maven Central?) for producing and consuming third party artifacts and ensures there is only one single version.

But most projects fear the free choice of versions (with osgi and p2) and instead prefer hard coding bundles in their feature.xml because "that's what we tested with".

[msohn]

I agree there are advantages of agreeing on one version (ideally the latest one) of 3rd party dependencies across simrel.
I would support a model like the one @guw proposed and participate with JGit and EGit.

[ruspl-afed]

I support "centralized" approach. For example, in Mylyn we need to work with several components in one environment and it's from "difficult" to "impossible" to achieve if we have a full set of potential combinations for third-party libraries.

As we discussed before, the main drawback of Orbit is its inability to react on changes relatively quick. If we can reduce "time to market" from months and weeks to days and hours and introduce automated rules to validate structure of SimRel contributions (Import-Package directive, feature.xml content, etc.) - I believe a lot of downstream projects will have less headache regarding how and when they should update its dependencies.

[guw]

As we discussed before, the main drawback of Orbit is its inability to react on changes relatively quick.

@ruspl-afed I disagree. It's not a drawback of Orbit but IMO a lack of engagement of people/committers. It requires effort to automate things but building a CI/CD pipeline can make things move quickly from PR to availability.

[ruspl-afed]

@guw If you remember the discussion around log4j some time ago, it was not about "let's update and release Orbit first and then let others consume from it".
In any case I agree that an automated check integrated to CI/CD pipeline is important.

[jonahgraham]

As we discussed before, the main drawback of Orbit is its inability to react on changes relatively quick.

In any case I agree that an automated check integrated to CI/CD pipeline is important.

There is a fully automated CI/CD pipeline for Eclipse Orbit - https://download.eclipse.org/tools/orbit/downloads/latest-I/ is populated within a couple of hours after a PR is merged using https://ci.eclipse.org/orbit/job/orbit-recipes/

The only thing missing is automatic license checking - I have added a PR for that in #16 but you can see the comments I added it will always fail for now.

[HannesWell]

Thank you Jonah for your elaboration and sorry for the late reply (just a bit too much to do at the moment).

receipt

PS it is an orbit recipe rather than receipt.

Thanks, fixed that.

For not OSGi compliant artifacts, use instructions element of a Maven-Target. [...]

Sounds fine to me, although I don't actually get the advantage of using a target files over a osgi.bnd file to store these items in.

There is probably not a strong advantage, but the advantages I see are, although they are less important:

• It can be done in the same file like the definition of the maven-coordinates
• If applicable, the same instructions (which can be flexible) can be re-used for multiple artifacts

migrating all receipts will be a greater task

I don't think it is worth the time to blanket migrate. Instead I think we should focus on putting new versions in the new flow, but at the same time delete the corresponding Orbit bundle. It is the latter part that hasn't happened already for lots of bundles.

The retention policy for Orbit means if people want to consume older versions they can use older R-builds.

By removing all the out of date recipes we can get down to a more manageable number of things to maintain going forward.

Agree, that makes more sense. 👍🏽

A while ago I thought that we would eventually have Orbit have m2e target locations, however with Ed's work I don't think we should host them in Orbit, rather in the new stuff of Ed's.

@jonahgraham I think it's important to have a central location hosting (signed) Maven artifacts for simrel projects. This makes it easier to align projects on a single version.

Since in Orbit the latest version is intended to be used, projects can achieve the same if they just use the latest version in their Maven-Target. Hopefully we even get dependabot updates for that: dependabot/dependabot-core#6913

It is actually not too hard to set up the signing and license checks for Maven targets.

I understand that Orbit, while it makes it hard to update to new versions, had some comfort in providing a single stop for many third-party deps and often somebody else takes care about problems, but Maven targets really have many benefits as I have described here:
eclipse/xtext#2207 (comment)

Although my hope is that many projects (ideally all) use Maven targets and will happily get the mentioned benefits, at least for the case of non-OSGi compliant targets we will need one central place to coordinate the generated Metadata.
At the same time we can contribute to the upstream project to add OSGi compliant artifacts, so that the number of non-complaint artifacts is reduced over time.
Nevertheless, realistically this won't happen for all artifacts. For some project the maintainers maybe refuse to add OSGi metadata for whatever reason or because the artifacts are patched in other ways, as Jonah elaborated.

The reason I believe Orbit should (continue to) be this central instance is community involvement. If we create a second thing next to Orbit than people continue to blame Orbit for not being active and abstain from getting involved.

When Orbit gets its own Github orga, my suggestion would be to move Ed's SimRel-Maven project to this orga as well.

I do think there is an opportunity, though. Maybe it requires a policy change at the simrel (planning council) level for how projects contribute.

For example, lets say there would be a decision that projects:

1. _must_ use `Import-Package` for anything not originating from eclipse.org and

2. _must_ only add bundles to their contributed `feature.xml`, which are developed at eclipse.org

Yes that would really help in any case!
I highly appreciate Ed's work and effort in aligning SimRel projects on single versions, but with the requirements you mentioned it would be much simpler to align to a single version of a t-p artifact, even if some projects don't use the latest version in their TP and repo. Then the CBI-Aggreator should do that job automatically as far as possible.

I support "centralized" approach. For example, in Mylyn we need to work with several components in one environment and it's from "difficult" to "impossible" to achieve if we have a full set of potential combinations for third-party libraries.

Can you explain in more detail why it is possible with Orbit but not with Maven-Targets?

[laeubi]

I can only add that maven-targets wrapping feature was never meant as a "management" tool of BND instructions.
It was always meant as giving projects an easy and flexible way to use thing right now for their project giving them some kind of autonomy compared to waiting for a centralized aproach to catch up (e.g. orbit) or wait until the project itself adds the required metadata, and use that in a project scope within the build and the IDE and thats probably the most important part about it!

If one wants to maintain a central place like orbit, I think using the raw bnd-plugin is much more suitable and as BND allows applying the same bnd file to multiple items as well I don't think it is less flexible, especially as you can combine this with other approaches (e.g. download and repack additional items into an uber-bundle what is not possible in general with maven targets).

So to summarize, even though the feature might be used like that, it is not really targeting such use-case, because if there is a centralized update-site, for the consumer it doesn't matter that such builds can not be reused in the IDE directly as a user will almost always use the just published P2 site...

[merks]

This is in progress here

https://github.com/eclipse-orbit/orbit-simrel/blob/main/maven-bnd/README.md

The plans are described here:

https://github.com/orgs/eclipse-orbit/discussions/49