Build an aggregated repository containing all direct-from-maven bundles as well as all Orbit bundles #3
Comments
|
Perhaps we can find time soon to chat about the best approach to move forward on this prototype work which is maintained here: https://git.eclipse.org/c/oomph/org.eclipse.oomph.incubator.git/tree/orbit |
|
I'll send you an email to set up a time. |
|
In follow up to our discussions, you are correct that adding the older orbit repo to the aggregation pulls in a number of additional things, not only the one specified IU (though that prevents pulling in all the content). I split the one contribution into three so that I can easily see what's coming from the older orbit repo:
I will see which of those I can hunt down from Maven. Here are some that I already found: |
|
It seems a lost cause to try to eliminate these. The dependencies on them typically are very strict, or they are not available at Maven at all. Moreover, almost all of these are ending up in SimRel so someone is still getting them from older Orbit repositories. Even the platform is doing that: So I think it would be good to include these in the aggregated result. Are you okay with that? |
Yes I am. |
|
@merks will there be a release specific stable url for this like we have for orbit. |
|
Yes, these are published using the same JustJ-provided infrastructure as is used by EMF. So when there is a release, the most recent milestone is mirrored to the final release location. So just as EMF has these:
Eventually these two will exist:
Exactly the same applies for these repositories: https://download.eclipse.org/oomph/simrel-maven/ Of course the release repositories are permanently retained. I follow the Platform's version numbering. The correspondence is well documented: |
|
@merks the point is about orbit: |
|
For background, is the note that I posted:
|
I'm really not sure the exact point. Yes, Orbit provides a recommend repository for the 2023-03 release but not yet for the 2023-06 release. Orbit can do that because it's gone through a release cycle, but these new repositories have not gone through a release cycle. As for consumption in your targets, you ought to be using https://download.eclipse.org/tools/orbit/downloads/latest-S/ so that you coordinate on the versions for the current release cycle, not on the versions used during the previous release cycle. So it concerns me that you appear to only use releases and furthermore that you expect that approach to be self evident. In any case, https://download.eclipse.org/tools/orbit/downloads/latest-S contains exactly the same bundles as https://download.eclipse.org/tools/orbit/downloads/latest-R currently, so https://download.eclipse.org/oomph/simrel-orbit/milestone/latest also contains exactly the same bundles as you are already using. An this repository will always exist and will always point at the version you ought to be using in combination with your SimRel contributions... |
|
@merks there is https://download.eclipse.org/tools/orbit/downloads/2023-06 already available as is https://download.eclipse.org/releases/2023-06/ the point is e.g. for wizard we need a stable url accross releases |
|
First, thank you for your effort @merks I have a question. For example Platform, CDT, Passage and may be others use |
|
Which wizard? I could provide another location that composes what's there now, but I'd have to develop some automated way to maintaining that and it's not entirely clearly to me why/where you need a stable URL in advance. Yes, folks can be notified but one has to figure out if that's some Github issue, or some bugzilla and then hope there is really someone listening on the other end. Or we can assume that folks are not entirely passive and read the cross-projects notifications and act accordingly. There are even reports generated for these repos: We can see that com.sun.jna is very popular:
In the end, I think the process is very clear. Everyone use the latest version. |
|
BTW, I think the above is a perhaps good example where "we/Orbit" should probably delete the 5 older versions or certainly the 3 5.x versions... |
in Xtext we have a new Project wizard that creates target platforms => both would need changes basically after quite week to consume a late showing up repo. what i currently do is this: the remaining work until release is just "paying attention/running nightly builds" |
|
Seems that I failed to formulate my question in the right way. The question is how to sync 3rd party library changes across several projects. GitHub/Bugzilla could work if everyone will read all the notification carefully. We can't always rely on that, because for projects like Platform there could be hundreds of notifications without clear separation of which of them regarding 3rd party changes. As far as I understood "It is currently planned to reduce what's available via Orbit". So, if we will not have Orbit as a sync point, how should we coordinate the library upgrade from Maven?
I doubt that this quote means "whoever updated from Maven faster won". Usually changes are introduced by Platform and then being adopted by downstream projects. Otherwise it could be a pain for EPP producers and other consumers.
In this case we returned back to Orbit concept. Orbit was a 3rd party sync point for many years. What should we use as a list of "recommended" versions of 3rd party if we "reduce what's available via Orbit"? How should we announce the library removal properly? |
|
I hate to second guess your designs, but might it not be better to use https://download.eclipse.org/releases/2023-06/ ? In any case, as I said, I could have some other location with names like 2023-06 that compose what I already am producing, but would I need to develop that and I would like to fully understand why that's needed and how that will be used. I'm Oomphing the Xtext setup to have a closer look. |
|
When I asked @akurtakov how coordination will work without a central point like Orbit, he said, "if everyone just uses the available version that will work too. " It struck me that this is a reasonable statement against which I see no strong counter argument. If you want to know what the platform is using, you can look here; https://github.com/merks/simrel-maven/blob/main/platform/REPORT.md I've been working with them to ensure that they use the latest version as soon as it's available. Your assumption that the Platform generally leads is just not correct. E.g., the Platform doesn't use guava but WindowBuilder pulls a new one directly from Maven. So the only strategy that works now, given folks can and do pull directly from Maven in their own *.target, is for everyone to use the latest. The repositories I'm publishing are generated from this information and literally compare the versions being used in each *.target to the versions available in Maven and updates them automatically, ultimately providing a single point of reference. Here is the current list of recommended versions, though it's arguable too long (too many versions) and one should use the latest version: https://download.eclipse.org/oomph/simrel-orbit/milestone/S202304041533/index.html Here is the report for that repository: I think it's the best that can be done given the way projects can and do pull directly from Maven themselves. Or do you have a different suggestion? |
|
I've enhanced the update site publisher to generate "simrel alias" composites as follows:
This composite is generated automatically and will point at the latest nightly build of 4.28, unless there is a milestone build for 4.28 in which case it will point at the latest milestone build (as it currently does), unless there is a release build of 4.28 in which case it will point to the 4.28 release and will never change again. The process repeats when I increment to 4.29 at the start of the 2023-09 release cycle, creating the 2023-09 repositories with the first nightly build thereafter. I think that addresses your concerned, correct? |
|
nice, yes this what i am looking for. gonna test tomorrow |
|
Thank you Ed for this work!
Nevertheless eventually, in the glory future, I believe everybody should use Maven-targets directly and we should hopefully be able to use the dependabot directly. @laeubi started an effort to enable that:
It would be great if those who know Ruby help with the PR and if everybody else expresses support for it (Thumps up or constructive comments), so that the dependabot devs see that it would be important for a larger audience. Since the strategy is to always use the latest version the results are equivalent but it would have multiple advantages:
If the latest version does not work for you, you should make sure that your OSGi metadata make that clear (they should do that anyway :) ) and be prepared that this version is present in an assembled application (which could happen anyway too, since application builders can use Maven targets too). Given that Maven targets are supported since Tycho 2.7.5 (maybe even a bit earlier) the only inconvenience I see for them is the requirement to set up an extra PGP signing step. AFAIK it has already been discussed (multiple times) if just the original signatures from Maven-Central could be retained and copied into the p2 metadata, but I don't know anymore where this was and what the result was (I guess there is a reason to sign them again). |
|
@merks adopting went quite smooth. many thanks for your efforts |
|
I very much look forward to the glory future! Realistically though, very few projects have significant resource to invest in releng, and most do not have leading experts involved in that effort as the Platform team does, so it will be a gradual transition... With regards to keeping the original PGP signature, imagine if we did preserve the PGP signature from Maven, and properly fetched the corresponding key to record both of those. (One needs access to key server and because of ID collisions, one needs to verify that you've really found the correct corresponding key for each signature.) Now also imagine that we do not add our own PGP signature using a pre-trusted key for which the user is not prompted. Then imagine the user being presented with a list of hundreds of keys to trust, suggesting that they carefully check each against a trusted key store. 😱 I don't think those will be happy users. They'll quickly figure out how never to be prompted again. 😝 |
So whats the difference, given that the eclipse key is trusted always? The resigning is literally only a "fig leaf" over this problem to simulate a way of trust that actually is not present at all, so what would be the difference in just trusting everything if we resign everything so the user is never asked for trust? And about the key-server, I already suggested that eclipse should simply have their own and not store the key in the meta-data (as this completely defeats the purpose of revocation checking) and build a real web-of-trust ... |
|
If everything is signed with an Eclipse key that's trusted always, then it's not a problem. I suggested imagining what would happen if one did not do that. As for trust and fig leaves, I think downstream consumers can assume that we Eclipse developers use these 3rd party libraries during our daily development, that we have run all our tests with them, and that we have some level of trust in them such that we feel it's perfectly okay to foist them on our downstream consumers via redistribution. I'm not sure we can do any better. Or? I asked for a key server, I was told no, and that does seem self defeating: |
Sure but whats the difference between
versus
I really doubt that anyone is reviewing each binary and we can "trust" them, just keep in mind that Log4Shell was never discovered by any of the millions of (downstream) consumers, so why should anyone "trust" Eclipse more, than items from Apache, Redhat, ... whatever. So in the end users install things that (hopefully) others have tested an reviewed, the fact that it was once downloaded by by an Eclipse Server does not add any "security". |
|
The difference is that for 1 the user sees no prompts (just as they see no prompts when everything was jar-signed), whereas for 2 the users is prompted with so much information they cannot reasonably be expected to do anything but rubber stamp it, which creates a bad impression that is not to anyone's benefit. Is anyone "safer" or more "secure" either way? Was everyone safer when everything was jar-signed? Not really. I think that's your point and on that I fully agree. I think all this only provides for certification of origin. I would generally trust things that come from Eclipse, Apache, or IBM, or Redhat, but probably not something that comes from Viruses4U.com. Is there a better basis for trust that we could reasonably provide? We can't even guarantee that the code is bug free, nor can we state that the code has no unknown lurking CVE. So I think saying "this came from us and we believe it to fit for its intended purpose" is the best we can do. |
So you actually would have trust four keys not hundreds of keys :-) So lets say an eclipse project includes something signed by That's why from my personal view, I better would approve the four keys you mentioned but getting know what this is from than always trust everything any of the > 400 (!) Eclipse projects somehow redistribute. And thats why I always stated that resigning arbitrary thing we not even have build is bad (from my POV). |
|
Maybe I make a bad assumption that each library (Maven group ID) is signed by some different key, or maybe you make a bad assumption that all Apache projects signed their stuff with a single key. I would have liked to be able to trust every key signed by the webmaster's key. This way trusting Eclipse would indeed be trusting one key, as in your first paragraph, rather than > 400, as in your last paragraph. But then we get into the key server discussion again... Note that for SimRel (the Eclipse Installer) I gather up all the keys used to sign all the PGP-signed content, enrich them with key server information, and automatically trust that set: https://git.eclipse.org/c/oomph/org.eclipse.oomph.git/tree/setups/keys/trusted-keys.asc So if there end up being 80 Eclipse project keys from all the participating projects, those will all be trusted automatically. |
The truth will be somewhere in between :-)
As mentioned, I think such a service is crucial if you want to really build trust on PGP, just assume one of the project keys are tampered (either by intend or accident) as recently happens by Github (and I assume they really do a lot to prevent such reputation loss), we currently have no way to effectively revoke it, even worse they are all automatically trusted even though I probably have never installed anything from such a project. |
|
Yes, I will likely revisit the key store issue. That being said, the same mechanism that lets me deliver trusted keys would also let me deliver information about revoked keys. The prototype logic I have already implemented removes from the set of trusted keys, any key that's revoked, so even if the key list listed as trusted (and I add it to my trusted keys file) it will still result in a prompt and that prompt will explicitly point out that a revoked key is involved. But we have strayed far from primary topic of this issue. 😜 |
Yes definitely, that's why I still appreciate this service. I just wanted to outline how the future hopefully (at least for me, but probably also for others) will look like and to advertise Christoph's issue/PR in dependabot, hoping for more attention and support. 😃
Understand, thanks for the explanation and the following discussion.
What I found interesting about the conclusion in that issues are the following two points: From my understanding the first bullet point is misleading, because AFAIK the Eclipse IDE did not move to PGP in that sense that it is preferred over jar-signing and we only want to use PGP. I assumed that support for PGP was only added (mainly to P2) because Maven-Central mandates it as the only from of signature and we wanted to consume original, unmodified jars from Maven-Central while having some from of validation. And the only way was to support PGP. Is that correct? |
|
It is factually true that the Eclipse IT team was not consulted about any need for new services. Also, the initial p2 PGP implementation/framework did not anticipate this need either and in fact it took an approach that would work without needing to consult a key store by storing also the public key, unlike Maven, which only maintains a signature but not a resource mapping to the corresponding public key. Hence no need nor attempt at consultation with the IT team. At no point does there appear to have been an effort to reuse the PGP signatures provided by Maven central but rather to reuse the same concepts in p2 as are used by Maven and to sign things with our own keys. None of these decisions were mine nor was I consulted in any of the preliminary designs and decision making processes. I don't know if alternatives to PGP were considered. I often find myself between a rock and a hard place, making the best of what's there and trying to improve upon it, while dealing with constructive criticism from all sides... |
|
Understand. Thanks for your elaboration. |
Oomph has been building a repository that is computed by composing the direct-from-maven dependencies of all the target platforms of all the projects participating in SimRel that define such dependencies, updated to their latest available version from Maven central:
https://download.eclipse.org/oomph/simrel-maven
The details are documented here:
https://github.com/merks/simrel-maven#readme
Now Oomph is building a repository that is the aggregation of these direct-from-maven dependencies along with the contents of an Orbit repository:
https://download.eclipse.org/oomph/simrel-orbit/
The aggregation is currently defined like this:
The reason for the inclusion of org.apache.commons.httpclient from https://download.eclipse.org/tools/orbit/downloads/drops/R20201118194144/repository is because otherwise the aggregation fails like this:
Of course we could just make that repository a validation repository, but I don't think it makes sense to build a repository where some of the IUs can't actually be installed because of missing requirements. (The direct-from-maven repository is separately tested for such requirement completeness.)
The https://download.eclipse.org/eclipse/updates/4.27/R-4.27-202303020300 repository is added as validation repository because some IUs explicitly require org.eclipse.osgi but they would likely be better of with package imports of the org.osgi packages they are actually using, e.g.,
The build generates reports and we can see that there are a great many duplicates:
We should consider whether it makes sense for Orbit to continue to provide bundles for which there are also direct-from-maven versions because of course an important SimRel goal is to reduce or eliminate the number of duplicates. We saw during the last release cycle all the problems around gson versions, so duplicates can and do cause problems even for bundles that are not singletons.
The text was updated successfully, but these errors were encountered: