Requests for comments: how does opam-repository scale? #23789

hannesm · 2023-05-14T16:10:30Z

I've observed over the years that there's the sentiment that "no package shall be removed from opam-repository" (I still don't quite understand the resoning behind it -- maybe it is lock files that fail to register the git commit of the opam-repository? Maybe it is for some platforms that require the opam-repository for removing opam packages?

So, I'd like to raise the question why this sentiment exists. Answers (or pointers to arguments) are highly welcome.

Why am I asking this? Well, several years back Louis worked on "removing packages that aren't ever being installed" (with the reasoning that if foo in version 6.0.0 and in version 6.0.1 (with the very same dependencies) are available, 6.0.1 is always chosen (which makes sense due to it may very well be a bugfix).

Now, I also observe that rarely, really rarely old package releases get bumped their minor version -- i.e. the "long term support" of opam packages does not really exist (happy if you prove me wrong): bug fixes are incorporated with new features and API changes, and so new (major) versions are released.

Taking a step back, it is pretty clear that collecting more and more packages will lead to larger amount of work for the solver (which needs all the packages being parsed, and then find a solution). This means that the solver needs to improve speed-wise roughly every year. This is rather challenging, and in the end leads to opam not being very usable on smaller computers (Raspberry PI, or even tinier computers...).

Also with carbon footprint in mind, I fail to see why opam-repository may not delete packages. In the end, it is a git repository -- if you wish to install compiler in an arcane version, it is fine to roll back your opam-repository git commit to an arcane commit that fits nicely. The amount of work for the opam-repo CI could as well be minimized by removing old opam packages (since a revdep run will be much smaller).

Please bear with me if you've already answered this, and have written up the design rationale (and maybe the answer how opam-repository will scale). Comments and feedback is welcome. Once I understand the reasoning, it'll be much easier for me to figure out how to move forward. Thanks a lot. //cc @AltGr @avsm @kit-ty-kate @mseri @samoht

The text was updated successfully, but these errors were encountered:

hannesm · 2023-11-28T14:15:19Z

As a first observation: the opam repository is a git repository, so anyone who wants to install old releases can always use an old commit.

One idea I discussed with @mro is to add an expiration date on each opam package when publishing, i.e. "this is being maintained for 3 months". I was initially skeptical, but think this is a fine idea since various contributors (authors) and companies have different release cycles, and for me personally code that I don't use and touch for several years should either be released as stable (with a 1.x) and expiration of infinite; or will bitrot.

And for reference, I found these PRs and their discussion pretty insightful:

Especially the comment #11400 (comment) is worth reading, where different people have different opinions (in respect to whether it "is good for reproducibility to remove packages"). And note to myself, there's opam admin check --obsolete.

hannesm · 2023-12-05T11:49:17Z

Looks like nobody else cares (and Louis cared earlier, as outlined in closed issues and pull request). Closing.

mseri · 2023-12-05T12:56:49Z

I do actually care, but I don't have a good solution yet and, worse, there is not yet a real consensus on what to do. I hope we will settle on some periodic stable releases and an archive repository where old packages go to rest, or some other compromise that will make things slimmer.

mseri · 2023-12-05T12:57:55Z

If you ask me we should leave this open as an opportunity to get more ideas. There was also a discuss (or even more than one) thread where we had a discussion about this. If I can find it, I'll link it

samoht · 2023-12-05T22:07:34Z

I also care.

But what I would like to see before deleting packages is a process to move these deprecated/unmaintained packages to a separate repository. Keeping them around is really helpful when you need to test large-scale language features (for new release, but also to check langage statistics for old releases).

I think the main blocker for this right now is to have the appropriate tooling (for instance to split an existing opam-repo easily).

hannesm · 2023-12-07T12:48:22Z

I doubt this is a tooling issue to be honest, since there's opam admin check --obsolete around (surprise: it may need some improvements), as well as opam admin check --installability.

It feels more like a policy decision about what should opam-repository achieve - together with what compilers to support (see related issues #24868 ocaml/opam#5748 -- but also ocaml/infrastructure#87 ocaml/infrastructure#48).

But what I would like to see before deleting packages is a process to move these deprecated/unmaintained packages to a separate repository.

What prevents you from adding a tag before deletion of anything in this repository, and establish a (e.g.) quarterly routine on what to delete (i.e. have a quarterly tag, 2023Q1..Q4)?

Maybe it is just me, using a ~10 year old laptop, who suffers from this issue -- but it looks like the "opam.ocaml.org update job" is having quite some trouble (related to the amount of packages).

To have a concrete proposal that can be discussed:

Let's have a quarterly delete job. Each deletion will be preceeded by a git tag (so someone interested in earlier releases can just opam repo add old-stuff https://github.com/ocaml/opam-repository.git#2023Q4).
For Q4/2023, there will be a tag, and only support OCaml 4.08 (see https://discuss.ocaml.org/t/raising-the-minimum-tested-version-of-ocaml-to-4-05-or-4-08-from-4-02/12464) and opam 2.1 (to be discussed what the minimum opam version should be, certainly 2.0)
Also, all "available: false" packages will be removed
opam admin check --installabe will be executed and evaluated whether the output list should be removed
The upside is: smaller repository, fewer packages -> fewer work on all clients
Additionally, for each package, only the latest patch version will be retained: i.e. if there's a package with versions 0.1.0, 0.1.1, 0.2.0, 0.2.1, 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1 - only 0.1.1, 0.2.1, 1.0.2, 1.1.1 are kept.
Exemptions for the above rule are the following set of packages: ocamlformat
After removal, the output of opam admin check --installable will be evaluated
If there's breakage, old packages can be revived (and added to the above set of exemptions)
since we now have a path how packages are removed, any maintainer/author can opt in to add their package (+version) to the set of "to-be-deleted" packages (at the end of each quarter)

Yes, this will break "opam lockfiles" - in theory at least. In practise, feel free to point to hyperlinks of lockfiles that are broken by such a mechanism (and yes, I really really think these lockfiles need to be fixed to include the repository commit so that they are sustainable).

How can we "discuss" such a policy? Is this issue good for it? How can we reach something that we then act on? Or will we just sit down and wait until it became so annoying that nobody is using it anymore?

kit-ty-kate · 2023-12-07T13:43:22Z

On opam’s side we’re thinking about adding a compressed format for the repository after 2.2.0 (ocaml/opam#5648). Such format would eliminate the issue for opam update at least.

For the solver/clutter/git-growth/… though the problem still remains and removing packages could be a solution. It is also to note that we currently have nobody knowledgeable enough to work on the solver on a regular manner.

samoht · 2023-12-07T15:51:29Z

What I do not really like with tags and saying that you can always look back at the history to install old packages is that we are constantly improving the quality of the metadata of opam-repository. So when you use an old opam-repo, you loose these improvements unless you have a (custom) merge of this old repo with the live repo.

My ideal workflow would be to have layered repositories with a well-defined merge strategy - so our automated system could easily test old packages with recent metadata fixes.

samoht · 2023-12-08T09:26:30Z

So a few more details on what I have in mind. I'm describing an hypothetical worfklow - we can get there incrementally or not.

We have 3 official opam repositories instead of one: unstable/stable/archive
optionnaly, there's a new opam field x-opam-repository-status that contains unstable/stable/archive (if we have want the opam client to do something clever about those)
optionanly, there's a new opam field x-opam-repository-expiry with an expiry date for a package
every opam repository has a different policy:
- unstable: no gatekeepers - a very light CI ("it lints and it builds on ubuntu+latest OCaml release, no x-platform checks, no revdeps"). Strong encouragement to have an expiration date for the packages
- stable: same as today but with all the unstable/old/deprecated packages removed. The CI runs all of todays' checks, but just on the stable repo.
- archive: all the old/deprecated packages. No (or very few) CI checks - archive packages are supposed to always build.
the lifecycle of a package could be:
- people can submit new packages to unstable and stable (these correspond to different level of CI/process checks)
- optionally, there's a weekly/monthly bulk build of unstable packages with extra CI check - the ones that pass the extra CI checks (including revdeps with stable packages) are submitted in bulk to the stable repository
- there's a weekly/monthly bulk build of packages "on the deprecation list", that we want to delete from stable and submit to archive. The CI runs with stable+archive. We need to fix the constraint before adding anything into archive (ie. archive packages should always build). This is super important when there's a new OCaml release (we should check that it doesn't break archive packages and if yes, add the new constraints). When the packages are added to archive, they are removed from stable
we could host the 3 repositories as different branches in ocaml/opam-repository

What I am trying to get is some kind of guarantees that unstable/old/deprecated package metadata continue to somehow be maintained but without putting too much hassle on our CI infrastucture and on opam-repo gatekeepers. And I'm also keen to have a system where "submitting a simple package for fun" doesn't add more load to our system/process.

What do you think?

arbipher · 2023-12-13T05:43:08Z

I am new and started volunteer work in opam admin a few weeks ago. I haven't done much yet except for just attending a few weekly meetings. I shall have more time from this week. I am so glad to see this issue discussing the scaling problem. Here are some topics appearing in your discussions and my opinion.

Retiring old packages

I don't agree with retiring old packages directly. The direct reason is we cannot guarantee the packages will observe the plausible invariant from semantic versioning so that e.g. projects or libraries depending on 6.0.0 can always run without problems after changing dependency to 6.0.1. Reproducibility is such a painful problem.

However, I do agree that we should suggest to use a better alternative of a package version. No matter when users try installing a package with a non-optimal version, or opam (.opam) detects a non-optimal version is specified. But does it require the user to always have a most recent view of the latest opam-repo?

So my point is the package versioning suggests a good package choice but not reliable. That's the reason lockfiles are widely used.

Using GitHub as the backend

Another unclear point to me is when evaluating a design suggestion above, shall we assume that opam repo will stick to using GitHub (git) backend or is it also discussable? It will be helpful to consider whether a suggest is mainly for better design or a better implementation.

We have 3 official opam repositories instead of one: unstable/stable/archive

It has some debian flavor but I have to say using opam where there is just one official central repo is much easier than using ubuntu, with which I have to manually tune it the src-list for many times. And there do have some popular repo e.g. https://coq.inria.fr/opam/released. Does the repo kind (unstable/stable/archive) also apply to other repoes, or is it actually a package metadata?

CI burden, Package author burden, Opam repo admin burden

I would like to observe for some more time and hand on more work to figure out where the CI burden and repo admin burden comes from.

I think x-opam-repository-expiry may increase the Package author burden because it's too complicated and rarely possible to anticipate. I may lose some context or experience for this.

I agree with the fact that CI sometimes runs slower to block things. However, in my very limited observation, some CI issues due to the service itself could be improved (accidental error). Some CI errors are caused by package incompatiblity (essential error), and they do rescue users from suffering from this problem. That's one big achievement of the current admin team and opam repo workflow outperforms many other package management tools I have used, in my opinion. I don't think changing the CI arrangement can reduce this problem. Or maybe it helps, if more packages are staying the unstable without moving into the stable then the overall CI check can be reduced.

If we can offload some checking on the publishing side, some burdens are transferred from the CI/admin side to the author side.

p.s. In recap, I am very interested this problem for long time. I have some plan to survey (and surveyed a little, but not sort them out yet) more package managers especially for programming languages.

mro · 2023-12-13T08:59:10Z

My take on scaling cares about growth and reliability. Growth There is no such thing as eternal growth, so if we design for a liveable future, we should reach a saturation in size. IMO this is simple via expiration. It can be 10 years for "LTS" versions (e.g. the last one before a major/minor jump) but it has to be less than "eternity". There is no maintenance promised during these 10 years, just mere existence. A HTTP 200 instead a 404. The eol date should be in the metadata inside the tarball (and a http header). Who needs it longer (we come to that in a second), should be able to easily mirror/vendor single version tarballs. Reliability IMO we should embrace reproducible builds. This means we need reliably retrieveable sources incl. toolchain. Certain versions may do, it doesn't have to be any version. But once declared such (think 'best before Jan 1st 2040) it must to be available under the same url (or redirected). No need for 5 nines availability and neither bandwidth nor latency shouldn't matter too much, as you can easily cache/download stuff. Maybe signing the tarballs may be useful. (Web of trust, uh) In the end it's an url registry with signed, expiring entries. For reliability (legal, technical, moral) we shouldn't tie ourself to any third party concerning the build toolchain. GitHub may have been fashionable recently, but IMO the opam repository should be consumable without touching billionaire-run infrastructure let alone namespaces. I would want my project to be buildable by peers in Teheran, Moskow, Paris, Beijing and New York alike. And if not, it should be our decision (ocaml.org) and not a billionaire-clerk's. Monitoring may be helpful. Hope this makes sense, Marcus

…

On Tue, 12 Dec 2023 21:43:20 -0800 Weng Shiwei 翁士伟 ***@***.***> wrote: I am new and started volunteer work in opam admin a few weeks ago. I haven't done much yet except for just attending a few weekly meetings. I shall have more time from this week. I am so glad to see this issue discussing the scaling problem. Here are some topics appearing in your discussions and my opinion. **Retiring old packages** I don't agree with retiring old packages directly. The direct reason is we cannot guarantee the packages will observe the plausible invariant from _semantic versioning_ so that e.g. projects or libraries depending on 6.0.0 can always run without problems after changing dependency to 6.0.1. Reproducibility is such a painful problem. However, I do agree that we should suggest to use a better alternative of a package version. No matter when users try installing a package with a non-optimal version, or opam (`.opam`) detects a non-optimal version is specified. But does it require the user to always have a most recent view of the latest opam-repo? So my point is the package versioning suggests a good package choice but not reliable. That's the reason lockfiles are widely used. **Using GitHub as the backend** Another unclear point to me is when evaluating a design suggestion above, shall we assume that opam repo will stick to using GitHub (git) backend or is it also discussable? It will be helpful to consider whether a suggest is mainly for better design or a better implementation. > We have 3 official opam repositories instead of one: unstable/stable/archive It has some debian flavor but I have to say using opam where there is just one official central repo is much easier than using ubuntu, with which I have to manually tune it the src-list for many times. And there do have some popular repo e.g. `https://coq.inria.fr/opam/released`. Does the repo kind (unstable/stable/archive) also apply to other repoes, or is it actually a package metadata? **CI burden, Package author burden, Opam repo admin burden** I would like to observe for some more time and hand on more work to figure out where the CI burden and repo admin burden comes from. I think `x-opam-repository-expiry` may increase the _Package author burden_ because it's too complicated and rarely possible to anticipate. I may lose some context or experience for this. I agree with the fact that CI sometimes runs slower to block things. However, in my _very limited observation_, some CI issues due to the service itself could be improved (_accidental error_). Some CI errors are caused by package incompatiblity (_essential error_), and they do rescue users from suffering from this problem. That's one big achievement of the current admin team and opam repo workflow outperforms many other package management tools I have used, in my opinion. I don't think changing the CI arrangement can reduce this problem. Or maybe it helps, if more packages are staying the unstable without moving into the stable then the overall CI check can be reduced. If we can offload some checking on the publishing side, some burdens are transferred from the CI/admin side to the author side. p.s. In recap, I am very interested this problem for long time. I have some plan to survey (and surveyed a little, but not sort them out yet) more package managers especially for programming languages. -- Reply to this email directly or view it on GitHub: #23789 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- mro ***@***.***>

hannesm · 2023-12-13T15:08:38Z

@arbipher you say "lockfiles are widely used" -- would you mind to elaborate on that? From what I can see, some people use lockfiles. Do you have a specific insight in which domain "lockfiles are widely used"?

There are some loud voices on "lockfiles" that prevent any progress on pressing issues since "lockfiles may break".

FWIW, if you're keen on reproducibility - lockfiles don't achieve this at all. You'll need to put some more effort (and include all the metadata into your build-information, together with environment variables and system packages that have been used). If you're fine with "sometimes you may be able to reproduce the same version / behaviour", obviously lockfiles are fine.

arbipher · 2023-12-13T15:28:36Z

@hannesm Ah, I realize I made a vague word. I mean lockfiles are widely used for projects within other package managers e.g. npm, rubygem, cabal, etc. I don't observe it's widely used in opam.

I don't concern about reproducibility most of the time except for some manual instructions for building from source (in preparing research artifact). I do agree with your saying that lockfiles are far from ensuring reproducibility, but just a freezing of explicit package dependencies.

reynir · 2023-12-19T12:23:09Z

Surely it must be possible to comb through opam-repository.git to get a maximal opam-repository with the latest and greatest metadata so your lock files don't break? Given that this problem is solvable I think it shouldn't block removing packages.

Personally I have at least two packages I would like removed. They are let-if and ssh-agent-unix. The former was a fun experiment to show that you can get something similar to the "if let" construct from e.g. Rust, but it is not something I would use or recommend anyone using, and I have not heard of or found any usage of this library. The latter was accidentally published due to dune-release by default publishing all opam packages in a repository. The initially published package was of very poor quality (I didn't intend on publishing it), I don't think anyone uses it and I don't really want to maintain that package (though I am happy to maintain ssh-agent). I am sure there are many other packages that are in a similar poor state.

Given the many, many CPU cycles, bandwidth and energy being wasted on this due to the number of users I think it is important to take a decision sooner rather than later. I believe this would make a great impact towards the carbon footprint policy, too.

mseri · 2023-12-19T12:51:17Z

Personally I have at least two packages I would like removed. They are let-if and ssh-agent-unix.

Even though this is a partial answer, the removal can already be achieved with available: false. Removing the packages themselves breaks all macos switches due to an issue with macos patch. It should be fixed in opam 2.2

PS I have not yet added a comment here since I don't yet have a clear vision on how to make it work properly, but I am following the thread and I would like to reach a good and scalable solution

hannesm · 2023-12-22T13:52:52Z

Even though this is a partial answer, the removal can already be achieved with available: false. Removing the packages themselves breaks all macos switches due to an issue with macos patch. It should be fixed in opam 2.2

That' makes me sad. I thought that was something of the past, and would hope that such a bugfix would be ported to the 2.1 series and released as soon as possible.

While a available: false "marks it as not available", still the opam file is around, needs to be created on disk, and needs to be parsed. Thus, it is a rather CO2-intense "solution".

kit-ty-kate · 2023-12-22T14:01:38Z

It should be fixed in opam 2.2

It is not currently slotted for 2.2.0 (though that could change) but could be slotted for 2.2.1. This is ocaml/opam#5400 and it needs a review from @dra27 for Windows who is not available at the moment

gridbugs · 2024-01-09T01:05:40Z

TL;DR I suggest approaching package management much more like debian and providing a community repo with which is entirely maintained by package authors.

I think there's some tension around opam being held to the expectations of popular language package managers like npm while opam's philosophy is much closer to that of debian's package manager.

As an npm user I expect old packages to always be available (possibly with loud warnings on install) and I expect to release my own packages with very little effort to myself or npm's maintainers. Further, I don't expect npm to handle my package's CI, and I accept the risk that I might accidentally install a malicious package from npm. All the maintenance burden is placed on package authors so the human cost of repo maintenance is distributed in a way that scales as the repo grows.

As a debian user I expect that every package in its repo has been vetted and tested by the debian repo maintainers (whom I implicitly trust as a debian user), but I don't expect to be able to install old versions of a package. Since the repo maintainers choose which packages are in the repo, and there are fewer versions of each package in the repo, the human and CO2 cost of maintaining the repo is limited.

The debian repo isn't expected to grow very quickly as new software that's both useful and mature enough to consider including doesn't appear that frequently, however as OCaml becomes more popular it's likely we'll see the rate of new packages being released go up. Opam attempts to have the security benefits of debian, but it's expensive to scale since random people can suggest to add new packages via a PR which creates work for the human maintainers and every new package in the repo adds an ongoing maintenance cost. There's no blessed alternative way to release OCaml packages and so all the new users who want to make their hobby projects available by releasing them will be inadvertently adding to the repo maintenance burden.

I propose three things:

Lean into the debian-ness of opam to reduce the repo maintenance cost. Include a smaller set of packages and don't keep old versions of packages in the repo. To allow reproducible builds, encourage users to use lockfiles either with opam-monorepo or dune (once it gets lockfiles). Lockfiles need to include the source urls of the packages they download since the opam repo won't include old versions of packages.
Every 6 months or so cut a new stable release of the repo, possibly adding/removing packages and including major updates to packages. The only other times a package will be updated is for bugfixes. Maybe do LTS releases less frequently that are maintained for longer.
Make a blessed community repo with no human checks or centralised CI. Users who just want to release a hobby project but don't expect anyone but themselves to use it will still be able to release their package without incurring any maintenance cost. As with npm, packages are released instantly (no review process), old packages are never removed, maintenance is done entirely by package authors, and users install packages at their own risk. Even if we just did this and not the previous two steps, I expect this to reduce much of the maintenance burden as new users come to the language.

samoht · 2024-01-09T12:45:59Z

@gridbugs what you propose is similar to what I am saying in #23789 (comment). Any specific difference you want to highlight?

This is also compatible with the short-term goal of triming the central repository.

So, I propose we aim for the following final state:

3 branches in opam/opam-repository: bazaar, main and archive
a straightforward CI set-up for bazaar (just run one distro / + the latest OCaml version / no rev-deps) and archive, no change to the CI for main

To get there:

we move all the packages suggested by @hannesm from main to archive (and we document that policy, ideally with some automatic tools to help do this more automatically in the future)
We make one bulk build for all the newly archived packages to check if there area easy build failures that we can fix. We mark everything else as "allowed to fail" or something like that. I'm happy to help triage/fix this, as I did a bunch of bulk build fixes in the past.
we make sure dune-release and opam-publish can easily publish on the bazaar branch
we make sure running release readiness checks work (or any large-scale tests when we want to test for runtime changes) when main+archive are used together

WDYT?

gridbugs · 2024-01-10T00:51:46Z

That generally sounds good to me.

A small thing but let's not call it bazaar as I doubt many people will get the reference. What about community or unstable. Maybe also rename main to stable or core or something since main is just the default name and its semantics in the context of the opam repos is unclear.

we make sure dune-release and opam-publish can easily publish on the bazaar branch
I really like this idea! I'm a huge fan of the commands in npm and cargo that let you publish a new package or version with a single command and no additional steps. I'd love to have this for ocaml.

@samoht What do you think about the idea of cutting official releases of the stable repo semi-regularly and reserving breaking changes for those releases? Kinda like how nixos has two big releases a year but also has an unstable branch you can follow if you want to subscribe to the rolling release model.

c-cube · 2024-01-10T02:56:45Z

@gridbugs #23789 (comment)

I hope you're not suggesting that, if a user publishes a new version of their package one day, they might have to wait months before having it available on opam?

If it's only a cadence for removing old releases that are subsumed, I imagine it can be useful, though.

gridbugs · 2024-01-10T03:43:49Z

I hope you're not suggesting that, if a user publishes a new version of their package one day, they might have to wait months before having it available on opam?

Only if they want to release their package into the stable repo. I'm proposing to use the unstable/community repo for quick releases and the stable repo for slow, debian-esque releases.

Leonidas-from-XIV · 2024-01-10T09:55:00Z

There are some loud voices on "lockfiles" that prevent any progress on pressing issues since "lockfiles may break".

As someone using lockfiles, I'd say that lockfiles done right is not an an aspect that creates additional problems. True, if you just have a list of packages and versions (opam export style) pulling a package will break but from experience I saw plenty of "locked" builds break because of conf-package changes.

The approach in both Dune and opam-monorepo lockfiles doesn't have a problem with packages disappearing from the repo because once locked neither of them need access to the opam-repository. opam-monorepo doesn't because it assumes all packages can be built with dune, Dune doesn't because it copies the build instructions from the packages at lock-time.

Of course, creating a new, updated lockfile for a project that depends on potentially removed packages is still a problem, but it is the same problem as you'd have with opam install ..

hannesm · 2024-01-10T09:55:29Z

I disagree with the approach to split "opam-repository" into three branches. My intuition is, similar to Debian, this mainly adds confusion to newcomers, tardiness in getting software out.

What I value about the opam-repository is the quickness of updates (bugfixes, new packages), how easy it is to install a package released yesterday, and also the impressive high quality (thanks to both manual checking and CI systems). I don't want to loose any of these properties.

From a package authors perspective, I'd be confused whether to pick the "bazaar"/"community" or "stable" branch to submit something. How would I decide? As an opam user, how would I decide which branch to use? As a "zero-configuration CI system", which branch would be used (why stable? why community?)?

If we're moving to a repository where it is fine to remove (or archive) packages, it is fine to submit to the "stable" branch, or am I misunderstanding something?

Goal

The goal I have in mind, since the start of this issue, is to reduce the repository size. The impact is manifold:

less CI work of the opam-repo-cI (fewer dependencies / reverse dependencies),
less work for the opam.ocaml.org automation (which does something for each package),
less work for the documentation generation on ocaml.org,
fewer data to be transmitted (from opam.ocaml.org to each client) and preserved (on the client disks),
this also means fewer work for the opam client (less files to read and decode, less work for the solver),
also importantly fewer human hours spent on "adjusting upper bounds".

And all of that while keeping the user experience (and workflows/tooling) for (a) package authors and (b) OCaml users the same (for 99.9% of users).

Even though I don't see much value of an "archive" branch, I'm fine to have this around, and instead of "removing packages", keeping them in the archive branch (so people can do bulk builds and fix packages). The workflows could be:

a package deletion is scheduled, and done on the main branch - this doesn't impact the archive branch
a package is added/modified on the main branch -> here the diff is applied to the archive branch (automatically, if it cannot be applied, a GitHub issue is opened with the failure output) - this can be merged manually or automatically, or collected and merged in bulk after a monthly bulk build.

Things needed

This also means that not much needs to be changed:

decision to be done about the archive branch (should it be kept up to date for each merged PR, or every month? who's in charge and having the capabilities and hours to take care of it?) -- likely best those people who use that branch and want to keep it around
the system (maybe a GitHub action?) that kicks in once a PR is merged to apply the diff to the archive branch needs to be developed (once the semantics are settled)
the bulk build (AFAIU these are invoked manually) is triggered periodically for the archive branch (and someone - eventually the same people who care about the first item - should observe the output and decide on changes needed or when to merge)

There's explicitly nothing needed in terms of "adding things to dune-release/opam-publish" or "revising documentation for package authors / opam users".

Please let me know what you think, and which goal(s) you have in mind that are not covered above. Thanks for reading.

hannesm · 2024-01-10T10:08:26Z

There are some loud voices on "lockfiles" that prevent any progress on pressing issues since "lockfiles may break".

As someone using lockfiles, I'd say that lockfiles done right is not an an aspect that creates additional problems. True, if you just have a list of packages and versions (opam export style) pulling a package will break but from experience I saw plenty of "locked" builds break because of conf-package changes.

I'm not sure what "opam export" is, at least with opam 2.1.5 this is not a valid subcommand. Are these 'plenty of "locked" builds' publicly available? Or are these locked builds locked away from the free and open source community?

The approach in both Dune and opam-monorepo lockfiles doesn't have a problem with packages disappearing from the repo because once locked neither of them need access to the opam-repository. opam-monorepo doesn't because it assumes all packages can be built with dune, Dune doesn't because it copies the build instructions from the packages at lock-time.

Maybe you want to look into "opam switch export --full --freeze" as well - here you get something independent of opam-repository, including all patch files, everything versioned.

From my experience, opam-monorepo does some intermediate step where it uses "opam lock" (maybe the "opam-monorepo lock" command?), thus if you consider the "lock" and "pull" step separately (and exchange the output of "lock"), you can get stuck.

I've no clue about "dune package management" and their lockfiles. But I trust you to have a good sense of what is needed in there to be independent of opam-repository changes.

Leonidas-from-XIV · 2024-01-10T10:31:21Z

I'm not sure what "opam export" is, at least with opam 2.1.5 this is not a valid subcommand.

Excuse me, I misrembembered the command, I meant opam switch export.

Are these 'plenty of "locked" builds' publicly available? Or are these locked builds locked away from the free and open source community?

These locked builds are locked away from the FLOSS community because the software in question was a proprietary CRUD backend. However, I don't think that this is an issue, I believe we should also have the OCaml ecosystem work for proprietary software. And selling OCaml to your managers is hard if you need to fix a supposedly locked docker build every 2 weeks because some (well-meant!) change in opam-repository made the docker build fall over.

Maybe you want to look into "opam switch export --full --freeze" as well - here you get something independent of opam-repository, including all patch files, everything versioned.

Ah, yes that's nice indeed! Unfortunately also well hidden.

In any case, I don't want to derail this into a discussion into lockfiles, just wanted to point out that for lockfiles and locked builds the way opam-repository is scaled does not make that much of a difference in my opinion.

bn-d · 2024-01-23T23:38:52Z

I am not convinced (or simply did not understand at all) the debian-ish solution works, unless the repository maintainers take the full responsibility of patching stable packages when new compiler releases.

Are we allowing multiple versions of the same package co-exist in the stable repository?
Which version of the OCaml compiler does the stable target? Are we going to have 1 stable repository per "supported" OCaml version or only the latest? Because of package compatibility, the stable repository could very a lot depending on version.
If we are allowing multiple versions of the same package co-exist in stable, are we going to do all supported OCaml compiler versions * all versions of packages in bulk build? (And this really not debian-ish
If we are only allowing 1 version per package in stable, what if two popular packages pinned two different versions of the same dependency? What is the resolution mechanism in this situation?
Do we have guarantee that a package exists in the current release of stable will be in the next release? If yes, how do we encounter the situation when a compiler upgrade breaks a large bunch of packages, and some of the owners do not have time to fix the build before the next release?

My 2 cents would be to

Have a repository per OCaml compiler version
When start a new compiler version, start it from an orphan branch, so no commit history. Only includes package:
- In the last version AND
- Buildable in new version AND
- Had been used by someone in x amount of times
- Potentially can limit the number of version moved to new repository, and only move the latest or required version
On the client side, there are git magics (i.e. shallow clone or sparse checkout) can be done to save disk spaces.
When you publish a package, it only publishes to compatible repositories.

Or just go's approach.

toots · 2024-01-24T20:27:15Z

Well I totally underestimated how early 8am feels here.. :-/ Is there some notes from the meeting somewhere? I looked at opam/opam-repository on discord but haven't found anything.

mseri · 2024-01-24T21:34:07Z

We have very detailed notes written down, we will summarize all the details in a coherent text to share in the next few days. We made some good progress imho, and followup meeting is scheduled for Feb 7.

yawaramin · 2024-01-24T22:30:25Z

Sorry I missed the meeting. I don't know if dune's upcoming package management feature was discussed, but I feel it is relevant in this context eg ocaml/dune#7680 (comment)

raphael-proust · 2024-01-25T08:59:21Z

Here are the notes for the meeting we had discussing the future of the opam-repository on 2024-01-24.

I take responsibility for errors in transcriptions and such (I was involved in note-taking and in editing for readability). Please let me know if I misrepresented some points that were made and I can edit this comment.

Agenda

Introduction, reminder of the current issues related to scaling
Identifying interested parties (package users, package maintainers/publishers/etc., opam-repository maintainers)
Identifying needs, wants, and cants for each interested party
Discussing change proposals w.r.t. the list of needs/wants/cants/etc.

Notes

Present: mseri, raphaël, Marcus, Hannes, rjbou, 0x265e, Shiwei, Kate, Marek, Thomas, Boning, reynir, dmitri, dinosaure, V
Note takers: raphaël, mseri

Introduction

Opam is unique because of the high-quality of packages (high installability rate, etc.)
Publication of packages is fast (compared to Debian and other repositories with good checks)
The repository has been stable (updated metadata and installability of packages)
Some big packages are hard to publish updates for because of the CI can’t scale infinitely (e.g., foundational packages such as dune)
Some packages have been broken for a while, or have flaky installability
The size of the repository (in number of file) makes synchronisation slow and costly
There is an old unwritten policy to not remove packages
- but some maintainers have asked us to be able to remove their outaded or unmaintained packages
We can use the repository for large scale tests of the compiler (using old packages)
Many failures are due to depexts, maintaining them takes time from opam-repository maintainers

Can we keep the nice properties (high-quality, quick publication, stability) whilst avoiding the issues of scaling

Interested Parties

Package users (people who run opam install directly or through other tools)
- industrial users often have the resources to roll out their own fork of opam-repository
Package maintainers/publishers (people who create and release the packages on the opam-repository)
opam-repository maintainers (curating the submission of the releases on the opam-repository)
tooling maintainers and users of these tools, for example:
- the CI used by opam repository (aka. opam-repo-ci)
- other CI tools (e.g. github actions or others)
- tools to publish the packages or opam/dune plugins using data from the repository
- alternative installers like esy
- ocaml/ocaml developers and associates

Needs/Wants/Cants of the interested parties

for package users:
- keep it simple: should be simple to explain and not cumbersome to use
- keep quality high
for package users and opam-repository maintainers and tooling maintainers:
- slim down the repository (number of files, amount of data, speed of update)
repository maintainers:
- cannot handle more work in terms of volume and complexity
package publishers:
- keep release publications speed
- ability to remove old versions of packages? deprecate versions? make non-available by default?
- some people do not want the burden of maintaining their packages' reverse dependencies and want less checks
  - note from opam-repository maintainers: this is not a problem for the opam-repository maintainers to keep track of in most cases
ocaml/ocaml developers and associates:
- ability to test on the large number of projects (e.g., ocaml 5)

Proposals

solution 1: have two classes of packages: the maintained ones and the "unmaintained" ones

this can either be implemented using a separate repository or separate branches
- separate repository makes the main repository leaner in the long term for publishers for example
- but a separate repository may be more confusing (two distinct sets of MRs, two distinct sets of issues, etc.)
in this case the maintained would keep the full CI and the unmaintained will not (or have only a minimal one)
for the unmaintained one can use "strict" bounds (add the bounds when archiving to make it so the archived packages keep installability) or add a hash of the commit when this was removed so that can be tested automatically against a specific version of the repository (x-archival-commit-hash field)
- Note added by Kate after the meeting: maybe strict bounds with the exception of the ocaml package so ocaml/ocaml developers and associates can test newer versions; Note added by raphaël after the meeting: maybe we still apply strict bounds for ocaml but we make an opam-ed script to restore bounds to pre-archival state?
opam can already check dependency constraints to ensure coinstallability and then tell us what can be moved periodically (say every 30 days)

solution 2: have separate branches for separate ocaml versions

when a new ocaml version is created, only the packages from the previous versions which happen to build on the new version are added to a new orphan branch
each branch only has high-quality packages
negative: it requires package publication to use multiple MRs (one per compiler version that it is compatible with) which makes publishing and maintenance harder

solution 3: use commits/tags in opam git repository

this can be used when someone wants a certain older package that would be not available anymore
negative: this will add more burden on user as users would have to search for which commit or tag the package they want is, and this gets exponentially complicated if users are looking for more than one package and opam (the tool) is trying to reconciliate two different repositories at different times
counter-argument: there could be ad-hoc tools to handle those problems

solution 4: move the broken packages (uninstallable, test failings, etc.) to an archive repository

this solution is similar to the first one (splitting the repository into two "maintained" and "unmaintained" packages) but only moves broken packages to the archive repository
- this is less ambitious but can be a first step towards solution 1
if the CI does not find breakages we can possibly move the old packages automatically to the archive (could work even if people don't use semantic versioning)
- Note added by Kate after the meeting: this is not possible to do automatically without having an extremely reliable CI, which is not the case currently

solution 5: have two classes of packages, but it's "experimental"/"playground"/"lite" packages and more thoroughly tested packages

EDIT: there are clarifications in this response

but why not three? (the third being "no test" repository)
- we should probably not invite people to publish packages without CI work (the playground repo)
having stable releases like debian periodically with the compiler that are super high quality

Miscellaneous notes

which versions of the compiler should be supported by opam-repository?
- we could say that all compilers < XYZ are no longer supported and then we move the packages installable only on those compilers in the archive repo
archival can break projects, including those with lockfiles, but this is already the case when packages with security issues are marked as unavailable or when tarballs disappear
- lockfiles could include hash of opam-repository commit in order to avoid the possibility of breakage (e.g. this is being done in opam-monorepo)
- Note added by raphaël after the meeting: similarly it could be recommended (or at least documented) how to set up your project to build against a given commit hash of the repo

Summary

The consensus seems to be on splitting opam-repository into two:

One repository containing "maintained" packages (opam-repository, aka. opam.ocaml.org)
One repository containing "unmaintained" packages (an archive repository, name to be determined)

The specifics (tools, which packages would be moved, …) need to be figured out and we are proposing a next meeting to discuss that in 2 week (7th of February, same time, same place).

Preliminary agenda for that meeting:

Are there any major push-backs against the consensus found at the end of the last meeting?
Are there any kind of comments on the consensus found at the end of the last meeting or on the way the meeting was ran?
List the tools to help define the unmaintained packages (e.g. opam admin check --obsolete)
List the CI systems that will need to be done
Write a new constitution for opam-repository specifying what to do when a maintainer wants to remove or move their package
TBD

avsm · 2024-01-25T11:55:08Z

Thanks Raphael, very useful notes. At a quick scan, all the "wants" seem compatible with my proposal over in https://github.com/avsm/opam-repo-roadmap-thoughts.

@mseri has given me detailed comments on that one in September (in a hackmd which I can no longer find -- @mseri, might you link to that if you can dig it out?), and I'd like to refine that spec towards consensus with this very useful meeting.

samoht · 2024-01-25T13:58:49Z

Thanks for the notes. Just a quick comment:

solution 5: have two classes of packages, but it's "experimental"/"playground"/"lite" packages and more thoroughly tested packages
but why not three? (the third being "no test" repository)
we should probably not invite people to publish packages without CI work (the playground repo)
having stable releases like debian periodically with the compiler that are super high quality

Something has been lost in the retranscription of the notes. I suggested extending solution 1 with another repository containing bleeding edge/unstable packages and defining clear policies (and CI checks) to move packages between those three sets. I was also suggesting that we add more sets (some of them temporary, some of them not) in the future (for instance, to test package universe before compiler release or to do bulk builds of large package releases). So it would be nice if the technical and process solutions we develop could be scaled to more than two sets.

But I also agree that before having three sets, we have 2, so solution 1 is a good first step in that direction (as long as we don't ossify in technological and process choices that will stop future evolutions).

mseri · 2024-01-25T14:40:38Z

@avsm found: https://hackmd.io/@mseri/By49zJh6n

@samoht I think that was my mistake, I had misunderstood at first. I think you make a valid point that goes a bit also in the direction suggested by @bn-d: the moment we see that it works well with 2, if we have a different or finer strategy that makes sense and is still seamless and easy to use and maintain, then we can keep iterating further on the idea.

avsm · 2024-01-26T14:45:10Z

Thanks @mseri! If anyone else wants to add comments over mseri's HackMD above, I can try to present this at the next meeting briefly.

raphael-proust · 2024-01-29T09:34:14Z

Thanks Raphael, very useful notes. At a quick scan, all the "wants" seem compatible with my proposal over in https://github.com/avsm/opam-repo-roadmap-thoughts.

I don't think I understand the merge-queue process very well. I get the general idea, but I fail to see how it makes life easier for opam repository maintainers. Specifically, I don't understand:

they can safely send reasonable-looking PRs to the merge queue without blocking on complete reverse dependency tests

Why don't we need to block on rev-deps tests? Are we just sending everything to the mergeq and then writing fixes to it afterwards based on the results of the mergeq CI? Is it meant to be simpler because we can write a batch fix for multiple submissions or is there another reason that I missed?

the merge queue can publish a set of differential failures against its baseline trunk, and fixes to metadata (such as upper bounds) can be applied against the merge queue.

Is this automation for the previous bullet point (i.e., automation for fixing the revdeps failure that happen in the mergeq because we merge without checking them)? Who's maintaining it?

multiple big package releases would all be tested simultaneously

Is it not going to make it more complicated for opam-repo maintainers because there'll be errors mixed from multiple different sources? Like: “Is this failing because package foo was updated or because package bar was updated?”

All in all, the mergeq process seem to make it simpler for CI maintainers. And as such it's definitely worth considering because the CI is a major issue. CI maintainers don't actually appear in the list of people who would benefit from this change. So maybe you meant CI maintainers rather than opam repository maintainers??

I think the mergeq could be a good idea. Especially if it came with solid tooling and a clear plan of what the CI does and doesn't check at different points.

Thanks Raphael, very useful notes. At a quick scan, all the "wants" seem compatible with my proposal over in https://github.com/avsm/opam-repo-roadmap-thoughts.

One of the "wants/cants" is

repository maintainers cannot handle more work (in terms of volume and complexity)

and I think that the proposed “bi-annual versioned releases” might add complexity. Specifically there might come a time when we need to push a fix to such a release because some incompatibility is discovered later. There might also be some added pressure to work on (dune or opam or whatever) so that it makes it before the cutoff date of the release.

I'm wondering whether these releases can be managed as a separate project from the opam-repository 🤔

hannesm · 2024-01-29T10:49:03Z

Thanks for the notes @raphael-proust. I appreciate to have a next meeting with the proposed agenda.

@avsm IMHO your proposal contains some ideas to make the "repository maintenance" more efficient. As far as I followed the discussion in the meeting, we're not there yet - and I'd appreciate to continue the discussion (instead of diverging to discuss a technical "merge queue" approach, which I also think is orthogonal to where we are at the moment).

kit-ty-kate · 2024-02-20T18:30:51Z

Hi, sorry for the lack of updates and meeting notes for the meeting 2 weeks ago, both Raphaël and I were busy with other things.

Here are the meeting notes:

Public meeting on 2024-02-07

See discuss announce

Agenda

(We didn't really follow the agenda closely. The original planned agenda is still left here.)

Are there any major push-backs against the consensus found at the end of the last meeting?
Are there any kind of comments on the consensus found at the end of the last meeting or on the way the meeting was ran?
List the tools to help define the unmaintained packages (e.g. opam admin check --obsolete)
List the CI systems that will need to be done
Write a new constitution for opam-repository specifying what to do when a maintainer wants to remove or move their package

Roll call

Were present during this meeting:
Raphaël, Hannes, Shiwei, Kate, paurkedal

Notes

No major push-back against the consensus from the last meeting registered on github issue and other channels.
No additional comments on the meeting.
- Question about the number of file limit in the context of deep checkout of the git repo.
  - The .git/ directory size is not 1:1 with the content of the repo.
  - Shallow checkout is possible.
Tools:
- opam-repo-ci and opam-health-check can detect broken packages that we would want to move to the archive repository first
- we need a tool to collect the packages that we should definitely keep (i.e., "packages that are used")
  - pseudo code:
```
needed_packages = ref []
for p in $latest_version_of_each_packages
  for v in $versions_of_ocaml_supported
    needed_packages :=
      p ::
      packages_that_would_be_returned_by_opam_install(p, v) ::
      !needed_packages
```
  - see Fix opam admin check --obsolete opam#5805
  - could miss packages that are used but not used inside of opam-repo
    - we would need to make a public call to declare use, may add some metadata somewhere, maybe have virtual packages we inject when doing the analysis
- should we add extention fields (x-*) in opam to declare intention of the authors/maintainers?:
  - example field contents:
    - semantic versioning: “I will maintain the latest version X.Y.Z and X.(Y - 1).Z or (X - 1).Y.Z"
      - comment: semantic versioning is arguably difficult to define in OCaml
    - usefulness of old versions: “I will maintain the latest version of the package only”
    - expiration date for maintenanceship: “I will maintain this for 1 year only”
    - anything not compatibly with old OCaml versions: “I will not maintain anything further than X OCaml releases away from the most recent”
  - what should be the default?
    - semantic with .Z?
    - move anything that require less than ocaml-version 4.08? (or another version)
  - what to remove?
    - everything not maintained and not needed by another package?
    - everything not maintained and broken?
    - everything not maintained and broken and not easily fixable by updating constraints?
    - everything not maintained and broken and not fixable by updating constraints or adding patches?
    - maybe start with a conservative removal (e.g., "not fixable") and then progressively move towards more removal (depending on feedback)
    - maybe do a spring clean where we ask the maintainers of the largest collection(s?) of packages (NL, mirage, etc.) what value they want to have for their packages' maintenance field
  - when would the fields be added? when a package is broken and we ask the maintainers what to do? how do we encourage maintainers to add the field?
How much CI should the archive repo have?
- Better question: How do we reduce the amount of testing needed in the archive repo?
  - Set hard constraint on all dependencies when the package is moved except for the ocaml package so that people who need to test older packages can try them with newer version of the compiler
Conclusion
- Kate and Raphaël will write a draft resolution and propose it at the next meeting
- next meeting is planned for Wednesday 21st of February, same time
- lots of small open questions but a consensus is emerging

Sorry again for this last minute update. We don't have the draft resolution in a reasonable shape but we hope to have that before the meeting tomorrow for those who can still make it.

yawaramin · 2024-02-20T19:25:13Z

I would really recommend to separate the ideas of package ecosystem health check from package availability in opam-repository. Dropping (versions of) packages from the repo will lead to a decline in trust. But dropping them from a separate opam ecosystem health check will be completely understandable because there are only limited resources at the end of the day.

Scala does this: the package registry ( https://index.scala-lang.org/ ) is separate from the community build ( https://github.com/VirtusLab/community-build3 )

This leaves the legitimate concern of expressing to users that a package has been checked as healthy or not, trying to minimize frustration with low-quality packages. We could implement a 'blue check' next to packages (+versions) which are in the proposed opam community build and passing the health check, and perhaps conversely separate icons for packages which are not in the community build or which are not passing health checks.

Based on this the community build could be super simple–maybe just the latest version of each package on the latest compiler version. Or expand to more versions if desired.

kit-ty-kate · 2024-02-20T19:35:48Z

I'm not sure to understand. What do you mean by "package ecosystem health check" ?

yawaramin · 2024-02-20T19:43:31Z

I mean:

opam-repo-ci and opam-health-check can detect broken packages that we would want to move to the archive repository first

So my suggestion is that, instead of moving any packages/versions into an archive, just remove them from the set of packages that are checked. Ie have a limited set of checked packages–maybe the most-downloaded ones? And give them a nice 'blue check' logo in the OCaml.org package index. This shows that the opam health check has been run on them. For the packages outside this set, remove the guarantee that the health check has been run.

kit-ty-kate · 2024-02-20T20:20:31Z

the goal of this discussion is not to just reduce CI load (it is actually a really small part of it), but to find a solution to the infinite growth of the opam-repository. You can check the stated interested parties in the first meeting notes:

#23789 (comment)

removing them from the set of packages that are check would not reduce the load of extra work for users of the opam tool (performance issues + packages are still visible), publishers (more time downloading the repository) and opam-repository maintainers (getting lost in an infinite repository)

samoht · 2024-02-20T21:52:49Z

@kit-ty-kate, if I remember correctly, a few parties agreed with @yawaramin in that first meeting -- it was mentioned that removing packages from the repository is not the only way to scale our ecosystem -- a lot of package ecosystem lives very well with a few orders of magnitude more packages than opam (I remember @dra27 saying that we should aim for our tools to support as many packages as npm instead of trying to reduce the total number of packages in opam and I agreed with him).

While I'm not opposed to splitting the repositories into various parts to ease some of the maintenance burden in the short term, I really would also like us to think about what will happen next and what needs to be done to ease the lives of opam-repository maintainers (do we need more maintainers? if yes, what is blocking people to contribute here?) and to reduce the size/cost/power consumption of our CI cluster.

Edit: Also, it would be helpful to be super clear and state that "growing the number of available packages in opam" is an essential goal of the opam-repository team -- as this is a proxy metric of a healthy ecosystem.

yawaramin · 2024-02-20T22:27:29Z

While I'm not opposed to splitting the repositories into various parts to ease some of the maintenance burden in the short term, I really would also like us to think about what will happen next and what needs to be done to ease the lives of opam-repository maintainers

Exactly my thought process here. If we are going to chop up the opam-repository into two or more slices, we should recognize that this is a temporary solution and in the long run we need a more practical solution which does not require more chopping up in the future.

It seems that the main issue is that as the repository scales up, more data is being moved around and stored, which is making opam clients and other processes more inefficient. I think we need to find a repository format that does not scale essentially linearly with the number of packages and that is efficient to query, download, and install packages even as it continues to scale up.

Trees, anyone?

avsm · 2024-02-21T11:18:16Z

One thing I've wanted from the very beginning of opam was the ability to refer to multiple repositories from within the metadata. @samoht ran out of time to build it into opam 1.0, but we're only feeling the pain a decade down the road ;-) It would be really nice if we could clone a lightweight "latest version of opam repository" but then bringing in other overlay repositories.

Ideally if we break up opam-repository into multiple versions, it should be possible to have a reasonable way to build combined versions without having to remember all the various repository URLs. Perhaps all that is needed is an opam-repo CLI tool that can combine/split up an opam repo into multiple repositories, and rewrite metadata appropriately so that they are either standalone or overlays. Most of the opam CIs have had this ability to distinguish standalone opam repos from overlay ones designed to be merged with an existing repo.

@raphael-proust to answer your questions above:

I don't think I understand the merge-queue process very well. I get the general idea, but I fail to see how it makes life easier for opam repository maintainers. Specifically, I don't understand:
they can safely send reasonable-looking PRs to the merge queue without blocking on complete reverse dependency tests
Why don't we need to block on rev-deps tests? Are we just sending everything to the mergeq and then writing fixes to it afterwards based on the results of the mergeq CI? Is it meant to be simpler because we can write a batch fix for multiple submissions or is there another reason that I missed?

That's exactly the reason why it's easier. The opam-repo is one big datastructure, and when maintainers apply fixes, they are applied to a batch of packages. Therefore the CI should be building the one datastructure that we intend to merge, and not individual changes. From the maintainer perspective, there will just be a "queue of incoming packages" and then we look at breakages and push the right metadata to make that work into the mergeq.

Right now it's very laborious to spot some revdeps breakage, then open a separate PR to fix those, then look for more breakage uncovered, repeat a few times, and it takes 5-6 PRs to merge a package, and a huge amount of unnecessary CI work. My goal with the mergeq is to reduce the amount of CI time needed by an order of magnitude, by removing all the duplicated work due to the lack of synchronisation around branches.

Is this automation for the previous bullet point (i.e., automation for fixing the revdeps failure that happen in the mergeq because we merge without checking them)? Who's maintaining it?

This will be the opam-repo maintainers, just as it happens today. Except instead of pushing revdeps failures in separate PRs and branches, you should focus the effort on the merge queue. When it passes, it becomes the main branch, and a new set of PRs can be chucked in the queue.

What does "pass" mean? That's why at the very beginning of the doc (https://github.com/avsm/opam-repo-roadmap-thoughts) before anything about merge queues, I put down a set of candidate metrics for how to measure the opam-repo health. I'm looking forward to reading your draft resolution to see how they align up -- I no doubt missed a bunch of ideas that are coming to light from your meetings.

(unfortunately, I've got a standing conflict at 2pm UK time with our faculty meetings at the university, so I can never make the community meetings you are organising during the academic term. I may be able to attend a synchronous meeting at some other time)

mseri · 2024-02-23T17:55:11Z

Dear all,

thanks for continuing the discussion. I attach here the notes of last meeting. In particular I would like to stress the following point from the conclusions:

it would be nice for more people to be present at the next meeting to discuss the push-backs discussed on the ticket and arrive at a more shared and comfortable consensus.
For this purpose here is a framadate link to find the best time and date for the next meeting: https://framadate.org/qD2Pb57B7h6xJ8U4
Please fill this poll as soon as possible to we can set the date for the next meeting.

Discussing the future of opam-repository

Public meeting on 2024-02-21.

Roll call

Were present during this meeting:
Marcello, Hannes, Shiwei, Kate, rjbou

Notes

Discussion about the ongoing discussion on github
- the solution we are currently proposing is not meant to be the final solution and we are all open to discuss and see how the repo will evolve in the future
- however we first need to understand how to remove packages and then do it, since this solves some important problems faced by the community (including but not limited to maintenance, CI speed, opam use of memory and disk space)
- the new ideas proposed in the github discussion are interesting and present a good aspiration to aim at, but they seem to require heavy changes to the current workflows and ecosystem, so they are likely years ahead their realization
- it is good and important to keep the discussiona and planning alive for the future of the opam-repository and in the meantime do what can be done in short time
- it is important to stress that none of the solutions we are proposing here prevents the implementation of the future suggestions as soon as the ecosystem is ready, so they are not mutually exclusive
Concerning the original plan of this meeting: the idea was to fix the ideas for the "spring cleaning" of the repository. We draft below, after the horizontal line, a possible and actuable plan of action.

Conclusion

the draft is not yet fixed but was discussed with and agreed by the people present at the meeting
it would be nice for more people to be present at the next meeting to discuss the push-backs discussed on the ticket and arrive at a more shared and comfortable consensus.
For this purpose here is a framadate link to find the best time and date for the next meeting: https://framadate.org/qD2Pb57B7h6xJ8U4
Please fill this poll as soon as possible to we can set the date for the next meeting.

Short term solution - Draft

Phase 0 (this here draft)

Write and agree on a new policy for opam-repository
- publish it on the opam-repository wiki
Write a discuss post to announce the new policy

Phase 1 (setup + unavailable/broken packages)

Create the ocaml/opam-repository-archive github repository
- Create a barebone README.md describing the project and a repo file
Create a CI for this new repository
- it should simply lint the files
- and check that every dependency constraints are of the form "dep" {... & <= "<latest version of dep available in opam-repository at the time of the PR or lower>"} except for "ocaml"
make a list of known unavailable or broken packages using opam-repo-ci and opam-health-check
- make a PR removing them from opam-repository
- make a PR adding them to opam-repository-archive
- write a Discuss post listing all the packages that are going to be removed, make sure everyone is using gpatch on the relevant OSs (*BSDs and macOS) and wait a week for comments. Then merge both if no reasonable negative comments
  - there is no runtime check for this, only at init
  - the check for gpatch is there only for bsd but not macos
  - action: it would be ideal to fix it on mac, either detecting gpatch in some way say with (patch --version | grep gnu) or git (which is the best option and there is already and open PR for this that would be nice to have backported to opam 2.1)
  - but this has happened in the past and we survived all the time, so perhaps is not the end of the world
  - use announce field to warn all users of the upcoming change
Push a new before-opam-repository-archive tag on opam-repository before any of the removal

Phase 2 (OCaml < 4.05 -- the least aggressive pruning)

Move all ocaml packages < 4.05 (oldest version still in use in the wild according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge
Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge

Phase 3 (spring cleaning)

Write a Discuss post to advertise the extension fields for maintainers to declare intention of maintainshing for their packages (to add to their packages and the tool will take the one in the latest version that has it)
- x-maintenance-intent: "latest" the maintainer will only maintain the latest version
- x-maintenance-intent: "1-major" the maintainer will only maintain the latest X(.Y)?(.Z)? version and (X-1)(.Y)?(.Z)?
- x-maintenance-intent: "1-minor" the maintainer will only maintain the latest X.Y(.Z)? version and X.(Y-1)(.Z)?
- x-maintenance-intent: "all-major" the maintainer will maintain every X for each X(.Y)?(.Z)?
- x-maintenance-intent: "all-minor" the maintainer will maintain every Y for each X.Y(.Z)? (where X is the latest)
- x-maintenance-intent: "all" the maintainer will maintain every single versions
Add support on dune release, opam publish, opam repo ci linter, ...
Ping maintainers of large sets of packages individually so they can add this field to the latest version of each of their packages
Create a script that reads every opam-repository packages, take the latest version of each packages that have the x-maintenance-intent field and lists every versions that are not maintained anymore
use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages

Discussion

The idea is that only the packages with the x-maintenance-indent will be considered for removal when the times come, the others will stay
Alternative is to be aggressive and just remove obsolete with a ping to the maintainers that we can find
This is good as a start but then we need to figure out if we can be more aggressive or if the process can be trimmed
Something to infer the mainenance would be a nice start to improve the workflow in the future

kit-ty-kate · 2024-02-27T17:16:45Z

For this purpose here is a framadate link to find the best time and date for the next meeting: https://framadate.org/qD2Pb57B7h6xJ8U4

Reminder for everyone who wants to come to the next meeting to please fill the form as soon as you know when you are available, so that we can plan when the meeting is going to be.

kit-ty-kate · 2024-02-28T19:14:57Z

Based on the above poll, it looks like most of the main interested parties so far are available on:

Monday the 4th of March at 14:00 GMT (15:00 CET)

so we propose to have the next public meeting at that time and date on https://meet.jit.si/opam-repo-meeting

kit-ty-kate · 2024-03-08T18:45:15Z

Meeting notes (2024/03/04)

Present: Kate, Anil, Marcello, reynir, Thomas, rjbou, Shon, Hannes, Shiwei, Ryan

Yawar was marked present but was in fact victim of a bug, see https://discuss.ocaml.org/t/discussions-on-the-future-of-the-opam-repository/13898/11, if it happens again in the future for anyone else, please ping us on Slack or Discord. We'll also try to keep a look at the Discuss page for people who have neither.

Agenda

discuss pushbacks/comments/…
continue writing the draft

Notes

Clarifications about the draft:

would the archive repo be an overlay over the opam-repository?
- yes. To ensure compatibility between the two repository upper-bound constraints on the dependencies available at the time of move, would be added
would the packages given by opam admin check --installable be removed directly?
- yes, in the draft that is part of phase 1
do we also want to remove packages that takes too long to solve?
- we're hoping that removing older package would improve the solver response, but it could be done in later phases if that's not the case
how often would the spring cleaning be done?
- this hasn't really been talked about but some were thinking of every 6 months or every year

Communications will need to be modified to include some quality metrics (installability, etc.) where it has historically focused on quantity ("we have that many packages and it's growing")

In addition to the repository health (size, scalability, etc.) there are also CI cost concerns in terms of energy use

Should retired packages have their documentation built for ocaml.org?

no because one of the goal is to reduce CI usage and cost associated with running this infrastructure
however in the current state that could create dead-links
- should we redirect to some other version?

Actionable: talk to the CI team responsible for the docs-ci in ocaml.org and see what they say.

If we want to be able to publish users' package faster/more npm-like manner, one of the values for the fields that specify the maintenance intent could be "don't touch my package ever, don't change metadata, nothing"; in this case the package could be moved immediately to archive as soon as it is broken.

When a package is moved to the archive it would be good to have a comment/commit-message/x-field/… that tells people why the package has been moved to the archive

It is useful to have this as a field so that it will be machine readable

Conclusion

it looked like a positive outcome from all interested parties so far
a discussion with the maintainers of the CI infrastructure for ocaml.org should be scheduled or they should be personally invited to the next public discussion
the draft has been polished a bit further and we are hoping for only a couple more meetings to be necessary to finalize it and propose it to the rest of the community
- areas to be polished further are marked with "TODO"

Draft

Phase 0 (this here draft)

Write and agree on a new policy for opam-repository
- publish it on the opam-repository wiki
Write a discuss post to announce the new policy
- advertise it on discord, mailing-list, ocaml-weekly-news, … https://ocaml.org/community
- write the motivation for the change in the discuss post
Reach out the infra team to make sure they are ok with the proposed document

Phase 1 (setup + unavailable/broken packages)

Create the ocaml/opam-repository-archive github repository
- Create a repo file
- Create a README.md
  - explaining the role of the repo
  - explaining the uses of the repo (opam repository add …)
  - explaining the additional x- fields in the files
  - explaining that files in here may be modified to improve installability and such
Create a CI for this new repository
- it should simply lint the files
  - checks that every dependency constraints are of the form "dep" {... & <= "<latest version of dep available in opam-repository at the time of the PR or lower>"} except for "ocaml"
  - checks if the x-reason-for-archival is set
  - checks if the x-opam-repository-commit-hash-at-time-of-archival is set
Add the archive repo to the health-check but run it more infrequently (~1/month)
Make a list of known unavailable or broken packages using opam-repo-ci, opam-health-check and opam admin check --installable
- make a PR removing them from opam-repository
- make a PR adding them to opam-repository-archive
write a Discuss post listing all the packages that are going to be removed and wait a week for comments, then merge if ok.
- in the post, explain the problem with gpatch
  - make sure everyone is either using gpatch on the relevant OSs (*BSDs and macOS), opam 2.1.6 or give an alternative (e.g., rm -r ~/.opam/repo/default/ && opam update default)
  - Will be fixed in opam 2.1.6: 2.1.6 backport commits opam#5870
- use announce field to warn all users of the upcoming changes
  - including a warning for gpatch issue if version <2.1.6
Push a new before-opam-repository-archive tag on opam-repository before any of the removal

Phase 2 (OCaml < 4.05 -- the least aggressive pruning)

Move all ocaml packages < 4.05 (oldest version still in use in the wild according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge
Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge

Phase 3 (spring cleaning)

Write a Discuss post to advertise the extension fields for maintainers to declare intention of maintainshing for their packages
- for usability, it is sufficient to add the field to the latest version of a package and then the tooling will resolve this
- in case of multiple values for this field in multiple versions of a package, the latest version is taken into account
Syntax for the new x-maintenance-intent field:
- the value would be a list of strings matching on version numbers, with special keywords (latest), (any) and (none)
- For example:
  - ["(latest)"] the maintainer will only maintain the latest version
  - ["(latest)" "(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and (X-1).Y.Z
  - ["(latest)" "(latest).(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and X.(Y-1).Z
  - ["(any).(latest)"] the maintainer will maintain every major version X for each X.Y.Z
  - ["(latest).(any).(latest)"] the maintainer will maintain every Y for each X.Y.Z (where X is the latest)
  - ["(any)"] the maintainer will maintain every single versions
  - ["(none)"] the maintainer will not maintain any version
- TODO: this is just a back of napkin type proposal, this should be rediscussed and make sure this proposal does allow to write any type of pattern and the semantic is agreed upon.
- The idea is that only the packages with the x-maintenance-intent will be considered for removal when the times come, the others will stay
Add support on dune release, opam publish, opam repo ci linter, ...
Ping maintainers of large sets of packages individually so they can add this field to the latest version of each of their packages
Create a script that reads every opam-repository packages, take the latest version of each packages that have the x-maintenance-intent field and lists every versions that are not maintained anymore
use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages
- TODO: do we need a mechanism to register interest in specific packages (a way to indicate "I depend on this package even though my work is not published in opam-repository")

Phase 4 (OCaml < 4.08)

Move all ocaml packages < 4.08 (oldest version of OCaml in use for maintained distributions according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge
Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge

Phase 5 (coasting)

Repeat Phase 3 every 6 or so months
Repeat Phase 4 everytime it is deemed that most people and distributions have migrated to a newer version of OCaml

kit-ty-kate · 2024-03-08T18:48:53Z

The next meeting will be on the same day / time again, the week after the next one. If that doesn't fit with your calendar this time, please don't hesitate to tell us.

Monday the 18th of March at 14:00 GMT (15:00 CET) at https://meet.jit.si/opam-repo-meeting

toots · 2024-03-09T01:24:32Z

I haven't been able to attend the meetings due to time zone and life constraints but the plan seems great. Thanks for the hard work.

kit-ty-kate · 2024-03-18T14:04:47Z

The meeting is starting now, is anyone else coming?

mbarbin · 2024-04-01T08:03:58Z

Hello, Thanks for this discussion!

Sorry I am late. I don't have much to contribute on the cleanup plan, thank you for the work there!

I'd like to comment on the future of opam and its growth, and bring up an idea for us to consider. It's in the realm of creating more flexible workflows for users, in a way that scales up. I'm not sure if it's been mentioned before, but I find it interesting and potentially beneficial for the future growth of opam.

What I was thinking of was to create a new repository for opam, which would list custom opam repositories that various users have (examples here, here, etc).

This "meta" repository would be a registry of link towards other repos, so in essence a map from a name, to a git URL. This is an "opam-repos-registry" rather than a "opam-packages-registry".

I am imagining that the process of adding a new entry to this repo could be similar to that of the existing main opam repository. Users would add an entry with a link to their repo, reserving a part of a new global namespace composed of the repo name (GitHub username, company name, etc.) and the name of individual packages defined there.

The barrier for inclusion could include a linting step for the added configuration files, with some basic verification that the URL given is indeed pointing to a valid opam-repository. We could certainly draw inspiration from the current process in the main opam-repository for deciding what other considerations are important before merging PRs.

By design, the opam maintainers would defer to the maintainer of each repo for defining the policy on maintenance, lifetime, and quality of packages listed in their custom repo.

This is not a fully fleshed-out proposal, it's an idea that we could implement now. It could provide useful data from the community and potentially be part of a larger, incremental solution. If the repo gains traction, we could consider modifying some tools to gradually expose its data. For example including the packages there into tools like sherlodoc, etc.

In a world where you'll soon be able to specify package dependencies directly with dune without requiring packages to be listed centrally, this could also offer a way for packages living outside the main repo to be discovered. Additionally, the unique repo names defined in the repos-registry would provide a canonical reference for each package.

I'm open to discussing this further if some find this idea appealing. I believe it has several beneficial properties, including the fact that it doesn't require immediate changes to existing tools (opam handles custom opam-repositories beautifully), and it relates to the topics discussed here.

And just to be clear, despite the timing of this post, this isn't an April Fools' joke. Unless the joke is on me and this idea has already been discussed or such meta repo already exists and I am just not aware of it!! 😄

mseri · 2024-04-03T13:42:03Z

Dear all, here are the notes from the last meeting. Sorry for the lengthy delay.

Meeting notes (2024/03/18)

Present: Kate, Marcello, rjbou, Ryan

Agenda

discuss pushbacks/comments/…
continue writing the draft

Notes

the change to use git apply instead of gpatch has been merged in opam 2.2 and a backport PR is already available for the upcoming opam 2.1.6. Once the new cygwin is released in ~1 week, we will get some testing via opam 2.2 and if everything works fine the patch for opam 2.1 will be merged and released. This point has been moved to Phase 0.
Update: this change was reverted as it was breaking the application of numerous patches in the opam-repository, a new attempt is currently under active development.
Phase 3: rephrased point on the maintainership for the announcement
Phase 3: allow to specify specific versions, e.g. cohttp will maintain also version 2.X for the time being
Phase 3: how do we specify maintainership from external entity that wants to keep a package in the opam-repository since they depend on it externally? We need a new field for these maintainers, suggestion: x-external-maintainer
Phase 3: add a lint in opam repo CI to ensure x-external-maintainer is carried over from one package to the next and possibly a warning to ensure it is present if specific versions are specified
Phase 3 and Phase 4 could be swapped around if needed.
Add requirements (meaning that we are pending on external actions) for each step in the proposal so that it is clear and boxable what are the requirements
TODO Add recap of what is the intent of each phase to ensure a smoother iteration and to ease gathering feedback
Action point: ping CI infrastructure people on slack or open an issue on the CI infrastructure repository to arrange meeting
Phase 6: automate Phase 5 so that we just have to review the two ADD/REMOVE PRs

Conclusion

the plan has become more stable and the first part is close to actionable
a discussion with the maintainers of the CI infrastructure for ocaml.org should be scheduled to start working on the necessary tooling
the draft has been polished a bit further and we are hoping to propose it to the rest of the community soon, with some areas for improvement remaining that should not block the first three phases of the project
- areas to be polished further are marked with "TODO"

Plan

Phase 0 (this here draft)

Write and agree on a new policy for opam-repository
- publish it on the opam-repository wiki
Reach out the infra team to make sure they are ok with the proposed document
Write a discuss post to announce the new policy
- advertise it on discord, mailing-list, ocaml-weekly-news, … https://ocaml.org/community
- write the motivation for the change in the discuss post
make sure everyone is using a version of opam that does not break when files are deleted, e.g. the upcoming opam 2.1.6 or 2.2.0, and give an alternative for people using older opam (e.g., rm -r ~/.opam/repo/default/ && opam update default)
- will be fixed in opam 2.1.6: 2.1.6 backport commits opam#5870
- use announce field to warn all users of the upcoming changes
  - includ a warning for the (g)patch issue if opam:version <2.1.6

Phase 1 (setup + unavailable/broken packages)

Create the ocaml/opam-repository-archive github repository
- Create a repo file
- Create a README.md
  - explaining the role of the repo
  - explaining the uses of the repo (opam repository add …)
  - explaining the additional x- fields in the files
  - explaining that files in here may be modified to improve installability and such
Create a CI for this new repository
- it should simply lint the files
  - checks that every dependency constraints are of the form "dep" {... & <= "<latest version of dep available in opam-repository at the time of the PR or lower>"} except for "ocaml"
  - checks if the x-reason-for-archival is set
  - checks if the x-opam-repository-commit-hash-at-time-of-archival is set
Add the archive repo to the health-check but run it more infrequently (~1/month)
Make a list of known unavailable or broken packages using opam-repo-ci, opam-health-check and opam admin check --installable
- make a PR removing them from opam-repository
- make a PR adding them to opam-repository-archive
write a Discuss post listing all the packages that are going to be removed and wait a week for comments, then merge if ok.
- in the post, explain the problem with (g)patch (see Phase 0)
Push a new before-opam-repository-archive tag on opam-repository before any of the removal

Phase 2 (OCaml < 4.05 -- the least aggressive pruning)

Move all ocaml packages < 4.05 (oldest version still in use in the wild according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge
Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge

Phase 3 (spring cleaning)

Write a Discuss post to advertise the extension fields for maintainers to declare intention of maintainshing for their packages
- for usability, it is sufficient to add the field to the latest version of a package and then the tooling will resolve this
- in case of multiple values for this field in multiple versions of a package, the latest version is taken into account
Syntax for the new x-maintenance-intent field:
- the value would be a list of strings matching on version numbers, with special keywords (latest), (any) and (none), or specific version numbers
- For example:
  - ["(latest)"] the maintainer will only maintain the latest version
  - ["(latest)" "(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and (X-1).Y.Z
  - ["(latest)" "(latest).(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and X.(Y-1).Z
  - ["(any).(latest)"] the maintainer will maintain every major version X for each X.Y.Z
  - ["(latest).(any).(latest)"] the maintainer will maintain every Y for each X.Y.Z (where X is the latest)
  - ["(any)"] the maintainer will maintain every single versions
  - ["(none)"] the maintainer will not maintain any version
  - ["1.3"] the maintainer will maintain the latest version of "1.3.Z"
  - ["2.(latest)"] the maintainer will maintain the latest minor version specifically of version "2" of the package
- Phase 3 is planned to happen on TODO: Fix a Date. Make sure that this proposal does allow to write any type of pattern that may apply to your packages and let us know if not, this is our chance to change it beforehand. (TODO: make this point boldface when doing the announcement)
- The idea is that only the packages with the x-maintenance-intent will be considered for removal when the times come, the others will stay
Add support on dune release, opam publish, opam repo ci linter, ...
Ping maintainers of large sets of packages individually so they can add this field to the latest version of each of their packages
Create a script that reads every opam-repository packages, take the latest version of each packages that have the x-maintenance-intent field and lists every versions that are not maintained anymore
use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages
- TODO: do we need a mechanism to register interest in specific packages (a way to indicate "I depend on this package even though my work is not published in opam-repository")

Phase 4 (OCaml < 4.08)

Move all ocaml packages < 4.08 (oldest version of OCaml in use for maintained distributions according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
- Wait for CentOS 7 EOL in May
- Announce it in a Discuss post and wait a week for comments then merge
Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive
- Announce it in a Discuss post and wait a week for comments then merge

Phase 5 (coasting)

Repeat Phase 3 every 6 or so months
Repeat Phase 4 everytime it is deemed that most people and distributions have migrated to a newer version of OCaml

mseri added the policy discussion label Jun 8, 2023

hannesm closed this as completed Dec 5, 2023

kit-ty-kate reopened this Dec 5, 2023

reynir mentioned this issue Dec 19, 2023

Mark let-if and ssh-agent-unix unavailable #24961

Merged

kit-ty-kate mentioned this issue Mar 4, 2024

2.1.6 backport commits ocaml/opam#5870

Draft

5 tasks

avsm mentioned this issue Mar 29, 2024

Mark older versions of result only compatible with <4.08 #25603

Closed

Requests for comments: how does opam-repository scale? #23789

Requests for comments: how does opam-repository scale? #23789

Comments

hannesm commented May 14, 2023

hannesm commented Nov 28, 2023 • edited

hannesm commented Dec 5, 2023

mseri commented Dec 5, 2023

mseri commented Dec 5, 2023

samoht commented Dec 5, 2023 • edited

hannesm commented Dec 7, 2023 • edited

kit-ty-kate commented Dec 7, 2023

samoht commented Dec 7, 2023

samoht commented Dec 8, 2023 • edited

arbipher commented Dec 13, 2023

mro commented Dec 13, 2023 via email

hannesm commented Dec 13, 2023

arbipher commented Dec 13, 2023

reynir commented Dec 19, 2023

mseri commented Dec 19, 2023

hannesm commented Dec 22, 2023

kit-ty-kate commented Dec 22, 2023

gridbugs commented Jan 9, 2024 • edited

samoht commented Jan 9, 2024 • edited

gridbugs commented Jan 10, 2024

c-cube commented Jan 10, 2024

gridbugs commented Jan 10, 2024

Leonidas-from-XIV commented Jan 10, 2024

hannesm commented Jan 10, 2024 • edited

Goal

Things needed

hannesm commented Jan 10, 2024

Leonidas-from-XIV commented Jan 10, 2024

bn-d commented Jan 23, 2024 • edited

toots commented Jan 24, 2024

mseri commented Jan 24, 2024

yawaramin commented Jan 24, 2024

raphael-proust commented Jan 25, 2024 • edited

Agenda

Notes

Introduction

Interested Parties

Needs/Wants/Cants of the interested parties

Proposals

solution 1: have two classes of packages: the maintained ones and the "unmaintained" ones

solution 2: have separate branches for separate ocaml versions

solution 3: use commits/tags in opam git repository

solution 4: move the broken packages (uninstallable, test failings, etc.) to an archive repository

solution 5: have two classes of packages, but it's "experimental"/"playground"/"lite" packages and more thoroughly tested packages

Miscellaneous notes

Summary

avsm commented Jan 25, 2024

samoht commented Jan 25, 2024 • edited

mseri commented Jan 25, 2024 • edited by kit-ty-kate

avsm commented Jan 26, 2024

raphael-proust commented Jan 29, 2024

hannesm commented Jan 29, 2024

kit-ty-kate commented Feb 20, 2024

Public meeting on 2024-02-07

Agenda

Roll call

Notes

yawaramin commented Feb 20, 2024

kit-ty-kate commented Feb 20, 2024

yawaramin commented Feb 20, 2024

kit-ty-kate commented Feb 20, 2024

samoht commented Feb 20, 2024 • edited

yawaramin commented Feb 20, 2024

avsm commented Feb 21, 2024 • edited

mseri commented Feb 23, 2024

Discussing the future of opam-repository

Roll call

Notes

Conclusion

Short term solution - Draft

Phase 0 (this here draft)

Phase 1 (setup + unavailable/broken packages)

Phase 2 (OCaml < 4.05 -- the least aggressive pruning)

Phase 3 (spring cleaning)

Discussion

kit-ty-kate commented Feb 27, 2024

hannesm commented Nov 28, 2023 •

edited

samoht commented Dec 5, 2023 •

edited

hannesm commented Dec 7, 2023 •

edited

samoht commented Dec 8, 2023 •

edited

gridbugs commented Jan 9, 2024 •

edited

samoht commented Jan 9, 2024 •

edited

hannesm commented Jan 10, 2024 •

edited

bn-d commented Jan 23, 2024 •

edited

raphael-proust commented Jan 25, 2024 •

edited

samoht commented Jan 25, 2024 •

edited

mseri commented Jan 25, 2024 •

edited by kit-ty-kate

samoht commented Feb 20, 2024 •

edited

avsm commented Feb 21, 2024 •

edited