New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get a story on 1.2 → 2.0 migration together #2918

Open
AltGr opened this Issue Apr 27, 2017 · 8 comments

Comments

Projects
None yet
5 participants
@AltGr
Member

AltGr commented Apr 27, 2017

Following the discussions this morning, there are many aspects on migration, and we may yet improve the story.

What is already here:

  • Migrating repositories (1.2 → 2.0), using opam admin upgrade. Add --mirror to serve both versions over http.
  • Using 1.2 repositories from opam 2.0: they are now converted on-the-fly; however, some files need to be fetched when the repository includes compilers with patches. While the results are cached (~/.cache/opam-compilers-to-packages/url-hashes), this can take a while on runs from a blank environment (we could host that cache somewhere ? Or squeeze the data through the opam 1.2 format ?)
  • Using 1.2 opam files from package sources, when pinning: they are converted on the fly

What can be improved:

  • Running opam 1.2 and 2.0 side-by-side. This can be done quite easily but requires manual work (e.g. alias opam2="OPAMROOT=~/.opam2 path/to/opam2", after installing opam-devel or downloading a pre-compiled opam 2 somewhere). It was suggested to even have opam 2 use ~/.opam2 instead (which could be automatically migrated from ~/.opam on the first run). Having the name of the binary be opam2 by default could help (but could also make things much worse for migration and scripts). One last idea is to choose ~/.opam or ~/.opam2 based on the exec name.

    As a last note, when it comes to shell setup and eval $(opam env), having both opam roots and PATH updates and being in different switches within opam 1 and opam 2 could lead to more trouble than it's worth (opam 2 is able to revert PATH changes for handling multiple roots, but 1.2 isn't).

What's possibly missing:

  • A simple tool to migrate a single opam file (or all opam files in a source tree). It shouldn't be more than 10 lines though :)
  • A reverse conversion 2.0 → 1.2 (for single packages and for repositories). Can be quite tricky, do we need it ? The main point would be to migrate ocaml/opam-repository to use 2.0 format by default, but still have backported updates on the 1.2 repository (the other options being to freeze it, or to fork)
  • As a middle-ground, we could start an "official 2.0 repository" that would be configured by default on top of ocaml/opam-repository on opam 2.0, allowing people who need the new features to submit there (and limiting their audience to opam 2.0 users, possibly giving 2.0 some traction). We should just make sure that no packages there end up outdated compared to their version in the 1.2 repo, if it exists. That would also give an experimental ground for running 2.0 linting and CI.
  • A way for project sources to include both 1.2 and 2.0 metadata ?

Please complete & comment!

@AltGr AltGr added the Epic label Apr 27, 2017

@samoht

This comment has been minimized.

Show comment
Hide comment
@samoht

samoht Apr 27, 2017

Member

Thanks for creating that list.

For the "Using 1.2 repositories from opam 2.0" item, I was wondering if we could add the checksum hash in the existing .comp files in the official repo, using a field that opam 1.2 would skip (need to check if that's possible). Or if that's not possible, we could put that checksum in a separate .url file, or something similar. That would avoid having everyone to recompute these checksums again and again.

Member

samoht commented Apr 27, 2017

Thanks for creating that list.

For the "Using 1.2 repositories from opam 2.0" item, I was wondering if we could add the checksum hash in the existing .comp files in the official repo, using a field that opam 1.2 would skip (need to check if that's possible). Or if that's not possible, we could put that checksum in a separate .url file, or something similar. That would avoid having everyone to recompute these checksums again and again.

@Drup

This comment has been minimized.

Show comment
Hide comment
@Drup

Drup Apr 27, 2017

Contributor

@AltGr I think the middle-ground is a decent solution. People would add most packages to the 1.2 repo, except for fancy features (compiler as packages, mostly).

I can confirm that it's fairly easy to stick to 1.2 opam file format when you use 2.0. The key point is to have opam-lint at hand to check for 1.2-well-formedness.
I don't believe you will have a decent generic reverse conversion tool, but a simple one for small packages could be helpful.

Contributor

Drup commented Apr 27, 2017

@AltGr I think the middle-ground is a decent solution. People would add most packages to the 1.2 repo, except for fancy features (compiler as packages, mostly).

I can confirm that it's fairly easy to stick to 1.2 opam file format when you use 2.0. The key point is to have opam-lint at hand to check for 1.2-well-formedness.
I don't believe you will have a decent generic reverse conversion tool, but a simple one for small packages could be helpful.

@samoht

This comment has been minimized.

Show comment
Hide comment
@samoht

samoht Apr 27, 2017

Member

Something else that we can do is to keep only the minimum amount possible of compilers in the main opam repo (e.g. remove all the PR compilers and stop the automatic sync) - as it is the main cause of friction for the transition between 1.2 and 2.0 and back, maybe this could help.

Member

samoht commented Apr 27, 2017

Something else that we can do is to keep only the minimum amount possible of compilers in the main opam repo (e.g. remove all the PR compilers and stop the automatic sync) - as it is the main cause of friction for the transition between 1.2 and 2.0 and back, maybe this could help.

@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Apr 27, 2017

Member

@samoht: yes, if we do that, we could have hardcoded compiler packages in the 2.0 repo, rather than ones dynamically generated by the translation step.

About "A simple tool to migrate a single opam file", actually it's possible using:

opam show --raw <opam-file>

Not the originally intended purpose, but it works :)

Member

AltGr commented Apr 27, 2017

@samoht: yes, if we do that, we could have hardcoded compiler packages in the 2.0 repo, rather than ones dynamically generated by the translation step.

About "A simple tool to migrate a single opam file", actually it's possible using:

opam show --raw <opam-file>

Not the originally intended purpose, but it works :)

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm May 2, 2017

Member

Something else that we can do is to keep only the minimum amount possible of compilers in the main opam repo (e.g. remove all the PR compilers and stop the automatic sync)

I'm not sure how popular this feature is -- we have no way of knowing from our stats. I could mail the caml list and suggest deprecating this feature until it makes a return in OPAM2 and see the feedback...

Member

avsm commented May 2, 2017

Something else that we can do is to keep only the minimum amount possible of compilers in the main opam repo (e.g. remove all the PR compilers and stop the automatic sync)

I'm not sure how popular this feature is -- we have no way of knowing from our stats. I could mail the caml list and suggest deprecating this feature until it makes a return in OPAM2 and see the feedback...

@Drup

This comment has been minimized.

Show comment
Hide comment
@Drup

Drup May 2, 2017

Contributor

@avsm If we get a decent opam file in the ocaml repo, we can just pin it. I expect users who try PRs to be fairly knowledgeable and capable of pinning, especially if instructions are given in ocaml's readme/hacking.

(Making a decent opam file for the dev repo is not so trivial, given the scheme for ocaml compilers in the opam repo, though)

Contributor

Drup commented May 2, 2017

@avsm If we get a decent opam file in the ocaml repo, we can just pin it. I expect users who try PRs to be fairly knowledgeable and capable of pinning, especially if instructions are given in ocaml's readme/hacking.

(Making a decent opam file for the dev repo is not so trivial, given the scheme for ocaml compilers in the opam repo, though)

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm May 19, 2017

Member

Putting an opam file in the ocaml repo is a blasted fine idea though... and it can be opam2 only as well...

Member

avsm commented May 19, 2017

Putting an opam file in the ocaml repo is a blasted fine idea though... and it can be opam2 only as well...

@dbuenzli

This comment has been minimized.

Show comment
Hide comment
@dbuenzli

dbuenzli May 19, 2017

Contributor

Mmmh there are also a few other things that could be tucked in such a PR, I'd really like to be able to odig changes ocaml.

Contributor

dbuenzli commented May 19, 2017

Mmmh there are also a few other things that could be tucked in such a PR, I'd really like to be able to odig changes ocaml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment