wrap: redesign

[ignatenkobrain]

I've thought a bit about designing our wrap-files.

How it should be in files structure
https://github.com/mesonbuild/wrap-odeint

• meson.build

https://github.com/miit-pm/RWD

• Actual code
• meson.build
• subprojects/
  • odeint.wrap

odeint.wrap

• Directory
• Upstream git repo - https://github.com/boostorg/odeint.git
• Upstream git revision/tag (OPTIONAL) - boost-1.57.0
• Wrap git repo - https://github.com/mesonbuild/wrap-odeint.git
• Wrap git revision/tag - 1.57.0

How it should work

1. Clone upstream repo to directory - git clone https://github.com/boostorg/odeint.git
2. Check out needed revision/tag - git checkout boost-1.57.0
3. Add remote wrap repo - git remote add wrap https://github.com/mesonbuild/wrap-odeint.git
4. Fetch wrap repo - git fetch wrap
5. Merge wrap revision/tag to current state - git merge wrap/1.57.0

[jpakkane]

I tried this and apparently you can only do git merge if the wrap repo was branched off the upstream repo. In this case it makes more sense to keep the wrap git repo fully up to date and not do any merges user-side.

[ignatenkobrain]

@jpakkane no, you can merge if it's not branched from upstream repo

[ignatenkobrain]

@jpakkane

[brain@x1carbon tmp]$ git clone https://github.com/boostorg/odeint.gitCloning into 'odeint'...
remote: Counting objects: 15708, done.
remote: Total 15708 (delta 0), reused 0 (delta 0), pack-reused 15708
Receiving objects: 100% (15708/15708), 3.37 MiB | 567.00 KiB/s, done.
Resolving deltas: 100% (9756/9756), done.
Checking connectivity... done.
[brain@x1carbon tmp]$ cd odeint/
[brain@x1carbon odeint]$ git remote add wrap https://github.com/mesonbuild/odeint.git
[brain@x1carbon odeint]$ git fetch wrap 
warning: no common commits
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 5 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/mesonbuild/odeint
 * [new branch]      master     -> wrap/master
[brain@x1carbon odeint]$ git merge wrap/master --no-edit 
Merge made by the 'recursive' strategy.
 meson.build | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 meson.build

[ignatenkobrain]

actually mesonbuild's odeint is plain repo with only meson.build

[jpakkane]

Here's some of the things I have in mind. Let's start by ignoring everything about implementation and hosting and instead just think how it would be used.

Ideally you would have a web site. On it you can search for wraps. We probably need to support multiple versions. So for libfoo we would provide libfoo-0.1, libfoo-0.2, libfoo-1.0 etc. To provide for updates we would probably have something like the Debian release versioning where you append a number whenever the build definition is updated. So we would have libfoo-1.0-1, libfoo-1.0-2 etc.

When the user finds a library they want, they click on the newest version and they get a plaintext wrap file they can write to their project. Later we might want to add a specific tool to do this automatically (i.e. 'wrap-tool install libfoo', 'wrap-tool update' etc).

For implementation I'd really prefer to do no Git operations except pull and checkout on the client at all. The reason is that there are a million ways to shoot yourself in the head with Git (just think what can happen if you try to update a checkout that has merged two unrelated repos and then they start conflicting). I do not ever want to get to the situation where I have to give Git tech support to our users. That just leads to misery. We should reserve Git (and svn, hg etc) support for cases where Meson definitions are upstream and we just check out a repo immutably.

The wraps that we provide should be similar in spirit to the patch files in rpm and deb files: minimal and explicit. One of the biggest complaints people seem to have about containers is that you just get a magic bunch of bytes which are very hard or impossible to audit. Because of this we can't do the simple thing of just branching upstream repo, adding our own build defs on top and putting the whole shebang up for download.

The project should operate so that everyone can easily submit updates and even new projects for use. Maybe something like having a Github repo per project and then generating downloadable wrap files from those with webhooks. This probably requires having a build machine outside of Github with something like mod_python.

One issue I have been struggling with is how to map these releases into Git repos. The libfoo project might have releases 0.1, 1.0, 0.1.1, 0.2 etc and they might be created in any order (bugfix releases for 0.1 might easily come after the release of 0.2). We also need to keep providing all old release versions because people might not upgrade (which is reasonable). Maybe have a branch for each release and have our version number be the amount of commits after the first release?

[ignatenkobrain]

Ideally you would have a web site. On it you can search for wraps. We probably need to support multiple versions. So for libfoo we would provide libfoo-0.1, libfoo-0.2, libfoo-1.0 etc. To provide for updates we would probably have something like the Debian release versioning where you append a number whenever the build definition is updated. So we would have libfoo-1.0-1, libfoo-1.0-2 etc.
When the user finds a library they want, they click on the newest version and they get a plaintext wrap file they can write to their project. Later we might want to add a specific tool to do this automatically (i.e. 'wrap-tool install libfoo', 'wrap-tool update' etc).

Sounds good to me. I agree for site (but first let's store it just in git repos).

The project should operate so that everyone can easily submit updates and even new projects for use. Maybe something like having a Github repo per project and then generating downloadable wrap files from those with webhooks. This probably requires having a build machine outside of Github with something like mod_python.

Yes, let's store on github and do releases. As you know, machine is not a problem for me.

One issue I have been struggling with is how to map these releases into Git repos. The libfoo project might have releases 0.1, 1.0, 0.1.1, 0.2 etc and they might be created in any order (bugfix releases for 0.1 might easily come after the release of 0.2). We also need to keep providing all old release versions because people might not upgrade (which is reasonable). Maybe have a branch for each release and have our version number be the amount of commits after the first release?

Hm, I think we should do:

1. create branch for (0.1, 0.2, 1.0, 0.1.1)
2. keep up to date them with meson new features (but I think we should do this for only supported upstream versions, if users want to get other versions - they can send pull requests.)

Why we should do releases? Why don't use latest checkout from git? We will provide on site last snapshot from git. We also can set up CI to check that we are not breaking builds.

[jpakkane]

Releases are important for many use cases. As an example people might want to run their own copies of the repo. They can then just download the wrap file and the two archives, audit them as they see fit (some organisations are crazy serious about this) and then just point the urls in the wrap files to their internal copies. This is, roughly, how Google Chrome development is done AFAICT.

We should also not require the use of Git on the client. Lots of people use other revision control systems, some security policies do not allow git checkouts from outside the walled garden, some people require build reproducibility for ten years and can't rely on our service being available and so on.

In summary, running your own mirroring wrap service needs to be easy and low overhead and the only way to do that that I can see is to do releases.

[ignatenkobrain]

https://github.com/mesonbuild/odeint/archive/312fcf8f375c798360437706461473a53bb53749/meson-odeint-312fcf8.tar.gz

Github allow us to download specific commit in archive, so users will not use git at all

[jpakkane]

The problem with that is that Github appends the commit id hash to the directory name inside the .tar.gz so you can't just extract it over the source dir. This is fixable with preprocessing but I'd really like if downloads from Github were handled in the exact same way as downloads from other places (i.e. they would have the correct subdirectory so you could just extract them without worrying).

[ignatenkobrain]

If we will do releases odeint-1.0-1 then directory also will have wrong prefix. So we will not able to make additional releases. So I think we need to preprocess directory.