Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2.0 #658

Open
samoht opened this issue Mar 20, 2019 · 33 comments

Comments

Projects
None yet
6 participants
@samoht
Copy link
Member

commented Mar 20, 2019

This is the tracking issue to prepare the 2.0 release.

Here are the missing items before the 2.0 code cut:

  • graphQL client [edit: we are move graphQL client bindings in OCaml to post 2.0]
  • mirageOS support
  • Complete the CHANGELOG
  • Complete the API migration guide

Once the code is released, we can iterate with minor 2.0.x release to get in a state where everyone is happy. Then, we can have:

  • publish the new website with the new tutorial
  • publish and write a blog post
  • update the online docs

Any thoughts?

@avsm

This comment has been minimized.

Copy link
Member

commented Mar 20, 2019

Looks good. If you have an opam remote that tracks the packages, I can also refresh docs.mirage.io to include bleeding edge versions of the Irmin 2.0 RCs.

@samoht

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2019

After discussing with @zshipko it appears than we don't want to ship the OCaml graphQL client right now. They still need a few more tweaks before being totally happy with it. So it's better to not ship something that we know will change for sure in the very near future.

@hannesm

This comment has been minimized.

Copy link
Member

commented Mar 20, 2019

@samoht thanks! about the MirageOS compatibility:

  • @andreas fixed graphql-parser already :D
  • I think #659 is what we discussed in Marrakesh which should work (NB: I haven't tested this, Sys.getcwd () returns an absolute path, now . is a relative one -- is that ok?)
  • the third issue is that Canopy doesn't hold their store anymore (you should be able to reproduce by testing Engil/Canopy#95 and see whether it comes up with a webserver, for me it fails to read .config/uuid from the git remote (which is there according to printfs after the clone)):
let store_config = Irmin_mem.config ()
let repo _ = Store.Repo.v store_config

let store () =
  match remote_branch () with
  | None        -> repo () >>= Store.master
  | Some branch -> repo () >>= fun r -> Store.of_branch r branch

This store () leads to an empty repository most of the time. I've worked around this issue by introducing a mutable cell that keeps a handle to the store, and is updated after each pull (see hannesm/Canopy@104546c for the commit) -- I'm not sure whether this is a bug in irmin or not, maybe the semantics should be as is!?

datakit pushed a commit to moby/datakit.logs that referenced this issue Mar 20, 2019

datakit
6 items modified
Updated
{moby/datakit 644[8018a3d76b789013362094212a75186468c91392] master avsm open "support the webmachine 0.6.0+ interface" 
[|0 avsm: "This however conflicts with irmin-http, which requires webmachine<0.6.0.\r\n\r\nSo t";
  474873810 avsm: "waiting on mirage/irmin#658"|]} 
Removed  {moby/datakit 4db4dbf81f8a820161134bb2ca0045bc8fef704e}
{moby/datakit 5fc28829c279a97bde71344f1e189c837de8d4df}
{moby/datakit 76bbd2ecf7044ea67195724ad05d087591d21c81}
{moby/datakit 95cfbdb1b7f538cea708bcca36f47c69a76a1228}
{moby/datakit bc1dd099a052288ee6164eed24181119efbec587}
@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Mar 20, 2019

Based on the outstanding PRs and issues it seems do-able to plan the release for April 3rd. This gives us two weeks to get everything together, which should be enough for what's listed below.

If anyone thinks that timeline is too aggressive then there's no harm in extending it, however I think it would be best to make the release as soon as possible. Also, let me know if there's anything I'm missing here.

Timeline

  • March 29th
    • finish outstanding PRs
    • finalize CHANGES.md and migration guide
    • make sure tutorial is completely up-to-date
  • April 1st/2nd
    • final code review
  • April 3rd
    • publish Irmin 2.0.0 to opam

After this we can start on the 2.0.x release, which will include a blog-post/larger announcement.

Release related materials

Outstanding issues and PRs

  • ocaml-git issue when pushing to git_daemon (ocaml-git#342)
  • Irmin_git: use . instead of Sys.getcwd () to allow MirageOS compatibilty (#659)
  • Updates to irmin-graphql server to clean-up API (#660 and one more PR expected)
  • Str removed from graphql-cohttp (andreas/ocaml-graphql-server#146), this will require a new release of graphql-cohttp
  • Canopy issue mentioned in the comment above

@samoht samoht pinned this issue Mar 20, 2019

@samoht

This comment has been minimized.

Copy link
Member Author

commented Mar 21, 2019

The timeline looks good -- we even be a bit more aggressive and plan for a 2.0 next week if everything goes well, and start to check that all the constraints of the rev-deps are in place.

The only big unknown for me is the issue that @hannesm is seeing while using Canopy. Would be great to check if this can be reproduced and understand the cause/impact a bit more.

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Mar 21, 2019

@hannesm I'm not able to reproduce this using Engil/Canopy#95 - is there something else I need to do other than run ./canopy?

@hannesm

This comment has been minimized.

Copy link
Member

commented Mar 21, 2019

@zshipko I tested this on solo5-hvt, not sure whether the unix backend will take another path in dependencies. I will re-try this weekend in a fresh switch and record the steps. Thanks for attempting to reproduce.

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Mar 22, 2019

Okay, I am able to reproduce the issue using the solo5-hvt backend - I will look into this a bit more

@hannesm

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

@zshipko good to know, did you make progress on that?

@samoht

This comment has been minimized.

Copy link
Member Author

commented Mar 27, 2019

@dinosaure told me that it should be fixed by mirage/ocaml-git#338 (to check)

@hannesm

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

@samoht I'm not sure. the clone/checkout/pull works fine -- and on Unix, even Canopy with your PR seems to work fine... but I've not read through git/irmin changes to have an expert opinion on this..

@samoht

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

@dinosaure @zshipko did you get more understanding of the root cause of the MirageOS issue? Should we delay the release? (it's perfectly fine to delay the release, but I just would like to understand what happens :p)

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Apr 3, 2019

Unfortunately, I haven't been able to find out anything else about the Canopy/MirageOS issue, so I'm not sure about the release. I'm hoping @dinosaure has more insight!

@dinosaure

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

I'm currently on some issues of ocaml-git. At this stage, I'm currently on the weird bug about hvt, in other side, I'm waiting a feedback from @hannesm about a deep change on the smart protocol where I mostly fix it blindfolded. If you can post-pone the release about one day, I can give you a report tomorrow.

@samoht

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

That sounds good, let's delay the release for a few days to be sure we haven't missed anything :-)

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

for MirageOS, it would be nice to add support of a kv_ro and kv_rw to the mirage command line utility which uses irmin+remote git as storage (see generic_kv_rw/ro). at configure-time I'd like to select which kv implementation is used (unix/mem/git), and runtime provide the necessary data (file path, maximum size, remote url) as boot parameters. NB: @linse and me tried and failed, and atm hardcoding irmin-kv/rw into unikernels :/ (see https://github.com/hannesm/functoria-playground for some failed attempts at dealing with functoria)

@samoht

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

@hannesm agreed but I guess this can be done after the 2.0 release.

[edit] or do you mean we still have some changes to do for irmin-mirage?

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

@samoht it can be delayed, yes.

in respect to irmin-mirage and git-mirage and the new release, I'd be really happy if push and pull would work on kvm/hvt/freestanding, as well as Canopy.

Canopy

Applying your patch does clone the store, but doesn't keep the data in the store () below (i.e. when pull is called (by main), and i dump the values after the Sync.pull_exn t upstream `Set, i see data -- but subsequent operations that use store () get an empty store -- canopy then complains on startup that it can't find .config/uuid)).

let store_config = Irmin_mem.config ()
let repo _ = Store.Repo.v store_config

let store () =
  match remote_branch () with
  | None        -> repo () >>= Store.master
  | Some branch -> repo () >>= fun r -> Store.of_branch r branch

push and pull

this is a unikernel that uses the mirage-kv API and irmin+git in the backend. It listens on tcp port 1234 and 1235.

  • if a client connects on port 1234, it pulls (Store.connect .. KV.fold) the given remote and prints it to the client.
  • if a client connects on port 1235 and enters a "key:value", this key -> value is pushed to the remote repository

issues i saw today:

  • pull works via https and the git protocol
  • push doesn't work

good news:

2019-04-03 20:58:24 -00:00: DBG [irmin.git] push https://miragebot:foobar23@github.com/miragebot/testdata
2019-04-03 20:58:24 -00:00: DBG [git.sync.http] Launch the GET request to https://github.com/miragebot/testdata/info/refs?service=git-receive-pack.
2019-04-03 20:58:24 -00:00: DBG [cohttp] Send a request to https://github.com/miragebot/testdata/info/refs?service=git-receive-pack.
2019-04-03 20:58:24 -00:00: ERR [git.sync.http] The HTTP decoder returns an error: (`No_assert_predicate #predicate).
2019-04-03 20:58:24 -00:00: ERR [application] error while pushing data (`Smart (`No_assert_predicate #predicate))
@dinosaure

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

@hannesm mirage/ocaml-git#351 should fix one of your reported error. The last case when you want to push via HTTPS on testcalendar is not fixed yet however. However, could you test PR?

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

to report back, I think that most issues are fixed with mirage/ocaml-git#351

I switched to using Irmin.Store.KV and Sync directly (instead of KV_RW) to have tight control over push and pull operations -- since I wanted to support mutliple writers. What I'd like:

    let config = Irmin_mem.config () in
    Store.Repo.v config >|= Store.master

are actually doing -- i.e. does the Store.Repo.v and/or Store.master already do network communication (they can't since they don't know the remote yet, or?)? why are they in the Lwt.t monad?

@samoht

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2019

@hannesm does this mean that the current KV_RW interface is useless? Or is there anything that we can do to give finer control on push/pull?

Good catch for the pretty-printers, I'll add these tomorrow.

@dinosaure do you understand what happens with git/solo5?

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

@samoht the current KV_RW interface is fine if you're the only writer - there's explicit push control (batch / write), but no control over pull at all.

the pull could either be periodically (not so nice) or a KV_RW should offer a specific sync operation, called by the application once it receives a webhook / notification that the backend changed. there's also need to control merge in the multiple writer store, but only during set and remove (these operations atm offer only a write_error, but no way to merge that).

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Apr 16, 2019

After being away for the last week I'm trying to understand what is needed to make this release happen.

Is mirage/ocaml-git#353 what needs attention right now? I am happy to spend some time trying to understand what's happening there and can also take care of the pretty-printers and documentation update requested by @hannesm above.

It looks like this means we will need to postpone the irmin 2.0.0 until after the next ocaml-git update since these latest changes depend on the development version of ocaml-git (see: #668). This is unfortunate, but if Canopy, and other stable applications are broken then it makes no sense to move forward until these things are dealt with.

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

so, is there any release plan? if nobody apart from myself is interested in irmin + git remote on MirageOS, then you should go ahead and release without supporting that. I'm rather low on time atm (including the foreseeable future), and thus won't be able to debug and test much further (I hope the above provided examples are helpful). What occured to me is that irmin-mirage now depends on graphql, and I'm not sure why this should be the case -- maybe the irmin-mirage opam package should be split into multiple ones?

@yomimono

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

I'm interested in irmin + git remote on MirageOS but similarly not ready to take advantage of it in the near future, so I shouldn't block a release.

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented Apr 22, 2019

Well we're waiting on a new ocaml-git release and @samoht is out of town for the next two weeks, so I plan on spending this time debugging any known issues/preparing and updating documentation.

At this point it's best to make a new issue for any problems you encounter (rather than adding it to this thread) - that way we can keep track of what needs attention between the 2.0.0 and 2.0.x releases.

I don't think we're really thinking about releasing a broken version of irmin-git + remote on MirageOS, however since I'm not specifically using it for anything it's hard to find and reproduce bugs. One things that could help with this is some kind of unikernel test-suite (I will need to think about this!)

As for the graphql dependency in irmin-mirage - this was added to both the unix and mirage backends in hopes to eventually replace the current HTTP server. I suppose that Irmin_mirage.Graphql could live in a package like irmin-mirage-graphql or something, is this an issue because of executable size or something else?

@hannesm

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

@zshipko sounds good to me. yes, the graphql is not only a build-dependency (which adds to build time), but also runtime code size for functionality i do not plan to use. I like to have systems without any http (server nor client) code in them ;)

unikernel test suite: well, i linked to smaller ones i developed just for irmin, there's also some work on an e2e mirage test system (see https://github.com/mato/e2e-mirage-solo5/), which may be useful to have with irmin unikernels (where you may need a git daemon elsewhere as well).

I commented since I saw that the release plan was early April, now it's rather end of April. I have some irmin+git unikernels deployed, thanks to @dinosaure there are fewer issues in git now, and fewer pins needed, but i guess e.g. the canopy issue should be looked into more deeply (you tried canopy on unix and it worked, without unix it didn't -- this would worry me if I'd be in the process of releasing a library which behaviour should be the same independent of the platform).

@samoht

This comment has been minimized.

Copy link
Member Author

commented May 7, 2019

What's the status of the release? @zshipko what are the last blockers?

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented May 7, 2019

@samoht At this point we're mainly waiting on the ocaml-git release before we can start our release.

Other than that, there is #671 and #681, which also seem important to address.

@dinosaure

This comment has been minimized.

Copy link
Member

commented May 7, 2019

So encore is under the pipe to release, eqaf too and issues about ocaml-git are fixed. So no big blockers, I think on my side. It's just I take a time to push on the release button.

@samoht

This comment has been minimized.

Copy link
Member Author

commented May 7, 2019

no pb, good to know there are no blockers :-) Let us know if you need some help to release things.

@hannesm

This comment has been minimized.

Copy link
Member

commented May 7, 2019

i guess the canopy issue (see #658 (comment) #658 (comment)) still needs debugging and fixing, no?

(see Engil/Canopy#95 for the specific PR to canopy which according to @zshipko works on mirage+unix, but fails with mirage+solo5)

@zshipko

This comment has been minimized.

Copy link
Collaborator

commented May 7, 2019

Yeah, sorry @hannesm! That is definitely another issue that should be looked at before the release. I haven't had any further luck in tracking down the bug though, so another set of eyes could be helpful here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.