datakit: update to latest versions of Irmin/Mirage/Cdmliner #529

samoht · 2017-04-22T10:14:58Z

This patch make Irmin 1.1 works with DataKit. As Irmin 1.1 only
works with cmdliner >= 1.0 and MirageOS >= 3.0, also update
these dependencies.

The main change is the removal of the body Ivfs_tree to use the new
Irmin API. The new Irmin Tree API is pretty similar (on purpose) but
it seems that we are loosing caching of file hashes. I plan to re-add
that in a later commit, after checking it is really needed.

Ideally datakit-client will also move to something closer to the Irmin API
but it is not a blocker for API compatibility.

Signed-off-by: Thomas Gazagnaire thomas@gazagnaire.org

samoht · 2017-04-22T10:16:02Z

Note: this needs the dev version of hvsock (to support MirageOS 3), + mirage/irmin#432 and mirage/irmin#431 to be merged first (and Irmin 1.1 to be released)

talex5 · 2017-04-24T10:16:21Z

CI is unhappy because the depext line also adds some (conflicting) pins.

talex5

Looking good so far :-)

talex5 · 2017-04-24T10:17:19Z

src/datakit/ivfs.ml

@@ -2,11 +2,14 @@ open Astring
 open Rresult
 open Lwt.Infix

+let src = Logs.Src.create "ivfs" ~doc:"Irmin VFS"


Should be "DataKit" here?

talex5 · 2017-04-24T10:38:58Z

src/datakit/ivfs.ml

@@ -674,43 +677,38 @@ module Make (Store : Ivfs_tree.STORE) = struct

  (* Note: can't use [Store.fast_forward_head] because it can
     sometimes return [false] on success (when already up-to-date). *)


Is this still true?

talex5 · 2017-04-24T10:48:25Z

src/datakit/ivfs_merge.ml

-
-type path = Path.t
-type step = Path.Step.t
+let src = Logs.Src.create "ivfs.merge" ~doc:"Irmin VFS"


samoht · 2017-04-24T14:44:52Z

I've updated the PR with @talex5 comments, but I haven't fixed the CI yet. Working on that now.

avsm · 2017-04-24T16:57:11Z

src/datakit/ivfs_blob.ml

-let pp_buf ppf buf = Fmt.pf ppf "%S" (Cstruct.to_string buf)
-let pp ppf t = Fmt.pf ppf "%a" Fmt.(Dump.list pp_buf) !t
+let pp_buf ppf buf = Fmt.string ppf (Cstruct.to_string buf)
+let pp ppf t = Fmt.pf ppf "%a" Fmt.(list ~sep:(unit "") pp_buf) !t


I think the unit "" can be replaced by nop from Fmt

avsm · 2017-04-24T17:00:06Z

src/datakit/main.ml

-  let merge _ = merge []
-  module Path = Ivfs_tree.Path
-end
+  Irmin.Info.v ~date ~author:"datakit <datakit@docker.com>" msg


could we remove the datakit@docker.com email entirely? This is now a moby/datakit repo, but could the default commit could also just be datakit without an email address if that's valid in Git?

Email is mandatory, so Irmin will add the default irmin@openmirage.org email to it. But yes @docker.com should go away.

avsm · 2017-04-24T17:01:15Z

src/datakit_io.ml

        protect (Lwt_unix.mkdir dir) 0o755;
      ) in
    Lwt_pool.use mkdir_pool (fun () -> aux dirname)

+  let file_exists f =
+    Lwt.catch (fun () -> Lwt_unix.file_exists f) (function
+        (* See https://github.com/ocsigen/lwt/issues/316 *)


This issue is now fixed in Lwt>=2.7.1 -- update constraint and remove this workaround?

True. Note that this is a copy/paste from Irmin-unix to avoid depending on nocrypto and hence libgmp, so it needs to be fixed upstream too.

samoht · 2017-04-25T09:21:23Z

The ci/datakit/github error is legit, it's a regression in Irmin 1.0. Should be fixed by mirage/irmin#438

samoht · 2017-04-25T11:13:38Z

Yay, all green!

Need a rebase of protocol-9p (that @talex5 is working on) before it can be merged.

talex5 · 2017-04-25T12:52:52Z

Try pinning my ping-mirage-3 ocaml-9p branch instead.

This patch make Irmin 1.1 works with DataKit. As Irmin 1.1 only works with cmdliner >= 1.0 and MirageOS >= 3.0, also update these dependencies. The main change is the removal of the body Ivfs_tree to use the new Irmin API. The new Irmin Tree API is pretty similar (on purpose) but it seems that we are loosing caching of file hashes. I plan to re-add that in a later commit, after checking it is really needed. Ideally datakit-client will also move to something closer to the Irmin API but it is not a blocker for API compatibility. Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

…xed upstream Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

samoht · 2017-04-25T14:24:18Z

All green, merging!

samoht force-pushed the new-new-irmin branch from 82e9800 to 2eaca2e Compare April 22, 2017 10:15

talex5 reviewed Apr 24, 2017

View reviewed changes

samoht force-pushed the new-new-irmin branch from 413a060 to 8182f92 Compare April 24, 2017 14:43

samoht force-pushed the new-new-irmin branch 5 times, most recently from 7516cdf to d33a472 Compare April 24, 2017 16:47

avsm reviewed Apr 24, 2017

View reviewed changes

samoht force-pushed the new-new-irmin branch 8 times, most recently from 4d2895f to 32152f2 Compare April 25, 2017 06:43

samoht force-pushed the new-new-irmin branch 2 times, most recently from 5dfd0c2 to 6b14d2f Compare April 25, 2017 11:01

samoht force-pushed the new-new-irmin branch from 6b14d2f to 67571c3 Compare April 25, 2017 11:37

samoht added 2 commits April 25, 2017 13:55

datakit: fixup name of log reporters

4507e69

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

samoht added 9 commits April 25, 2017 13:55

datakit: use S.Head.fast_forward directly as the behavior has been fi…

d2c777d

…xed upstream Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

datakit-server: depends on mirage-flow-lwt

b406909

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

datakit-io: remove workaround for lwt bug fixed upstream

62cae52

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

code cleanup: use Fmt.nop instead of Fmt.unit ""

83624f6

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

Replace @docker.com by @mobyproject.org

6d32d94

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

datakit-server needs mirage-flow-lwt

56cf66d

Remove hacks in Dockerfiles now that we are not building on hub anymore

00a2368

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

datakit-server: use a recent version of protocol-9p

ce7e6fc

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

revert Tree.empty -> Tree.empty() change

0d31175

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

samoht force-pushed the new-new-irmin branch from 67571c3 to 6f12b58 Compare April 25, 2017 12:55

make the CI happy

8a3bc5d

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>

samoht force-pushed the new-new-irmin branch from 6f12b58 to 8a3bc5d Compare April 25, 2017 14:01

samoht merged commit 71f8dc2 into moby:master Apr 25, 2017

samoht deleted the new-new-irmin branch April 25, 2017 14:24

This was referenced Apr 26, 2017

[followup #433] MirageOS 3 working branch #459

Closed

refer to mirage module types modules by new name #433

Closed

djs55 mentioned this pull request Sep 28, 2017

"git commit" seems to break the transaction interface #610

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datakit: update to latest versions of Irmin/Mirage/Cdmliner #529

datakit: update to latest versions of Irmin/Mirage/Cdmliner #529

samoht commented Apr 22, 2017 •

edited

Loading

samoht commented Apr 22, 2017 •

edited

Loading

talex5 commented Apr 24, 2017

talex5 left a comment

talex5 Apr 24, 2017

talex5 Apr 24, 2017

talex5 Apr 24, 2017

samoht commented Apr 24, 2017

avsm Apr 24, 2017

avsm Apr 24, 2017

samoht Apr 24, 2017

avsm Apr 24, 2017

samoht Apr 24, 2017

samoht commented Apr 25, 2017 •

edited

Loading

samoht commented Apr 25, 2017

talex5 commented Apr 25, 2017

samoht commented Apr 25, 2017

		@@ -674,43 +677,38 @@ module Make (Store : Ivfs_tree.STORE) = struct

		(* Note: can't use [Store.fast_forward_head] because it can
		sometimes return [false] on success (when already up-to-date). *)

datakit: update to latest versions of Irmin/Mirage/Cdmliner #529

datakit: update to latest versions of Irmin/Mirage/Cdmliner #529

Conversation

samoht commented Apr 22, 2017 • edited Loading

samoht commented Apr 22, 2017 • edited Loading

talex5 commented Apr 24, 2017

talex5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samoht commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samoht commented Apr 25, 2017 • edited Loading

samoht commented Apr 25, 2017

talex5 commented Apr 25, 2017

samoht commented Apr 25, 2017

samoht commented Apr 22, 2017 •

edited

Loading

samoht commented Apr 22, 2017 •

edited

Loading

samoht commented Apr 25, 2017 •

edited

Loading