Issues with pulling/pushing remote repo #379

Open
vitcozzolino opened this Issue Nov 10, 2016 · 7 comments

Projects

None yet

3 participants

@vitcozzolino

Hi,
I'm running this small piece of code to test a bit Irmin. At the moment I have a repository on BitBucket that I'm trying to first read and then write:

open Lwt
open Irmin_unix

module Main (C: V1_LWT.CONSOLE) = struct

module S = Irmin_git.FS(Irmin.Contents.String)(Irmin.Ref.String)(Irmin.Hash.SHA1)
module Sync = Irmin.Sync(S)

let config = Irmin_git.config ~root:"/tmp/test" ~bare:true ()
 let upstream = Irmin.remote_uri "https://xxxx:xxxx@bitbucket.org/vitcozzolino/irmin-test.git"

let read () =
  S.Repo.create config >>= S.master task
  >>= fun t  -> Sync.pull_exn (t "Syncing with upstream store") upstream `Update
  >>= fun () -> S.read_exn (t "get the README") ["README.md"]
  >>= fun r  -> Printf.printf "%s\n%!" r; return_unit 

let write () =
  S.Repo.create config >>= S.master task
  >>= fun t -> S.update (t "Updating my file")  ["README.md"] "NEW SENTENCE!"
  >>= fun () -> Lwt.return(Printf.printf "Done writing!\n%!")
  >>= fun () -> Sync.push (t "Pushin to remote stream") upstream >>= function
  | `Ok -> Printf.printf "End"; return_unit
  | `Error -> Printf.printf "Bad"; return_unit


let start console () =
  read () >>= fun () -> write ()

end

While reading doesn't cause me any trouble, writing fails constantly. This is the console output:

2016-11-10 15:22.19: INF [git.unix] Reading /tmp/test/.git/HEAD
2016-11-10 15:22.19: INF [git.unix] Reading /tmp/test/.git/refs/heads/master
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/objects/ba/27654e404861dc24119d19bd373fb6c75614b1
2016-11-10 15:22.20: INF [git.sync] SENDING: "0069want 6363392e72fdef2d87bf16b54a887f7c385cf274 agent=git/ogit.1.9.2 side-band-64k ofs-delta thin-pack\n"
2016-11-10 15:22.20: INF [git.sync] SENDING: "0000"
2016-11-10 15:22.20: INF [git.sync] SENDING: "0032have 6363392e72fdef2d87bf16b54a887f7c385cf274\n"
2016-11-10 15:22.20: INF [git.sync] SENDING: "0032have ba27654e404861dc24119d19bd373fb6c75614b1\n"
2016-11-10 15:22.20: INF [git.sync] SENDING: "0009done\n"
2016-11-10 15:22.20: INF [git.sync] remote: Total 0 (delta 0), reused 0 (delta 0)

2016-11-10 15:22.20: INF [git.sync] Received a pack file of 32 bytes.
2016-11-10 15:22.20: INF [git.unix] Writing /tmp/test/.git/refs/heads/master (/tmp/test/.git/tmp/masterf3f8a7write)
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/refs/heads/master
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/objects/63/63392e72fdef2d87bf16b54a887f7c385cf274
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/objects/5b/498a4c8cb53ee8e8dfd9f9d391daf851d77756
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/objects/57/602f510f286bc4c362396a408c09f7899758f2
#MY FIRST IRMIN TEST
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/HEAD
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/refs/heads/master
2016-11-10 15:22.20: INF [git.unix] Writing /tmp/test/.git/objects/32/2828bfd13722b3e3307b1c1979cb2a4674013c (/tmp/test/.git/tmp/2828bfd13722b3e3307b1c1979cb2a4674013cfd19a2write)
2016-11-10 15:22.20: INF [git.unix] Reading /tmp/test/.git/refs/heads/master
2016-11-10 15:22.20: INF [git.unix] Writing /tmp/test/.git/refs/heads/master (/tmp/test/.git/tmp/master17ffd3write)
Done writing!
2016-11-10 15:22.21: INF [git.unix] Reading /tmp/test/.git/refs/heads/master
2016-11-10 15:22.21: INF [git.unix] Reading /tmp/test/.git/objects/05/3b72fe752df4e4d541994a4b6f9d788fba39ae
2016-11-10 15:22.21: INF [git.sync] SENDING: "007f6363392e72fdef2d87bf16b54a887f7c385cf274 322828bfd13722b3e3307b1c1979cb2a4674013c refs/heads/master\000report-status ofs-delta"
2016-11-10 15:22.21: INF [git.sync] SENDING: "0000"
2016-11-10 15:22.21: INF [git.sync] SENDING: 544 bytes
Fatal error: exception (Failure
  "The connection has been closed by the server. This is usually due to an invalid client request.")
Raised at file "src/core/lwt.ml", line 789, characters 22-23
Called from file "src/unix/lwt_main.ml", line 34, characters 8-18
Called from file "main.ml", line 87, characters 5-10

I'm pretty sure I'm missing some instructions: I was not able to find the equivalent of "git add" and "git commit" in the API docs.
Moreover, I have noticed that even if I just run the read () function, the git repository appears corrupted. This is the output of git fsck --full:

Checking object directories: 100% (256/256), done.
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: packfile .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.pack index not opened
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
broken link from  commit 6363392e72fdef2d87bf16b54a887f7c385cf274
              to  commit f174c5f25ea29e26a68ce7ea52958be8f3d53d88
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
broken link from    tree 5b498a4c8cb53ee8e8dfd9f9d391daf851d77756
              to    tree 053b72fe752df4e4d541994a4b6f9d788fba39ae
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
missing tree 053b72fe752df4e4d541994a4b6f9d788fba39ae
error: wrong index v2 file size in .git/objects/pack/pack-bcfc9411d763fba12d4ea84e0a3c5dd86e6fd669.idx
missing commit f174c5f25ea29e26a68ce7ea52958be8f3d53d88

If I clone manually the repository (git clone), none of the above error messages appear.

@vitcozzolino vitcozzolino changed the title from Unable to successfully use Sync.push to Issues with pulling/pushing remote repo Nov 10, 2016
@samoht
Member
samoht commented Nov 10, 2016

Which version of ocaml-git are you using? git push is unfortunately not very stable so I am not too surprised. If you have time and are willing to look at fixing this I would welcome any patches :-)

@vitcozzolino

Hi samoht,
I'm using version 1.9.2. This is the output of opam show git:

             package: git
             version: 1.9.2
          repository: default
        upstream-url: https://github.com/mirage/ocaml-git/releases/download/1.9.2/git-1.9.2.tbz
       upstream-kind: http
   upstream-checksum: 21f45942e044efe1b7f9f592ef7b10e0
            homepage: https://github.com/mirage/ocaml-git
         bug-reports: https://github.com/mirage/ocaml-git/issues
            dev-repo: https://github.com/mirage/ocaml-git.git
              author: Thomas Gazagnaire
             license: ISC
                 doc: https://mirage.github.io/ocaml-git/
             depends: cmdliner & mstruct >= 1.3.1 & ocamlgraph & uri >= 1.3.12 & lwt >= 2.4.7 & mtime & logs & fmt & hex & astring
             depopts: mirage-types-lwt | mirage-http | mirage-flow | channel | cohttp | conduit | base-unix | camlzip | nocrypto
   installed-version: 1.9.2 [4.02.3]
  available-versions: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.9.0, 1.9.1, 1.9.2, dev~mirage
         description: Git format and protocol in pure OCaml

I'm still a neophyte in OCaml but will have anyway look at ocaml-git and see if I can find the issue :-)

@vitcozzolino

I've been exploring a bit the ocaml-git repo and started to debug it to understand where is the bug. I've added some more debug traces and modified the library code. To make it short I found out that the push command fails because it tries to read something from the input channel while actually there is nothing to read.

With the changes applied to the ocaml-git code, that's the output I receive (attaching only the relevant part):

[...]
2016-11-11 14:17.06: DBG [git.pack] add hash: 678e7bb1d4acd3b0ac9d9b3fbbe563af812f22f0
2016-11-11 14:17.06: DBG [git.sync] SENDING: 2738 bytes
2016-11-11 14:17.06: DBG [git.sync] PacketLine.input
2016-11-11 14:17.06: DBG [git.sync] WOW -> PacketLine.input_raw
2016-11-11 14:17.06: DBG [git.unix] READ_EXACTLY IN PROGRESS (4)
2016-11-11 14:17.06: DBG [git.unix] READ_EXACTLY DONE (A�W�) (0)
2016-11-11 14:17.06: DBG [git.sync] size (A�W�)
2016-11-11 14:17.06: ERR [git.sync] PacketLine.input: "0xA\230W\249" is not a valid integer
Fatal error: exception Git_sync.Make(IO)(Store).Error
Raised at file "src/core/lwt.ml", line 916, characters 22-23
Called from file "src/unix/lwt_main.ml", line 34, characters 8-18
Called from file "main.ml", line 102, characters 5-10

The following error disappeared Fatal error: exception (Failure connection has been closed by the server. This is usually due to an invalid client request.") and apparently the real cause is connected to the fact that the input_raw_exn function in git_sync.mlalways tried to read 4 bytes but, at that specific step, the ic buffer is actually empty.

For the sake of clarity, the aforementioned error disappeared because I've changed in the IO.read_exactly code to use Lwt_io.read_into in place of Lwt_io.read_into_exactly. A the moment I don't know if I broke something by doing this, but the rest of the code works as expected.

Don't know if this is useful or not, just wanted to share what I found so far.

@samoht
Member
samoht commented Nov 17, 2016

That's pretty useful, thanks! There is maybe an issue in the way ocaml-git handle short reads.

@kayceesrk
Contributor

@samoht has there been any progress on this issue?

@samoht
Member
samoht commented Jan 3, 2017

Unfortunately not...

@kayceesrk
Contributor

Ah well. Spent a few hours in vain debugging this. The other end terminates connection after ocaml-git sends the refs to be updated. Related: mirage/ocaml-git#139

@kayceesrk kayceesrk referenced this issue in kayceesrk/ezirmin Jan 3, 2017
Open

Sync.push is broken #4

@samoht samoht added bug git labels Feb 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment