New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uncaught exception Failure("input_value: bad bigarray kind") #94
Comments
This happens when unison is not compiled with the same version of ocaml on both machines. I don't know the ocaml version used in debian testing, but you should install the same one to compile in debian wheezy. |
Hi Well they actually sync with some other profiles. So they do not seem to have this issue always. This issue happens only with one of the sync profiles I use. |
If I remember correctly, it does not happen all the time. It depends on what is being synchronized. |
Thanks for your reply. I will try it with the proper builds. |
This problem has not been fixed in many years and it is extremely frustrating that an otherwise very nice piece of software that you rely on suddenly just breaks after some update and becomes useless. Would you like if your phone started crashing when you tried to call people that had phones with a newer or older version of Android than you have? Because that is how unison currently behaves. @bcpierce00, please devote all your effort into solving this bug once and for all. So in an effort to help come to the bottom of this problem I have some questions:
NB, I am not interested in the transport layer parts, e.g. ssh or socket, nor the rsync file content synchronisation part. Only the unison link layer protocol.
I have reviewed all the other open issues, and in my opinion none of them are more important to solve than this (with failing on symlinks being the most serious candidate to be almost as important). I have been using unison for many years (I know for sure I used it to sync between my laptop and desktop when studying, finished in 2006) but I do not know ocaml (I have mainly programmed in C). |
I do not know if it helps but I encountered this bug and explored a bit, while synchronizng on 2 computers : debian testing and ubuntu 17.04 It is a strange bug as only one file can make the difference, even on a "fresh" synchronization states. It is a simple text file, a Makefile text. Strange no ? What can I do to help more ? This is what I did 3) I comment the specific rule to ignore the file => it bugs $> unison config_file Proceed with propagating updates? [] y Fatal error: Lost connection with the server 4) add again the specific ignore rule => it works !! $> unison nacon_Textes |
Do you know what version(s) of the OCaml compiler was/were used on the two sides? |
This is hard to tell. I can say that the package version of ocaml on ubuntu 17.04 is 4.02.3-6 and it is 4.02.3-10 on debian testing. I assume these are the version used.... But libraries have also version numbers. For example, on ubuntu 17.04, liblwt-ocaml-dev is 2.5.2-1 and 2.5.2-2 on debian testing. |
OK, so it seems likely that the ocaml version is the same on both sides. And if you invoke Unison with "-path ..." to select just the file that failed in your previous tests, does it succeed or fail? (And if it fails, then can you do the same thing and also add "-debug all" and post the resulting trace here or send it to me?) |
(And, for good measure, maybe also add "-debug verbose" to get an even longer trace and post/send that too?) |
Sorry, I forgot... '$> unison -debug verbose -root ssh://dutech@nacon//home/dutech/Loria -root /home/dutech/Work/Loria -path Textes/Presentations/161004_MDP_INRA/Makefile > stdout_dump.txt 2> stderr_dump.txt' That gives me these two dump files. Hope that helps. |
I stumbled upon this too. I have two versions of a SQLite file. Sync it one way and it crashes. Do it the other way around and it works. The file size is about 8MB. Machine 1
Machine 2
Result
Absolutely reproducible. I could also provide those files in private if desired. Aiko |
Very interesting!
Yes, could you send the files to me? I don’t have your exact configuration to experiment with, but at least I can see if the problem can be replicated on my setup.
- Benjamin
… On Nov 17, 2017, at 4:07 PM, Aiko Barz ***@***.*** ***@***.***>> wrote:
I stumbled upon this too. I have two versions of a SQLite file. Sync it one way and it crashes. Do it the other way around and it works. The file size is about 8MB.
Machine 1
Debian/SID 64Bit
Unison 2.48.4-1
Machine 2
Debian/Stretch 64Bit
Unison 2.48.3-1
Result
Unison failed: Uncaught exception Failure("input_value: bad bigarray kind")
Raised by primitive operation at file "/tmp/buildd/unison-2.48.3/remote.ml", line 453, characters 18-45
Called from file "/tmp/buildd/unison-2.48.3/remote.ml", line 459, characters 23-61
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 75, characters 20-23
Re-raised at file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 135, characters 12-13
Called from file "list.ml", line 73, characters 12-15
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 31, characters 2-37
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 83, characters 17-46
Called from file "/tmp/buildd/unison-2.48.3/lwt/generic/lwt_unix_impl.ml", line 147, characters 6-40
Called from file "/tmp/buildd/unison-2.48.3/main.ml", line 202, characters 6-24
Called from file "/tmp/buildd/unison-2.48.3/main.ml", line 131, characters 4-9
Absolutely reproducible. I could also provide those files in private if desired.
Aiko
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGsIC0TFS1gkl06H1sWitiMRXojYK6sfks5s3fWKgaJpZM4PaOvw>.
<https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png> <https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png> <https://github.com/bcpierce00/unison> <#94 (comment)>
|
I was having this issue after editing tags of some FLAC files. I'm now using precompiled binaries from Charlie Herron on both machines (using the Thanks for Unison! |
OK, I tried with these files, but on my setup everything works.
The issue is likely to be the usual one: The Unison binaries on the two machines were probably compiled with incompatible versions of OCaml (one < 4.02 and one >= 4.02). The fix is to get newer binaries (or compile your own).
- Benjamin
… On Nov 17, 2017, at 4:07 PM, Aiko Barz ***@***.*** ***@***.***>> wrote:
I stumbled upon this too. I have two versions of a SQLite file. Sync it one way and it crashes. Do it the other way around and it works. The file size is about 8MB.
Machine 1
Debian/SID 64Bit
Unison 2.48.4-1
Machine 2
Debian/Stretch 64Bit
Unison 2.48.3-1
Result
Unison failed: Uncaught exception Failure("input_value: bad bigarray kind")
Raised by primitive operation at file "/tmp/buildd/unison-2.48.3/remote.ml", line 453, characters 18-45
Called from file "/tmp/buildd/unison-2.48.3/remote.ml", line 459, characters 23-61
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 75, characters 20-23
Re-raised at file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 135, characters 12-13
Called from file "list.ml", line 73, characters 12-15
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 31, characters 2-37
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 83, characters 17-46
Called from file "/tmp/buildd/unison-2.48.3/lwt/generic/lwt_unix_impl.ml", line 147, characters 6-40
Called from file "/tmp/buildd/unison-2.48.3/main.ml", line 202, characters 6-24
Called from file "/tmp/buildd/unison-2.48.3/main.ml", line 131, characters 4-9
Absolutely reproducible. I could also provide those files in private if desired.
Aiko
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGsIC0TFS1gkl06H1sWitiMRXojYK6sfks5s3fWKgaJpZM4PaOvw>.
<https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png> <https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png> <https://github.com/bcpierce00/unison> <#94 (comment)>
|
@bcpierce00 How would I solve this problem on a shared Linux cluster that I maintain but where a certain number of users expect their own version of Unison (who knows which OS they have locally) to work with whatever is on the cluster? |
I'm afraid you may need to ask people to make sure whatever version of Unison they are using is compiled with an OCaml compiler with version >= 4.02.0. (4.02.0 was released over three years ago, so this is not an outlandish requirement.) |
Hi, I have the same problem, but I believe both versions of unison were compiled with the same OCaml compiler version. One version of unison is from Raspbian (based on Debian Stretch) and the other version is from Ubuntu 16.04 LTS. Assuming on both systems the package maintainer compiles with the platform default version of OCaml, they both should've been compiled from OCaml 4.02.3: Ubuntu 16.04: Raspbian:
|
Ask the package maintainers?
I don't think so, but perhaps @brabalan knows more? |
ok, so I recompiled both packages from sources (took a while on the Pi, but was otherwise easy, thanks to dpkg-buildpackage). It now works. I would have to conclude that either the Raspbian or the Ubuntu package was not compiled with a version of ocaml that's not the default for that distro on that version. |
Hi @stollem, do you have that Raspbian Stretch running on a Pi 3 Model B, by any chance? If so, we'd have the very same setup, and I also have those bug issues. I'd be happy if you could share your compiled packages which resolved this :) |
Many thanks @stollem, unison works perfectly fine again :) |
I'm having the same issue but with a Debian 9 Stretch amd64 server and Kubuntu 16.04 client. I have tried to build unison in Debian Stretch with the same version of Kubuntu, but I'm afraid that I have to rebuild ALL the ocaml-nox dependencies and it's not affordable. I cannot understand why this strong dependency. It has no sense. I should have an unison server and run ANY client to connect. Any idea to solve this? |
I cannot understand why this strong dependency. It has no sense. I should have an unison server and run ANY client to connect.
Yes, this would be nice, but it’s not the way things work, and it’s not easy to fix. In particular, there was an incompatible change to the wire format for marshaled values between OCaml 4.01 and 4.02, so even the same versions of Unison will not work if they are compiled with different compilers from before and after this change.
- B
|
So in other words, the OCaml compiler developers don't give a shit about keeping binary compatibility (for any version, including minor bugfixes). Thus the internal OCaml marshal functionality is fundamentally unreliable and unsuitable as a means for external communication for unison. @bcpierce00, you need to find an external marshalling library which has a stable and versioned interface and use that, so that unison is able to detect and handle version mismatches in a proper way instead of failing catastrophically like it has done for many years due to ocaml's carelessness for binary compatibility. Using an external marshalling library is the only way to achieve reliable communication. Continuing using the internal stuff is guaranteed to result in future problems. |
Not likely I'm going to do that myself, but I'd be happy to consider a PR... |
"What can be done oil the Debian end?" is a question for Stephane. But if you are able to build on both machines it should not be very hard to install the same version of OCaml on both sides and then recompile Unison. |
"What can be done oil the Debian end?" is a question for Stephane. But if you
are able to build on both machines it should not be very hard to install the
same version of OCaml on both sides and then recompile Unison.
Doing this would be interesting to check that if the bug is still there
in that version.
I think the cause may be in fact fixed by commit 829024a ("Follow OCaml
'interface' rule", 2017-06-30). There was a bug in Unison's way of
interfacing OCaml and C (that may not have been "triggerable" earlier).
The commit has not been merged to the unison-2.48 branch. Could you try
to apply to that branch and check again if you manage to trigger the
bug?
|
On 05/09/18 23:26, G.raud wrote:
> "What can be done oil the Debian end?" is a question for Stephane. But if you
> are able to build on both machines it should not be very hard to install the
> same version of OCaml on both sides and then recompile Unison.
Doing this would be interesting to check that if the bug is still there
in that version.
I think the cause may be in fact fixed by commit 829024a ("Follow OCaml
'interface' rule", 2017-06-30). There was a bug in Unison's way of
interfacing OCaml and C (that may not have been "triggerable" earlier).
The commit has not been merged to the unison-2.48 branch. Could you try
to apply to that branch and check again if you manage to trigger the
bug?
I don't know if it could help or not, but unison had a new version (
2.48.3-1+b1) that was just a rebuild with the OCaml version that provided Stretch.
This was because Bug#899189 and it was pushed to the archive in the middle of July.
Could you check it?
…--
--
Linux User 152692 GPG: 05F4A7A949A2D9AA
Catalonia
-------------------------------------
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
|
Hello everyone and thank you for your answers! As mentioned by @lepalom, I've checked the version of the Unison package installed. So, I guess there is a "faulty" version on the Raspberry Pi. As @bcpierce00 mentions, I could also recompile Unison (which I already did a few times before). |
Check here: https://packages.debian.org/stretch/armel/unison/download If I'm not wrong for Raspberry Pi, armel arch is the correct one. |
@Encrypt Raspbian is a derivative of Debian with its own procedures. You should ask them to recompile Unison, they must have a procedure for that. You can refer them to the Debian bug. @lepalom According to this, using Debian's armel (or armhf) binary packages on Raspbian is not recommended. |
Hi, $ sudo apt purge unison
$ wget http://http.us.debian.org/debian/pool/main/u/unison/unison_2.48.3-1+b1_armhf.deb
$ sudo dpkg -i unison_2.48.3-1+b1_armhf.deb I have just filled a report for rasbian team: Regards |
Hello guys! First of all, thank you for the help you have all given me regarding this issue. I have good news though: an "apt-get update && apt-get upgrade" on my Raspberry Pi brought a new Unison package, version 2.48.3-1+b1! I'll have a closer look at how to create bug reports since I have never done any. Cheers! |
First of all, big thanks to the Unison developers for this excellent trustworthy tool I have been using for more than 15 years now!! Then the problem:
Wishlist @bcpierce00 @brabalan
Since the latest Linux binaries are 2.48.4, I could try the Mac version 2.48.15, but there is no indication which Ocaml version has been used for these binary, so I will be just shooting in the dark. |
So @bcpierce00, with which ocaml compiler version were the current binaries built? And can that be found out afterwards or do you "have to know"? Background: I'm trying to connect a mac and an ubuntu machine. The version available via apt is 2.48.3, but using the corresponding mac binary yields the dreaded bigarray error. Okay, so let's get caught up to the currently published 2.51 instead. Whelp, no binaries for linux in sight (see #200), so I'll have try and compile unison for linux myself -- but if I'm going through that trouble, I'd like to make sure I'm at least using a compatible ocaml compiler version. Thanks for your help! |
Doing
unison -version
will show you. But any recent (last few years) compiler should be fine.
- B
… On Dec 17, 2018, at 2:38 PM, Wolfgang Schnerring ***@***.*** ***@***.***>> wrote:
So @bcpierce00 <https://github.com/bcpierce00>, with which ocaml compiler version were the current binaries <https://github.com/bcpierce00/unison/releases/download/v2.51.2/Unison-2.51.2.OS.X.zip> built? And can that be found out afterwards or do you "have to know"?
Background: I'm trying to connect a mac and an ubuntu machine. The version available via apt is 2.48.3, but using the corresponding mac binary <https://github.com/bcpierce00/unison/releases/download/2.48.3/Unison-OS-X-2.48.3.zip> yields the dreaded bigarray error. Okay, so let's get caught up to the currently published 2.51 instead. Whelp, no binaries for linux in sight (see #200 <#200>), so I'll have try and compile unison for linux myself -- but if I'm going through that trouble, I'd like to make sure I'm at least using a compatible ocaml compiler version.
Thanks for your help!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGsICy8iQpOmxNMkuj30CMMu90dEkL5Kks5u5_K-gaJpZM4PaOvw>.
|
Thanks, I'll try and check But... you do realize that "any compiler of the last few years" is utterly incorrect? For example: The 2.48.3 OSX binary was "released on 29 Sep 2016", and the corresponding Debian binary is from around August 2015. Does this not both count as "last few years"? However they do not work together, which is the whole point of this long, long bug report right here. (By those release dates I'm guessing they were compiled with 4.03 and 4.02, respectively, but I don't know because that |
The switch to OCaml 4.02, which is where the breakage occurred IIRC, was in 2014. So, indeed, I should have said "last two or three years." |
Ah. So it's not every compiler version that breaks the serialization. That's good to know (earlier comments here sounded way more ominous to me about how strict the compatibility dependency on the compiler might be). With these hints, I've been quite successful!
Thanks again for your help! But yeah, another vote from me to send the Debian folks some cookies and get 2.51 included there (#200). This is really quite the hassle, I really enjoy using unison, but the way to get there is currently rather rocky. |
With unison version 2.48.3 on MacOSX and unison version 2.48.3 on Debian (yes, 2.48.3 on both sides) |
Interesting, it worked with 2.48.3 on raspi and static compiled 2.48.4 on ubuntu for many years. After editing an encrypted file through |
I hit this error using 2.40 installed from standard repository on a Fedora 29 laptop when syncing to a QNAP NAS. Other newer versions I tried to build failed in different ways, and likewise on Fedora 29, so getting a comparable version of Unison seemed fairly difficult. It seems awkward that Unison does not have an API that provides better compability i.e. unison versions can differ but still be compatible. Anyway now I hit error "input_value: bad bigarray kind" when syncing a single text file.... Sigh. On NAS : NAS unison built with "make UISTYLE=text NATIVE=false" On Fedora 29 : |
One solution would be to migrate from using https://caml.inria.fr/pub/docs/manual-ocaml/libref/Marshal.html which may be dependant on the ocaml compiler version to something with a specification, maybe https://github.com/janestreet/bin_prot |
Not just one solution. Discarding the fundamentaly unreliable internal ocaml marshaling and replace it with something else that is reliably is the only solution. My very strong suggestion is to use protobuf since that is a extremely mature project and is well documented. Started internally at Google in 2001 and publicly released in 2008. It has received a massive amount of testing; more testing than any other alternative suggested here. And there is zero risk that one single maintainer of a library looses interest and stops maintaining it. Protobuf will be maintained longer than unison will. https://ocamlverse.github.io/content/protocols.html has the following entry for protobuf:
Again, this bug should be the top priority. |
I’d be interested in switching to something more stable than the Marshal library has turned out to be. My hesitations are:
(0) I do not have time to do this myself. Someone else would need to take it on.
(1) It would be a shame to increase the difficulty of building Unison. Right now, all it takes to build the plain text UI version is a bare OCaml compiler. Any change that breaks this property should come with a very distinct improvement. Ditto any change that requires maintainers / developers to understand new external packages.
(2) Switching to something better than Marshal will, unfortunately, not help compatibility with all the older versions that are out there. (Conversely, it won’t make the compatibility story for older versions any worse either!)
(3) Clearly, the original Unison design didn’t do a good enough job of thinking about cross-version compatibility. If one were going to revamp the protocol, there would be a strong temptation to try to do better on this aspect. But, again, this is not something I can put energy into. (Switching to a more stable library will at least provide continuity across OCaml versions, though, so it may be worth doing even without trying to deal with compatibility across Unison versions.)
Best,
- Benjamin
… On Feb 1, 2020, at 6:23 AM, hlovdal ***@***.***> wrote:
Not just one solution. Discarding the fundamentaly unreliable internal ocaml marshaling and replace it with something else that is reliably is the only solution.
My very strong suggestion is to use protobuf <https://en.wikipedia.org/wiki/Protocol_Buffers> since that is a extremely mature project and is well documented <https://developers.google.com/protocol-buffers/>. Started internally at Google in 2001 and publicly released in 2008. It has received a massive amount of testing; more testing than any other alternative suggested here. And there is zero risk that one single maintainer of a library looses interest and stops maintaining it. Protobuf will be maintained longer than unison will.
https://ocamlverse.github.io/content/protocols.html <https://ocamlverse.github.io/content/protocols.html> has the following entry for protobuf:
Protobuf
ocaml-protoc-plugin <https://github.com/issuu/ocaml-protoc-plugin/>: Full findings to Protobuf messages and types. Implements the full proto3 specification.
protoc <https://github.com/mransan/ocaml-protoc>: Compiler from protobuf files to OCaml types.
ppx_deriving_protobuf <https://github.com/ocaml-ppx/ppx_deriving_protobuf>: Derive Protobuf files from OCaml types.
Again, this bug should be the top priority.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#94?email_source=notifications&email_token=ABVQQC25OZHAFTZV22HNOX3RAVLTLA5CNFSM4D3I5PYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKQ2XCQ#issuecomment-581020554>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVQQC7KS7HG3HZQFKPGJJLRAVLTLANCNFSM4D3I5PYA>.
|
Glad to hear that!
I gave it a try, using ppx_deriving_protobuf: https://github.com/glondu/unison/tree/protobuf It took me two days. It seems to work (at least on Linux), but needs more testing.
Installing build-dependencies is easy with opam, but it might indeed be difficult to bring them in a non-opam environment, such as when packaging for another distribution. Updating the Makefile-based build system turned out to be very easy, thanks to ocamlfind. I don't know dune enough to update the dune-based build system.
My goal here was for Unison to keep compatibility with itself compiled with another OCaml version. Technically, my proposal moves the compatibility issue to ppx_deriving_protobuf itself, but the ppx ecosystem seems to be well-behaved and supported (but only time will tell).
This is not what I did. My changes are simple and systematic (even though the diff is big). Changes that could possibly affect performance or usability are:
Notably on 32 bit systems, some limits might be reached more easily. |
Hello, I'm trying to use unison for the first time and I can't because of this particular issue. (Trying between debian buster and debian bullseye). I ended up compiling unison version 2.48.4 statically to try to get around this problem. But I'm still running into trouble.
While searching for this error, I was lead here. Is this somehow the same marshaling problem or is this a separate issue all together? Thanks |
These two are using different versions of OCaml, so expect trouble.
Do you use the same binary on both sides? (Did you set "servercmd" on the local side so that it points to the right binary on the remote side?) |
One thing that can be done immediately to prepare to switch to something other than the internal ocaml marshalling is to write code to negotiate the marchalling algorithm to use when the program starts a connection. Unless the existing, olders clients does not have a problem with such a new negotiation addition to the communication, the sending part should be disabled when making releases, but by having all versions since now ready to receive will make switching later just work out of the box for all the latest releases. |
Yes, in fact I had already removed the debian packages of unison from both sides, so it couldn't work otherwise. I also tried this using the socket mode (where I explicitly called the correct unison binary on the server side), and it gave me the same results. |
@glondu I see that you are experimenting with protobuf and bin-prot. I'd be happy to give a try, which one of the solution do you think is more promising? |
@niol protobuf and bin-prot had their own shortcomings, I eventually ended up with my own implementation: https://github.com/glondu/unison/tree/umarshal |
Debian testing x64 Unison 2.48.3(Deb packages) <-> Debian Wheezy x32 Unison 2.48.4 (Github build)
Uncaught exception Failure("input_value: bad bigarray kind")
Raised by primitive operation at file "/tmp/buildd/unison-2.48.3/remote.ml", line 453, characters 18-45
Called from file "/tmp/buildd/unison-2.48.3/remote.ml", line 459, characters 23-61
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 75, characters 20-23
Re-raised at file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 135, characters 12-13
Called from file "list.ml", line 73, characters 12-15
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 31, characters 2-37
Called from file "/tmp/buildd/unison-2.48.3/lwt/lwt.ml", line 83, characters 17-46
Called from file "/tmp/buildd/unison-2.48.3/lwt/generic/lwt_unix_impl.ml", line 147, characters 6-40
Called from file "/tmp/buildd/unison-2.48.3/uigtk2.ml", line 3581, characters 6-125
Called from file "/tmp/buildd/unison-2.48.3/uigtk2.ml", line 156, characters 18-22
The text was updated successfully, but these errors were encountered: