[Mono] Compression broken on OS X

[haf]

trying some utility functions/gzip_encode/gzip_decode: Failed: 
gzip_encode/gzip_decode-Falsifiable, after 2 tests (0 shrinks) (StdGen (823287983,295851370)):
""
with exception:
System.IO.IOException: Internal error (no progress possible) ReadInternal
  at System.IO.Compression.DeflateStreamNative.CheckResult (Int32 result, System.String where) [0x00000] in <filename unknown>:0 
  at System.IO.Compression.DeflateStreamNative.ReadZStream (IntPtr buffer, Int32 length) [0x00000] in <filename unknown>:0 
  at System.IO.Compression.DeflateStream.ReadInternal (System.Byte[] array, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.Compression.DeflateStream.Read (System.Byte[] dest, Int32 dest_offset, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.Compression.GZipStream.Read (System.Byte[] dest, Int32 dest_offset, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.Stream.CopyTo (System.IO.Stream destination, Int32 bufferSize) [0x00000] in <filename unknown>:0 
  at System.IO.Stream.CopyTo (System.IO.Stream destination) [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.IO.Stream:CopyTo (System.IO.Stream)
  at Suave.Utils+Compression.gzip_decode (System.Byte[] bytes) [0x00000] in <filename unknown>:0 
  at Suave.Tests.Utils+utilities@19-2.Invoke (System.String str) [0x00000] in <filename unknown>:0 
  at FsCheck.Testable.evaluate[String,Unit] (Microsoft.FSharp.Core.FSharpFunc`2 body, System.String a) [0x00000] in <filename unknown>:0 


 (00:00:00.0522263)
27 tests run: 26 passed, 0 ignored, 1 failed, 0 errored (00:00:03.6112184)

This is when we're running compression in FsCheck property which tries many different combinations of the same code path. The unit test that doesn't run it in a loop doesn't seem to fail, so it might be some global variable that messes it up, or there might be something that is not threadsafe in mono.

[haf]

So it seems this error comes from misrepresentation of lengths. The stack trace shows a call to CopyTo, which presumably calls Length on the source stream. Perhaps the source stream has a bug that mis-represents how long it is, I suppose the CopyTo method will read more than what's possible.

Mono links against zlib statically, and here's the description of Z_BUF_ERROR:

 Z_BUF_ERROR if no progress is possible (for example avail_in or avail_out was zero). Note that Z_BUF_ERROR is not fatal, and deflate() can be called again with more input and more output space to continue compressing. 

from http://www.zlib.net/manual.html

And here and here's the code in question that links to zlib:

Running on OS X 10.8.5, zlib: version 1.2.5, April 19th, 2010

Mono version 3.4.0 ((no/d4511ef Tue Mar 25 14:35:52 EDT 2014)

Commit 8dd19ae

[haf]

This is not a problem on CentOS 6.5.

[haf]

Categorising as minor bug because it's only relevant on non-production OSes.

[haf]

Now it's a problem on CentOS, too. Upgrading to major. Filed a bug https://bugzilla.xamarin.com/show_bug.cgi?id=22346

[ademar]

Yeap, started ocurring on Arch Linux as well

[ademar@cygnus suave]$ mono -V
Mono JIT compiler version 3.4.0 (tarball Wed Jun 25 20:39:03 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        LLVM:          supported, not enabled.
        GC:            sgen
[ademar@cygnus suave]$ uname -a
Linux cygnus 3.15.3-1-ARCH #1 SMP PREEMPT Tue Jul 1 07:32:45 CEST 2014 x86_64 GNU/Linux

[ademar]

Perhaps we should use a different compression lib ?

[haf]

Yes we should - perhaps we can use a .Net-only compression lib inside mono, even? Do you think they'd accept that as a PR? Right now they're using a C-based lib.

[ademar]

Yeah, I think it makes sense. I wonder why it has not been done already.

[ademar]

I think the bug is here https://github.com/mono/mono/blob/master/mcs/class/System/System.IO.Compression/DeflateStream.cs#L460

It should not throw upon Z_BUF_ERROR

according to the docs

deflate() returns Z_OK if some progress has been made ... Z_BUF_ERROR if no progress is possible (for example avail_in or avail_out was zero). Note that Z_BUF_ERROR is not fatal, and deflate() can be called again with more input and more output space to continue compressing.

http://www.zlib.net/manual.html

[ademar]

The problem here was that we were passing an empty array to GZipStream.