Error: zlib_deflate with Haxe 3.4.0-rc.1 and zlib 1.2.10, for Flash 11+ #5941

Closed
demurgos opened this Issue Jan 16, 2017 · 29 comments

Projects

None yet

4 participants

@demurgos
demurgos commented Jan 16, 2017 edited

Hi,
I updated the packages on my system yesterday and got the new version of Haxe (3.4.0-rc.1 instead of 3.3.0) and got a new version of zlib (1.2.10). Updating to these versions broke my toolchain, I simply receive the error zlib_deflate. I am fairly new to Haxe so I couldn't find the exact cause of the issue, but I created a minimal repository demonstrating the issue.

The following command fails:

$ haxe -main Main -swf-version 11 -debug -swf main.swf
Error: zlib_deflate

The error message is very cryptic, is there a way to get a better error message with a flag ?

If I remove "-debug", it works: haxe -main Main -swf-version 11 -swf main.swf. If I change the version to 9 or 10, it works too: haxe -main Main -swf-version 9 -debug -swf main.swf. If I change the version to 12, it fails.

So the error seems to be related to the debug option with Flash 11+ and zlib.

I am certain that it is a regression of Haxe or zlib: I compiled my project, wanted to install a package for my project so I first ran a system update, then I tried to compile my project again (without any change) and it started to fail.

Here is a copy of the README.md of the repository above (so there will always be a trace even if I delete the repo of issue). This repository simply contains a Main.hx file with the Hello World example of the documentation.


Example zlib issue

This repo demonstrate an issue with zlib, haxe and Flash.

The following command fails:

haxe -main Main -swf-version 11 -debug -swf main.swf

Example output:

# System
$ uname -a
Linux cyan 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux

# List the installed packages
$ pacman -Q | grep --regexp=haxe --regexp="zlib"
haxe 3.4.0rc1-1
lib32-zlib 1.2.10-1
zlib 1:1.2.10-1

# Haxe version
$ haxe -version
3.4.0

# Try to build (added verbose: -v)
$ haxe -main Main -swf-version 11 -debug -swf main.swf -v
Classpath : ;/usr/share/haxe/std/flash/_std/;/usr/share/haxe/std/
Defines : true;source-header=Generated by Haxe 3.4.0;haxe_ver=3.4;haxe3;flash9;flash11;flash10_3;flash10_2;flash10_1;flash10;flash;debug;dce=std
Parsed /usr/share/haxe/std/StdTypes.hx
Parsed /usr/share/haxe/std/String.hx
Parsed /usr/share/haxe/std/Array.hx
Parsed /usr/share/haxe/std/haxe/EnumTools.hx
Parsed /usr/share/haxe/std/flash/_std/Std.hx
Parsed /usr/share/haxe/std/flash/Boot.hx
Parsed Main.hx
Parsed /usr/share/haxe/std/Class.hx
Parsed /usr/share/haxe/std/flash/display/MovieClip.hx
Parsed /usr/share/haxe/std/flash/display/Sprite.hx
Parsed /usr/share/haxe/std/flash/display/DisplayObjectContainer.hx
Parsed /usr/share/haxe/std/flash/display/InteractiveObject.hx
Parsed /usr/share/haxe/std/flash/display/DisplayObject.hx
Parsed /usr/share/haxe/std/flash/events/EventDispatcher.hx
Parsed /usr/share/haxe/std/flash/display/IBitmapDrawable.hx
Parsed /usr/share/haxe/std/flash/events/IEventDispatcher.hx
Parsed /usr/share/haxe/std/flash/events/Event.hx
Parsed /usr/share/haxe/std/flash/accessibility/AccessibilityProperties.hx
Parsed /usr/share/haxe/std/flash/display/BlendMode.hx
Parsed /usr/share/haxe/std/flash/display/Shader.hx
Parsed /usr/share/haxe/std/flash/filters/BitmapFilter.hx
Parsed /usr/share/haxe/std/flash/display/LoaderInfo.hx
Parsed /usr/share/haxe/std/UInt.hx
Parsed /usr/share/haxe/std/flash/geom/Rectangle.hx
Parsed /usr/share/haxe/std/flash/display/Stage.hx
Parsed /usr/share/haxe/std/flash/geom/Transform.hx
Parsed /usr/share/haxe/std/flash/geom/Point.hx
Parsed /usr/share/haxe/std/flash/geom/Vector3D.hx
Parsed /usr/share/haxe/std/flash/accessibility/AccessibilityImplementation.hx
Parsed /usr/share/haxe/std/flash/ui/ContextMenu.hx
Parsed /usr/share/haxe/std/flash/text/TextSnapshot.hx
Parsed /usr/share/haxe/std/flash/display/Graphics.hx
Parsed /usr/share/haxe/std/flash/media/SoundTransform.hx
Parsed /usr/share/haxe/std/flash/display/FrameLabel.hx
Parsed /usr/share/haxe/std/flash/display/Scene.hx
Parsed /usr/share/haxe/std/flash/utils/Object.hx
Parsed /usr/share/haxe/std/flash/text/TextField.hx
Parsed /usr/share/haxe/std/flash/errors/Error.hx
Parsed /usr/share/haxe/std/haxe/PosInfos.hx
Parsed /usr/share/haxe/std/flash/text/AntiAliasType.hx
Parsed /usr/share/haxe/std/flash/text/TextFieldAutoSize.hx
Parsed /usr/share/haxe/std/flash/text/TextFormat.hx
Parsed /usr/share/haxe/std/flash/text/GridFitType.hx
Parsed /usr/share/haxe/std/flash/text/StyleSheet.hx
Parsed /usr/share/haxe/std/flash/text/TextInteractionMode.hx
Parsed /usr/share/haxe/std/flash/text/TextFieldType.hx
Parsed /usr/share/haxe/std/flash/text/TextLineMetrics.hx
Parsed /usr/share/haxe/std/flash/text/TextFormatAlign.hx
Parsed /usr/share/haxe/std/flash/text/TextFormatDisplay.hx
Parsed /usr/share/haxe/std/flash/display/BitmapData.hx
Parsed /usr/share/haxe/std/flash/geom/Matrix.hx
Parsed /usr/share/haxe/std/flash/display/GradientType.hx
Parsed /usr/share/haxe/std/flash/display/SpreadMethod.hx
Parsed /usr/share/haxe/std/flash/display/InterpolationMethod.hx
Parsed /usr/share/haxe/std/flash/Vector.hx
Parsed /usr/share/haxe/std/flash/display/IGraphicsData.hx
Parsed /usr/share/haxe/std/flash/display/GraphicsPathWinding.hx
Parsed /usr/share/haxe/std/flash/display/TriangleCulling.hx
Parsed /usr/share/haxe/std/flash/display/LineScaleMode.hx
Parsed /usr/share/haxe/std/flash/display/CapsStyle.hx
Parsed /usr/share/haxe/std/flash/display/JointStyle.hx
Parsed /usr/share/haxe/std/flash/geom/ColorTransform.hx
Parsed /usr/share/haxe/std/flash/utils/ByteArray.hx
Parsed /usr/share/haxe/std/flash/display/StageQuality.hx
Parsed /usr/share/haxe/std/flash/utils/IDataOutput2.hx
Parsed /usr/share/haxe/std/flash/utils/IDataInput2.hx
Parsed /usr/share/haxe/std/flash/utils/IDataOutput.hx
Parsed /usr/share/haxe/std/flash/utils/Endian.hx
Parsed /usr/share/haxe/std/flash/utils/IDataInput.hx
Parsed /usr/share/haxe/std/flash/utils/CompressionAlgorithm.hx
Parsed /usr/share/haxe/std/flash/display/NativeMenu.hx
Parsed /usr/share/haxe/std/flash/ui/ContextMenuBuiltInItems.hx
Parsed /usr/share/haxe/std/flash/ui/ContextMenuClipboardItems.hx
Parsed /usr/share/haxe/std/flash/net/URLRequest.hx
Parsed /usr/share/haxe/std/flash/net/URLRequestHeader.hx
Parsed /usr/share/haxe/std/flash/geom/Matrix3D.hx
Parsed /usr/share/haxe/std/flash/geom/PerspectiveProjection.hx
Parsed /usr/share/haxe/std/flash/geom/Orientation3D.hx
Parsed /usr/share/haxe/std/flash/display/StageAlign.hx
Parsed /usr/share/haxe/std/flash/display/ColorCorrection.hx
Parsed /usr/share/haxe/std/flash/display/ColorCorrectionSupport.hx
Parsed /usr/share/haxe/std/flash/display/StageDisplayState.hx
Parsed /usr/share/haxe/std/flash/display/StageScaleMode.hx
Parsed /usr/share/haxe/std/flash/display/Stage3D.hx
Parsed /usr/share/haxe/std/flash/media/StageVideo.hx
Parsed /usr/share/haxe/std/flash/media/AVStream.hx
Parsed /usr/share/haxe/std/flash/media/Camera.hx
Parsed /usr/share/haxe/std/flash/net/NetStream.hx
Parsed /usr/share/haxe/std/flash/net/NetStreamInfo.hx
Parsed /usr/share/haxe/std/flash/net/NetStreamMulticastInfo.hx
Parsed /usr/share/haxe/std/flash/media/VideoStreamSettings.hx
Parsed /usr/share/haxe/std/flash/net/NetConnection.hx
Parsed /usr/share/haxe/std/flash/media/Microphone.hx
Parsed /usr/share/haxe/std/flash/net/NetStreamPlayOptions.hx
Parsed /usr/share/haxe/std/flash/media/SoundCodec.hx
Parsed /usr/share/haxe/std/flash/media/MicrophoneEnhancedOptions.hx
Parsed /usr/share/haxe/std/flash/media/MicrophoneEnhancedMode.hx
Parsed /usr/share/haxe/std/flash/net/Responder.hx
Parsed /usr/share/haxe/std/flash/media/AVCaptionStyle.hx
Parsed /usr/share/haxe/std/flash/media/AVPlayState.hx
Parsed /usr/share/haxe/std/flash/media/AVSource.hx
Parsed /usr/share/haxe/std/flash/media/AVResult.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3D.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DProfile.hx
Parsed /usr/share/haxe/std/flash/display3D/textures/CubeTexture.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DTextureFormat.hx
Parsed /usr/share/haxe/std/flash/display3D/IndexBuffer3D.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DBufferUsage.hx
Parsed /usr/share/haxe/std/flash/display3D/Program3D.hx
Parsed /usr/share/haxe/std/flash/display3D/textures/RectangleTexture.hx
Parsed /usr/share/haxe/std/flash/display3D/textures/Texture.hx
Parsed /usr/share/haxe/std/flash/display3D/VertexBuffer3D.hx
Parsed /usr/share/haxe/std/flash/display3D/textures/VideoTexture.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DBlendFactor.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DTriangleFace.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DCompareMode.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DProgramType.hx
Parsed /usr/share/haxe/std/flash/display3D/textures/TextureBase.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DWrapMode.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DTextureFilter.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DMipFilter.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DStencilAction.hx
Parsed /usr/share/haxe/std/flash/display3D/Context3DVertexBufferFormat.hx
Parsed /usr/share/haxe/std/flash/display/ActionScriptVersion.hx
Parsed /usr/share/haxe/std/flash/system/ApplicationDomain.hx
Parsed /usr/share/haxe/std/flash/display/Loader.hx
Parsed /usr/share/haxe/std/flash/events/UncaughtErrorEvents.hx
Parsed /usr/share/haxe/std/flash/system/LoaderContext.hx
Parsed /usr/share/haxe/std/flash/system/ImageDecodingPolicy.hx
Parsed /usr/share/haxe/std/flash/system/SecurityDomain.hx
Parsed /usr/share/haxe/std/flash/display/ShaderData.hx
Parsed /usr/share/haxe/std/flash/display/ShaderPrecision.hx
Parsed /usr/share/haxe/std/flash/events/EventPhase.hx
Parsed /usr/share/haxe/std/EnumValue.hx
Parsed /usr/share/haxe/std/Enum.hx
Typing haxe.EnumTools.getName
Parsed /usr/share/haxe/std/flash/_std/Type.hx
Typing Type.getEnumName
Typing Type.getClassName
Typing Type.getClass
Typing Type.getEnum
Typing Type.getSuperClass
Typing Type.resolveClass
Typing Type.resolveEnum
Typing Type.createInstance
Typing Type.createEmptyInstance
Typing flash.Boot.skip_constructor
Typing Type.createEnum
Parsed /usr/share/haxe/std/flash/_std/Reflect.hx
Parsed /usr/share/haxe/std/haxe/Constraints.hx
Typing Reflect.isFunction
Typing Reflect.callMethod
Typing Reflect.hasField
Typing Reflect.field
Typing Reflect.setField
Typing Reflect.getProperty
Typing Reflect.setProperty
Typing Reflect.fields
Typing Reflect.compare
Typing Reflect.compareMethods
Typing Reflect.isObject
Typing Reflect.isEnumValue
Typing Reflect.deleteField
Typing Reflect.copy
Typing Reflect.makeVarArgs
Typing Type.createEnumIndex
Typing Type.describe
Parsed /usr/share/haxe/std/flash/xml/XML.hx
Parsed /usr/share/haxe/std/flash/xml/XMLList.hx
Parsed /usr/share/haxe/std/flash/utils/Namespace.hx
Parsed /usr/share/haxe/std/flash/utils/Function.hx
Parsed /usr/share/haxe/std/IntIterator.hx
Typing Std.string
Typing flash.Boot.__string_rec
Typing IntIterator.new
Typing IntIterator.hasNext
Typing IntIterator.next
Typing Type.getInstanceFields
Typing Type.getClassFields
Typing Type.getEnumConstructs
Typing Type.typeof
Typing Std.int
Typing Type.enumEq
Typing Type.enumConstructor
Typing Type.enumParameters
Typing Type.enumIndex
Typing Type.allEnums
Typing haxe.EnumTools.createByName
Typing haxe.EnumTools.createByIndex
Typing haxe.EnumTools.createAll
Typing haxe.EnumTools.getConstructors
Typing haxe.EnumValueTools.equals
Typing haxe.EnumValueTools.getName
Typing haxe.EnumValueTools.getParameters
Typing haxe.EnumValueTools.getIndex
Typing flash._Boot.RealBoot.new
Parsed /usr/share/haxe/std/flash/Lib.hx
Typing flash.Boot.start
Typing flash.Boot.doInitDelay
Typing flash.Boot.init
Typing flash.Lib.getTimer
Typing flash.Lib.eval
Typing flash.Lib.getURL
Typing flash.Lib.fscommand
Typing flash.Lib.trace
Typing flash.Lib.describeType
Typing flash.Lib.attach
Typing flash.Lib.as
Typing flash.Lib.redirectTraces
Parsed /usr/share/haxe/std/flash/external/ExternalInterface.hx
Parsed /usr/share/haxe/std/haxe/Log.hx
Typing haxe.Log.trace
Typing flash.Boot.__trace
Typing flash.Boot.getTrace
Parsed /usr/share/haxe/std/flash/filters/GlowFilter.hx
Typing flash.Lib.traceToConsole
Typing haxe.Log.clear
Typing flash.Boot.__clear_trace
Typing haxe.Log.setColor
Typing flash.Boot.__set_trace_color
Typing _UInt.UInt_Impl_.shr
Typing flash.Vector.ofArray
Typing flash.Vector.convert
Typing flash.Boot.IN_E
Typing flash.Boot.enum_to_string
Typing flash.Boot.__instanceof
Typing flash.Boot.__unprotect__
Typing flash.Boot.mapDynamic
Typing Std.is
Typing flash.Boot.filterDynamic
Typing flash.Boot.__init__
Typing Std.instance
Typing Std.parseInt
Typing Std.parseFloat
Typing Std.random
Parsed /usr/share/haxe/std/Math.hx
Typing Math.__init__
Typing Main.main
Parsed /usr/share/haxe/std/StdTypes.hx
Parsed /usr/share/haxe/std/String.hx
Parsed /usr/share/haxe/std/Array.hx
Parsed /usr/share/haxe/std/haxe/EnumTools.hx
Parsed /usr/share/haxe/std/Reflect.hx
Parsed /usr/share/haxe/std/haxe/Constraints.hx
Parsed /usr/share/haxe/std/EnumValue.hx
Parsed /usr/share/haxe/std/Enum.hx
Typing haxe.EnumTools.getName
Parsed /usr/share/haxe/std/Type.hx
Parsed /usr/share/haxe/std/Class.hx
Typing haxe.EnumTools.createByName
Typing haxe.EnumTools.createByIndex
Typing haxe.EnumTools.createAll
Typing haxe.EnumTools.getConstructors
Typing haxe.EnumValueTools.equals
Typing haxe.EnumValueTools.getName
Typing haxe.EnumValueTools.getParameters
Typing haxe.EnumValueTools.getIndex
Parsed /usr/share/haxe/std/Std.hx
Generating swf: main.swf
Error: zlib_deflate

# Print exit code of the haxe command
$ echo $?
1
@demurgos
demurgos commented Jan 16, 2017 edited

Possibly related lines of code:

I don't know how to run OCaml, but adding some exception logic or error handling here could help ? Or maybe it is just about building Haxe against the new version of zlib ?

@Simn
Member
Simn commented Jan 16, 2017

Can you set the environment variable OCAMLRUNPARAM=b and see if this gives you a stacktrace?

@demurgos
demurgos commented Jan 16, 2017 edited

I tried to update to zlib 1.2.11 but I still have the issue.

@Simn
Thank you for your quick answer. I tried the following command, but it gave me exactly the same result (failed):

$ OCAMLRUNPARAM=b haxe -main Main -swf-version 11 -debug -swf main.swf -v

Edit: removed output (it's exactly the same as in the first post: no additional stack trace)

@Simn
Member
Simn commented Jan 17, 2017

I just noticed that your sample is literally just Hello World. Needless to say I cannot reproduce this.

How did you install Haxe?

@demurgos

I installed haxe with my package manager. I am using pacman (for Arch Linux).

# Run as root
pacman -S haxe

Here is the package page. (I output the version of Haxe according to pacman and haxe in my first post).

The problem also occurs with my "real" project, but I was able to reduce it simply to the Hello World example from the manual and a few command line options. This is why I am almost certain that this is a regression from the previous version.

@Simn
Member
Simn commented Jan 17, 2017

@andyli: Any idea what could be going wrong here? The related code hasn't changed in years, so I can only assume this is some setup problem.

@demurgos

Well, just as an update: I made a fresh new installation of the whole system on a Virtual Machine, then just ran:

pacman -S haxe
git clone https://github.com/demurgos/haxe-zlib-issue.git
cd haxe-zlib-issue
haxe -main Main -swf-version 11 -debug -swf main.swf

And it failed: Error: zlib_deflate.

My main guess is that there was a change in the ABI (or API but it is less likely) of zlib.

@Simn What is your system ? Which version of Haxe and zlib do you have ?

@andyli
Member
andyli commented Jan 18, 2017

It's likely due to zlib updated as you said.
Since the haxe package has already flagged as outdated, hopefully the maintainer (Alexander Rødseth) will update it to 3.4.0.rc2 soon and eventually rebuilt with the latest zlib.

@andyli
Member
andyli commented Jan 18, 2017 edited

BTW, zlib 1.2.10 is indeed buggy and got some fixes in 1.2.11 according to its changelog:

Version 1.2.11 has these key improvements over 1.2.10:
Fix deflate stored bug when pulling last block from window
Permit immediate deflateParams changes before any deflate input
Due to the bug fixes, any installations of 1.2.9 or 1.2.10 should be immediately replaced with 1.2.11.

Are you using 1.2.11 now?

@demurgos
demurgos commented Jan 18, 2017 edited

Yes, I upgraded Sunday to 1.2.10 and Monday to 1.2.11 but still had the issue. (And I made my fresh installation today, with 1.2.11).

@andyli andyli referenced this issue in HaxeFoundation/ocamllibs Jan 19, 2017
@ncannasse ncannasse added mingw32 pcre binaries cc37b81
@demurgos

Hi, the package on the Arch repository was updated: this issue is now solved. I think that it was just about recompiling against the latest version of zlib.

In any case, it would be great if you could maybe add more explicit error messages when an error like this occurs. Thank you all for your quick answers and support.

@demurgos demurgos closed this Jan 19, 2017
@demurgos

Well, the situation is more complex in fact.

As I told you above, using the updated package solved my issue. For the example "Hello World" project at least. But I tried at the moment to work on my main project and I got the exact same error: Error: zlib_deflate. I am still not sure if it is an issue with Haxe or zlib so I'll try to find a new "minimal demonstration of the issue" and reopen a ticket: either here or for zlib.

I think that I will try to compile Haxe myself and add more checks when using zlib.

@demurgos
demurgos commented Jan 20, 2017 edited

I tracked the issue up to this line.
It happens when emitting the first ActionScript 3 tag, during the first iteration on str.

ch1 comes from deflate ch on this line.

It really starts to look like a zlib issue.

@demurgos
demurgos commented Jan 21, 2017 edited

Well, I replaced all the calls to the deflated channel to write to a file and then used this file and called deflate manually: I was able to build the swf file.
So, if I prepare all the content and then deflate with a single call, it works. If I let the compiler stream the content, it fails.

I also updated the example repo, I get the issue with:

haxelib run openfl build loader.lime -v flash
@demurgos

Hi,
I asked a friend to try to reproduce the issue (on Linux Mint), and he managed to do it:

The issue is related to the version of zlib but I feel that OCaml / Haxe has also a problem: I am still trying to find a minimal example for the issue but it is a real pain. If I call zlib_deflate directly, it works but if I provide the arguements using the methods called by the compiler, it crashes. I am still trying to figure out if I am missing some hidden state somewhere, but at one point I came to a state where the zlib_deflate error depended on an unused variable that I forgot to comment! Leaving the variable declared (but never touched!) caused the error on the other side of the program, removing it allowed zlib to work: I am still trying to understand how this could even happen.

Here is the smallest code I could find with the issue. To run it, I paste it at the top of src/main.ml to get all the dependencies (IO and Extc).
The string def_data is 65537 bytes long (1 more than the default bufsize of output_zip).

@demurgos demurgos reopened this Jan 22, 2017
@demurgos

Steps to reproduce:

  1. Install zlib 1.2.11 on your system
  2. git clone --recursive https://github.com/demurgos/haxe.git
  3. cd haxe
  4. git checkout zlib-issue
  5. make

At the end, make runs the built executable, this will call zlib_deflate and cause an Error: zlib_deflate.

The branch zlib-issue just adds this function to the main.

@demurgos

I updated the zlib-issue branch to demonstrate the effect of the unused variable: you have to comment def_data2, or zlib_deflate will fail. What's even worse is that building an equal string from Bytes.make instead of a literal works: something is badly broken but for the moment it exceeds my knowledge about OCaml, Haxe and zlib.

https://github.com/demurgos/haxe/blob/zlib-issue/src/main.ml#L55

@andyli
Member
andyli commented Jan 23, 2017 edited

When you recompile Haxe, which commit did you use?
I suspect f6c803d may fix it.

(make sure submodule is updated after checking out)

@andyli
Member
andyli commented Jan 23, 2017 edited

Okay, I reproduced the error with latest Haxe commit (91aac01) on Arch Linux (zlib 1.2.11). No idea what's wrong yet.

@demurgos

Yes, I used the latest version on the development branch (see the network).

@demurgos

Hi, I managed to isolate the problem from the compiler. Here is a new repo with just a few files that causes the issue.

@demurgos

Still on this new repo, I tried to clean it even more: I removed any warning about deprecated usage of strings, unused C functions and added a better error handling for zlib:

Error in `zlib_deflate` (extc_stubs.c), call to `deflate` failed: Z_STREAM_ERROR
@andyli
Member
andyli commented Jan 23, 2017

@demurgos Thanks a lot for reducing it! I start to think this is just a zlib problem, not related to OCaml or our code. Would you try to reproduce the problem with just C and zlib?

@demurgos

I am currently trying to log the calls to C functions to reproduce it only with C.

Do you have an explanation for the impact of def_data2 being commented or not ?

@demurgos
demurgos commented Jan 23, 2017 edited

Ok, I have it. I still don't know what happens but I have a start. zlib is used in multiple steps. You first need to create a stream and then you can pipe to it.

On the C side, initializing a streams creates a struct containing some metadata and a pointer to the z_stream used by zlib. This struct is then returned to OCaml and exposed as the opaque zstream type.

Then, when calling deflate to pipe data, this struct is provided and on the C code retrieves the pointer for the z_stream. The heart of the issue is that between the time when the struct is returned to OCaml and then received back to deflate, the address of the pointer in the struct changes!

I added logs in the C code to print some values. I ran with def_data2 commented (works) and uncommented (fails).

Working example (commented def_data2)

$ ./build-run.sh 
zlib_deflate_init2
zlib_deflate_init2: lvl: 9
zlib_deflate_init2: wbits: 15
zlib_deflate_init2: (z_streamp)(z): 0x7f1c1bbecd80
Calling zlib_deflate...
zlib_deflate
zlib_deflate: (z_streamp)(zv): 0x7f1c1bbecd80
zlib_deflate: spos: 0
zlib_deflate: slen: 65536
zlib_deflate: dpos: 0
zlib_deflate: dlen: 65536
zlib_deflate: flush: 2
zlib_deflate
zlib_deflate: (z_streamp)(zv): 0x7f1c1bbecd80
zlib_deflate: spos: 0
zlib_deflate: slen: 1
zlib_deflate: dpos: 0
zlib_deflate: dlen: 65536
zlib_deflate: flush: 4
zlib_deflate_end
zlib_deflate: (z_streamp)(zv): 0x7f1c1bbecd80
Success

Failing example (uncommented def_data2)

$ ./build-run.sh 
File "main.ml", line 8, characters 8-17:
Warning 26: unused variable def_data2.
zlib_deflate_init2
zlib_deflate_init2: lvl: 9
zlib_deflate_init2: wbits: 15
zlib_deflate_init2: (z_streamp)(z): 0x7f6bca991d80
Calling zlib_deflate...
zlib_deflate
zlib_deflate: (z_streamp)(zv): 0x7f6bca58af28
zlib_deflate: spos: 0
zlib_deflate: slen: 65536
zlib_deflate: dpos: 0
zlib_deflate: dlen: 65536
zlib_deflate: flush: 2
Fatal error: exception Failure("Error in `zlib_deflate` (extc_stubs.c), call to `deflate` failed: Z_STREAM_ERROR")

Notice how in the working example, the z_stream pointer is constant (0x7f1c1bbecd80) across calls, while in the failing case, the pointer changes from 0x7f6bca991d80 to 0x7f6bca58af28.

Currently, my hypothesis is that allocating a relatively large string in OCaml (def_data2) causes some memory reorganization but I do not know how it impacts the pointer.

@ncannasse Could you clarify this allocation ? I do not understand why you subtract one.

value z = alloc((sizeof(z_stream) + sizeof(value) - 1) / sizeof(value), Abstract_tag);

https://github.com/HaxeFoundation/ocamllibs/blob/master/extc/extc_stubs.c#L59

@demurgos

Looks like this a known recent error, it was fixed for the camlzip library.

@demurgos

🎉
I managed to fix it!

This is mainly thanks to the issue in the camlzip library (see link above). If I understand well, the issue was caused by the way Haxe allocated memory, it was subject to reallocation. I'll send in a few days, once I am sure that it works.

@ncannasse
Member
@demurgos demurgos referenced this issue in HaxeFoundation/ocamllibs Jan 24, 2017
Merged

Fix HaxeFoundation/haxe#5941. #23

@demurgos
demurgos commented Jan 24, 2017 edited

I sent a PR for this issue.

I am able to compile my real project with this fix, but I did not know exactly how to run all the tests (there's no CONTRIBUTING.md) so I did run any test suit locally. Please check it, even if I expect that this will pass: there is a lot of text because of the comments and error handling, but in the end I just changed how the z_stream pointer is accessed.

@ncannasse ncannasse pushed a commit to HaxeFoundation/ocamllibs that closed this issue Jan 25, 2017
@demurgos demurgos Fix HaxeFoundation/haxe#5941.
The zlib_issue was caused because the OCaml garbage collector moved
the zlib data allocated on the heap while zlib kept backreference. This
commit uses OCaml's custom memory allocation function to avoid this
issue. This also adds a .editorconfig file for *.c and a few comments
in extc_stubs.c.
0bacc99
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment