Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack smash(?) on Win64 causes C "float" function to return random results #5008

vicuna opened this Issue Mar 24, 2010 · 1 comment


None yet
2 participants
Copy link

vicuna commented Mar 24, 2010

Original bug ID: 5008
Reporter: Antoine Mine
Assigned to: @xavierleroy
Status: closed (set by @xavierleroy on 2011-05-29T10:19:59Z)
Resolution: fixed
Severity: crash
Version: 3.11.2
Fixed in version: 3.12.0+dev
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @alainfrisch

Bug description


  1. Symptoms

On ocaml-3.11.2 (also verified on ocaml-3.11.1) compiled with Visual Studio 9.0 (i.e., 2008) and Windows SDK 6.0A on Windows 7 in 64-bit mode.

We define a C function with "float" calling convention:

double ml_single_precision_add_float(double a, double b)
return (float) (a + b);

and call:

external add : t->t->t = "ml_single_precision_add" "ml_single_precision_add_float" "float"

let x = add 1. 2. in ...

within a large OCaml program whose source I cannot disclose, unfortunately.

Then, non-deterministically (approx. 2 out of 10 runs), x will get a value which is not 3. (checked with Printf.printf), although the C function gets the right arguments and computes the right result (checked with printf).
I was not able to construct a small program exhibiting the same behavior, unfortunately.

  1. Tentative explanation

I am not sure I understand fully what happens, but here is a guess.
After add returns successfully its result in XMM0, caml_call_gc is called.
It saves XMM0 in [rsp] and calls caml_garbage_collection, a C function without any argument.
On Win64, I was led to believe that a function must reserve 32 bytes on its stack, even a void function, as the space might be used by the callee as a scratchpad.
I guess some C code in caml_garbage_collection may smash the saved XMM0 in [rsp].

  1. Tentative fix

In amd64nt.asm, caml_call_gc, change:

    call    caml_garbage_collection


sub     rsp, 32
    call    caml_garbage_collection
add     rsp, 32

On my program, this seems to fix the issue.
Again, I do not claim to fully understand this.

I've also heard that the stack should be kept aligned on 16-bytes.
Here, rsp is decreased by an odd number of qwords.
I don't know whether this might cause a problem.

Best regards,



This comment has been minimized.

Copy link
Collaborator Author

vicuna commented Mar 28, 2010

Comment author: @xavierleroy

Hello Antoine,

Thanks for the detective work. You are correct that the bottom 32 bytes of stack are scratch space in the Win64 calling conventions. ocamlopt-generated code as well as other functions in amd64nt.asm carefully reserve this space, but caml_call_gc did not. I've applied your fix; it will go in release 3.12.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.