Increase stack size of ocamlopt.opt for windows

[vicuna]

Original bug ID: 5465
Reporter: shadinger
Assigned to: @alainfrisch
Status: closed (set by @xavierleroy on 2013-08-31T10:48:54Z)
Resolution: fixed
Priority: normal
Severity: feature
Platform: Mingw64 and Cygwin
OS: Windows7
OS Version: 64bits
Version: 3.12.1
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @protz @Chris00

Bug description

While compiling opa (www.opalang.org), ocamlopt.opt crashes due to a stack overflow. Unfortunately stack size of windows applications is decided at link time.

Therefore I had to change Makefile.nt (line 21) to:

CAMLOPT=boot/ocamlrun ./ocamlopt -nostdlib -I stdlib -I otherlibs/dynlink -cclib '-link -Wl,--stack,16777216'

was

CAMLOPT=boot/ocamlrun ./ocamlopt -nostdlib -I stdlib -I otherlibs/dynlink

The effect is to force a stacksize of 16MB for every native tool of ocaml. This may not be a problem but I didn't see any way to expand the stack only when compiling ocamlopt.opt.exe

Note: the arg chain is as follow:

1. ocamlopt is called with "-cclib '-link -Wl,--stack,16777216'"
2. then flexlink is called with "-link -Wl,--stack,16777216"
3. then gcc is called with "-Wl,--stack,16777216"
4. finally the linker is called with "--stack 16777216"

[vicuna]

Comment author: @lefessan

Have you tried to use "editbin /STACK:size" on ocamlopt.opt to try to fix the problem ?

I don't know in which Microsoft package "editbin" is available, so it might not be generic solution.

[vicuna]

Comment author: shadinger

I guess this should work but it would imply to install Visual Studio which is an overkill. I want to stick to Mingw64. I don't know any mingw64 equivalent to EDITBIN.

[vicuna]

Comment author: @xavierleroy

We should be able to pass this option for all executables produced by ocamlopt. Need to check what we do already for the MSVC ports and transfer this to the MinGW ports.

[vicuna]

Comment author: @lefessan

The way it is done for MSVC:

MKEXE=$(FLEXLINK) -exe -link /STACK:16777216

For MSVC64, the stack size is set to 33554432 instead of 16777216.
The MKEXE line should be modified accordingly in config/Makefile.mingw and config/Makefile.mingw64. For Cygwin, it must be done in the configure script, probably by using Unix-like options, by modifying the "gcc*,--cygwin*)" case
when mkexe is defined.

Right now, I have no Windows system to boot on, so a patch tested by a windows-user would be welcome !

[vicuna]

Comment author: @alainfrisch

I've implemented a new option -stack in flexlink. It allows to set the stack reserve in the resulting image, in a way independent of the toolchain (it patches the image directly).

In config/Makefile.msvc, we could thus replace:

MKEXE=$(FLEXLINK) -exe -link /STACK:16777216

with

MKEXE=$(FLEXLINK) -exe -stack 16777216

And do the same for the other ports (and in the configure script for Cygwin).

(Or should I use this value as a default in flexlink?)

[vicuna]

Comment author: @lefessan

Could you use two different values for 32 bits and 64 bits ?

[vicuna]

Comment author: @alainfrisch

Could you use two different values for 32 bits and 64 bits ?

If I define a default value in flexlink, yes of course. Otherwise, it is just a matter of deciding which number to put in the config/Makefile.*.

[vicuna]

Comment author: shadinger

Alain, great work.

Is there a way to use flexlink to change a posteriori the stack size of an existing EXE like EDITBIN does? Your ocaml code seems to be able to do it, but I don't know if it is possible from the flexlink CLI.

[vicuna]

Comment author: @alainfrisch

but I don't know if it is possible from the flexlink CLI.

Currently not, but this is a trivial addition. Will do it.

[vicuna]

Comment author: @xavierleroy

I see a description of a complete fix in the comments to this PR, but I don't see any change in config/Makefile.mingw*. Could someone who cares about the Windows/MinGW do what needs to be done, then resolve this PR? Thanks.

[vicuna]

Comment author: @alainfrisch

Xavier: I'll do it. I need to release a new version of flexlink, with the new -stack option, first.

[vicuna]

Comment author: @alainfrisch

I've released flexdll 0.28, which adds a -stack option (to set the stack reserve on the target image). One can also do:

flexlink -patch -stack 123456 foo.exe

to set the stack size on an existing image.

I'll update OCaml config Makefiles accordingly.

[vicuna]

Comment author: @alainfrisch

The trunk has been updated (commits 12263, 12264).

shadinger: Can you try it? If you confirm everything is ok, I'll push the change to the 4.00 branch.

[vicuna]

Comment author: shadinger

Great. I will try beginning of next week.

[vicuna]

Comment author: @alainfrisch

Pushed to version/4.00 (commit 12277).

[vicuna]

Comment author: shadinger

Alain, I have a flexlink fatal error while compiling under Mingw64 very early in the process: compilation of 'byterun' directory ends with:

x86_64-w64-mingw32-gcc -DOCAML_STDLIB_DIR='"C:/ocamlmgw64/lib"' -I"c:\flexdll" -
O -mms-bitfields -Wall -Wno-unused -c prims.c
flexlink -chain mingw64 -exe -o ocamlrun.exe prims.o -lws2_32 libcamlrun.a
** Fatal error: hd
make[1]: *** [ocamlrun.exe] Error 2
make[1]: Leaving directory `/home/Papa/ocaml-win64/byterun'
make: *** [coldstart] Error 2

I tried recompiling flexlink with my own compiler chain, it produces the same error.

I don't have a clue about this 'hd' Fatal error.

[vicuna]

Comment author: @alainfrisch

If you recompile flexlink with -g, you should be able to get a stacktrace (with OCAMLRUNPARAM=b=1).

I expect the exception to be raised by Reloc.get_output1. Can you instrument it and see which commands returns an empty result? (maybe "cygpath -S" ?)

[vicuna]

Comment author: shadinger

I changed Reloc.get_output1 to the following:

let get_output1 ?use_bash cmd =
Printf.eprintf "** Before %s\n" cmd;
let res = List.hd (get_output ?use_bash cmd) in
Printf.eprintf "** After\n";
res

I get the following result:
** Before x86_64-w64-mingw32-gcc -print-sysroot
** Fatal error: hd

Meanwhile, "x86_64-w64-mingw32-gcc -print-sysroot" or "gcc -print-sysroot" produces an empty result.

[vicuna]

Comment author: shadinger

Additional info:

The last fully working version for me is producing the following:
** Before x86_64-w64-mingw32-gcc -print-libgcc-file-name
** After

And "x86_64-w64-mingw32-gcc -print-libgcc-file-name" get the following output:
c:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.1/libgcc.a

[vicuna]

Comment author: @alainfrisch

Which version of mingw64 do you use? Has it been installed as a Cygwin package?

[vicuna]

Comment author: shadinger

I am using MinGW-w64 tdm64-gcc-4.6.1 with MSYS. It is not a cygwin package. MSYS does a better job than cygwin to solve the directory mess between windows and cygwin when compiling native 64 bits apps (not using the cygwin DLLs).

All the steps are described here:
https://github.com/shadinger/opalang/wiki/Compiling-ocaml-MinGW64-w64

[vicuna]

Comment author: @alainfrisch

I'm not familiar with msys. In particular, I don't know what I should use as a replacement for gcc -print-sysroot on this system.

Note that the official toolchain supported by OCaml is mingw64 packaged by Cygwin, not the msys one. If you can contribute a patch to fix flexdll for this system (without breaking the officially supported one), I'll happily integrate it, but unfortunately, I don't have time to work on it myself.