Error "The command line is too long" for very short command line (<250 characters) on Cygwin/Windows

[MSoegtropIMC]

Dear Dune Team,

In the Windows build for INRIA Coq I use dune to build e.g. lablgtk. Since a short while, possibly since switching to OCaml 4.08 (from 4.07) I get strange errors about long command lines. e.g.:

lablgtk-3.0.beta5-dune-build_err.txt: (cd _build/default && C:\bin\cygwin_coq64_ltac2_array\usr\x86_64-w64-mingw32\sys-root\mingw\bin\ocamlopt.opt.exe -w -40 -w -6-7-9-10-27-32-33-34-35-36-50-52 -no-strict-sequence -g -shared -linkall -I src -o src/lablgtk3.cmxs src/lablgtk3.cmxa)
lablgtk-3.0.beta5-dune-build_err.txt: The command line is too long.

The build does suceed if I reduce the path by one character from C:\bin\cygwin_coq64_ltac2_array to C:\bin\cygwin_coq64_ltac2_arra. This is reproducible.

I am quite sure that I did use much longer path names recently. The only thing I know has been changed is the switch to OCaml 4.08, but of cause Coq is a large project with many contributors. If this helps I can provide one click batch files which setup cygwin and build for the good and for the bad case on Windows. In case you think it could indeed be the switch to OCaml 4.08 I could also retest with OCaml 4.07 (it is a bit of work cause of the substantially different build procedures for OCaml 4.07 and 4.08 on Windows).

Btw.: I tested this with dune 2.0.0 and dune 2.1.0.

[dra27]

Commands which are invoked via cmd can easily blow the 8K limit (the underlying OS has a 32K limit, which IIRC is what you get with Unix.open_process, but that comes with its own complexity). The fact it’s only one character in the directory means that it’s always been close to the limit - Dune need only have added a single warning number or a new flag recently and the problem would emerge (ie I think the fact it used to work is not relevant to the problem)

A workaround for compiler invocations is to use -args (the bootstrap already does this) but that won’t help commands manually executed. I expect that the commands here also feature a large number of modules?

[MSoegtropIMC]

Well but the command has less than 250 characters. It is shown in the error message.

Btw.: why are you using CMD in dune? In case you are using it for redirection and quoting, I might have a superior alternative for Windows (a posix compliant shell interpreter written in C but it only supports what one typically puts into system commands, that is IO redirection, quoting and env assignments, but in this it should not be distingusishable from a proper posix shell, except for the 32K limit you mentioned).

[dra27]

Oops, somehow reading it on my phone I missed that the full command was in that readout!

Are you able to run that command manually (and see the error) and, if so, is the log clearer if you add -verbose?

[MSoegtropIMC]

Indeed the error seems to come from ocamlopt when it calls the linker. If I run from cygwin (the OCaml tools in my setup are MinGW):

$ cmd /C "C:\bin\cygwin_coq64_ltac2_array\usr\x86_64-w64-mingw32\sys-root\mingw\bin\ocamlopt.opt.exe -w -40 -w -6-7-9-10-27-32-33-34-35-36-50-52 -no-strict-sequence -g -shared -linkall -I src -o src/lablgtk3.cmxs src/lablgtk3.cmxa"
The command line is too long.
** Fatal error: Error during linking

File "caml_startup", line 1:
Error: Error during linking

So I would say this is not a dune but an OCaml issue. I try to find out what the command line is.

[dra27]

If you add -verbose to the ocamlopt paeans, it should show you

[MSoegtropIMC]

What really goes wild is the call to gcc by ocamlopt via flexlink. It has 8193 charaters (beyond the CMD limit) and the cygwin path appears 131 times in it, so each character in the path adds 131 characters to the command line.

Do you know whom I could contact in the OCaml world to discuss a possible replacement of CMD in OCaml tools on Windows?

Wasn't it discussed once that the OCaml tools use via files instead of command lines?

[dra27]

Can I see the command line? We're here, so for now let's talking on ocaml/dune instead of ocaml/ocaml :)

The compiler does not intentionally call cmd, but the only thing available to it is Sys.command which of course does - I'm extremely open to removing the Windows shell from simple command execution, but I think we will find a simpler solution to this problem than that Everest...