Use response files with hsc2hs

[mkscrg]

On Windows, projects break with .hsc files and many dependencies. Here's the error as produced by Cabal-1.22.4.0 (via stack-1.0.0):

--  While building package hsc2hs-windows-broken-0.0.0 using:
      C:\sr\setup-exe-cache\x86_64-windows\setup-Simple-Cabal-1.22.5.0-ghc-7.10.3.exe --builddir=.stack-work\dist\2672c1f3 build exe:hsc2hs-windows-broken --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: C:\vagrant\.stack-work\logs\hsc2hs-windows-broken-0.0.0.log

    Configuring hsc2hs-windows-broken-0.0.0...
    Preprocessing executable 'hsc2hs-windows-broken' for
    hsc2hs-windows-broken-0.0.0...
    setup-Simple-Cabal-1.22.5.0-ghc-7.10.3.exe:
    C:\Users\vagrant\AppData\Local\Programs\stack\x86_64-windows\ghc-7.10.3\bin\hsc2hs.exe:
    does not exist

I've made a Vagrant-ized repro in this Gist, including debug output from Stack/Cabal: https://gist.github.com/mkscrg/18c560b8eb303c6ebcf2

FWIW, this feels similar ghc#10777, which stack ran into w/ commercialhaskell/stack#466 and commercialhaskell/stack#795.

(I first reported this as a Stack issue: commercialhaskell/stack#1718)

[23Skidoo]

We're probably bumping into the same command-line length restriction we did with Haddock. If this is the case, the solution is to add support for response files to hsc2hs and use that in Cabal.

[gbaz]

@bgamari we probably need to open a new ticket on the ghc trac to track this?

[bgamari]

Indeed. See #13896.

[Mistuke]

Sorry to ask a stupid question, but I can't tell why it's failing and why a response file would help? I assume cabal invokes hsc2hs via process's runInteractive, which ultimately calls CreateProcess` which has a command line limit of 32k characters. So I don't quite understand the "why" and "what" here..

[gbaz]

The command line limit on windows is shorter than elsewhere: https://support.microsoft.com/en-us/help/830473/command-prompt-cmd.-exe-command-line-string-limitation

…

On Sun, Jul 2, 2017 at 7:02 PM Tamar Christina ***@***.***> wrote: Sorry to ask a stupid question, but I can't tell why it's failing and why a response file would help? I assume cabal invokes hsc2hs via process's runInteractive, which ultimately callsCreateProcess` which has a command line limit of 32k characters. So I don't quite understand the "why" and "what" here.. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEt2RE9v6CiCm521AA8JYr9RKkF6fuMks5sKCGAgaJpZM4HV0dz> .

[Mistuke]

That is the limit of commands issued through cmd.exe, It is not the limit of commands issued through API calls, as your own link points out. The limit when invoked through API calls is much different https://blogs.msdn.microsoft.com/oldnewthing/20031210-00/?p=41553/ So unless cabal is calling applications by invoking "cmd /K " then the cmd limit is irrelevant. There is the ShellExecute limit, but process only uses CreateProcess:

lpCommandLine [in, out, optional]
The command line to be executed. The maximum length of this string is 32,768 characters,
including the Unicode terminating null character. If lpApplicationName is NULL, the module
name portion of lpCommandLine is limited to MAX_PATH characters.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

So again, I am confused as to why it is failing, what interaction does it have with the shell that would subject it to the user-mode limit instead of the kernel-mode one. process does it right, so again, how does cabal create new processes.

[xplat]

I've run into the 32K command line limit. The largest i've seen so far was ~48K when attempting to build https://github.com/ermine-language/ermine on 64-bit Windows with stack (and a STACK_ROOT of C:\hs). Possibly using system GHC instead of a stack-installed one would get it under, but certainly not by much of a margin. I saw similar-looking problems with cabal new-build, but I didn't go as far as dumping and measuring the command line it was attempting to execute.

[Mistuke]

As far as I know, cabal does separate compilations for hsc modules. I find it highly suspicious that you hit ~48k when preprocessing a single module. but in any case, I was just wanting to make sure we don't make things more complicated for nothing. I'll take you at your word for it :)