Windows: Setup MSYS2

[snoyberg]

Goals:

• MSYS2 is easily available for builds with both cabal-install and stack
• The entire MSYS2 set of commands are not automatically added to the PATH (Note: I don't have this goal, and in my own Windows usage like having these tools on my PATH. I'm just recording a requirement I think I've heard from @mzero)
• Ideal: possible to install new packages via pacman, and make those libraries available during builds

MinGHC originally solved the MSYS problem using MSYS1, then moved to PortableGit. I finally found the portable installation of MSYS2 itself a few weeks ago, and stack master has moved over to that. There is no official release of stack with that change, but many users (myself included) have been testing it. I'd recommend going for that in the next HP, since it's the most official and most featureful option.

Since code is worth a thousand words, here's the code from stack that installs MSYS2:

https://github.com/commercialhaskell/stack/blob/master/src/Stack/Setup.hs#L849

The one catch is that, after unpacking the archive, you need to run the shell once to do some initialization, otherwise some utilities - at least pacman - will fail.

Download information on MSYS2 is available at https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml#L14.

Once MSYS2 is available on the OS, you need to be able to tell the build tools where to find it. Since the goal is to avoid cluttering PATH, I'd take two different routes:

• stack can add support for recognizing some environment variable (e.g. STACK_MSYS2=dir or perhaps HASKELL_MSYS2=dir) to detect the location of the MSYS2 install, and from then on stack just does its usual thing of modifying the PATH for child processes
• You can provide one of Neil's "switcher" batch scripts for cabal-install usage, which modifies the PATH to include all of these tools. See: https://github.com/fpco/minghc/blob/master/Installer.hs#L68

[gbaz]

I agree it is important to not have these tools on the path -- keeping them off the path makes it possible for me to continue to run cygwin for example, without which my windows life is very painful :-).

I have been discussing other alternatives as well with @randen as well, and was planning to create a thread once I had more clarity on the issues. This one, however will do just fine :-)

Note that as per #179 it appears that the next release already was planning to include these sans-switching-story, but it was still against portable-git rather than msys2. I'm not sure which version is now being used, what the tradeoffs are, and if the toolset should be swapped out.

I think the stack-like approach of modifying the path for child processes is the right one and should be integrated into cabal. If others agree, I can start working on a patch for that. In the interim, should that not land in time, switcher scripts are certainly better than nothing!

[gbaz]

(Note that this approach of using an environment with a locally-modified-path might feel sort of hacky, but as @danbornside pointed out to me, it is precisely what microsoft itself does with the devenv tool!)

[randen]

Regarding where msys2 comes from, either the version in portable-git or a separate portable install of msys2 alone, I am all for being consistent with what stack is using, seeing no reason that one flavor of msys2 is better than the other. It would also be nice if there were a Haskell-build-environment specific place where any of these tools would put and look for msys2. I think msys2 should be fairly stable and not be likely to have a version that only stack uses and another that is only compatible with, say, the Haskell Platform. So, assuming that one flavor of msys2 is sufficient for all our uses, it would be great to agree upon where to put it, what environment variable (and or registry entry) to use to record its location, etc.

[gbaz]

I'm not keen on recognizing the same env variable as any other tool at this point, nor trying to share directories. This would be ideal, of course, but it raises too many issues in terms of co-ordination -- what if that tool decides it needs a slightly different version, etc. While it could work out quite nicely, it is just one more thing to go wrong. As far as future co-ordination effort, once we have the precedent of a few tools doing this, then perhaps unification can be on the cards.

So it seems to me we have a twofold approach:

A) bundle the platform with a switcher script

B) also work on a patch to cabal so that it can recognize an env variable (I would in fact suggest something more tool-agnostic in terms of msys etc such as CABAL_EXTRA_PATH) and auto-prepend that to the path prior to running configure scripts.

Whenever that patch lands, great, we can stop the switcher script nonsense. Until then, support remains available.

(And a third component could be to try to detect if an appropriate msys is in one of the "usual suspect" locations and offer the option to the user to use that instead of installing a new one? That way there is no magic, but it also lets the user try to leverage existing tools when they exist.)

Thoughts?

[randen]

On the current Windows version of HP (7.10.2-a), a version of MSys2 is included. Assuming that stack will at some point (the next version?) be included in the HP, then the Windows version of HP should no longer include/install a copy of MSys2. Do I have that right? When the time comes, I just need to know what to do.

[gbaz]

@randen I don't think that makes sense? The fact that stack is included doesn't mean we want to only enable stack workflows. I.e. how does stack's inclusion help the following scenario: 1) install platform, 2) cabal sandbox init, 3) cabal install network.

I'd much rather just keep including MSys2, and also bundle the "switcher scripts", and also include the stack tool, as had been proposed in the "Improving the Get Haskell Experience" approach.

[randen]

I will note that including MSys2 in HP for Windows adds to the download size of HP and to the installed disk footprint. Perhaps "minimal" does at all refer to either disk space or download size, but strictly to the packages included? Sorry if I haven't followed through all the various discussions that may have taken place to clarify this. Clarifying to me, at least, what "minimal" means, would help me make sure I have the correct model to direct my work. The 'Improving the "Get Haskell Experience"' message seems to suggest that size might be part of this ("HP installer gets much smaller, and about as minimal as a working installation can get"), and I might just not be understanding what the latest understanding is on all this. Thank you for any further clarification.

[gbaz]

Yeah, "minimal" has been used to refer only to number of included packages. If we consider it from the standpoint of "how do I get network" we have two answers A) either it is bundled or B) it must be buildable. So this is a tradeoff of bundling prebuilt packages against bundling parts of MSys2, with the payoff being that users can now install "network" and related things on their own as well.

I know the terminology is a bit silly here, and "a smaller installer" will perhaps then be more of a payoff for os x and linux only while windows its a wash at best, but I think this is inline with the proposal in terms of the specific "you will be able to build network" promise :-)

[randen]

More clarification for me, if possible. In the "Improving" memo, Michael and Mark wrote "installer will no longer include pre-built, pre-installed packages other than GHC's set" so this means network will not be included, so as soon as a user needs the network package, they will need MSys2 to build it. Above that is the line "Use the stack model for package installation" which might mean, very concretely, use "stack.exe" to do package installation, or it might mean something more abstract as the "stack model", I am not sure. Assume it means the former, then stack will pull down MSys2 as soon as needed (so no need to include MSys2 in the HP installer--or if included, it should be registered so that stack.exe can find and use it). I am not very clear on what the latter would mean actually (perhaps it means modify cabal-install to have an option for the "stack model"?). But, again, when I need to do the work on this, I am not really sure what it is I need to do (or not do). [edit: just to note that our last two comments were both in-flight at the same time]

[gbaz]

I don't know if my last comment already answers this but, the "stack model" is only taken to mean the latter -- looking back at that message I can definitely see the confusion.

The way I understand this is that the "stack model" means that packages that would have been included as built before now are still "pinned" to those versions that would have been included. But now those packages need not actually be installed until needed, and that this also can be used to facilitate sharing.

You can see later that "including getting msys into the Windows installer" is mentioned as related work here. So we are supporting both cabal and stack workflows.

That means, as I understand it, that from the windows-specific standpoint, the 7.10.2-a story is "almost there," and the plan for a next release would be A) include the switcher scripts B) ensure we're using whatever the "right" version of Msys2 is, and C) just as in the other platform releases, just don't build the additional libraries.

Then the future work is patching cabal to accept a global shared location for msys2 from an env variable or the like, so as to render the need for explicit switchers unnecessary and facilitate future sharing of msys2 installs.

Sound reasonable?

[randen]

Thanks for the clarifications. I know there may be reasons against it, but the MSys2 which is installed by HP should be findable/usable by stack. Would that be only by user action (e.g., script, command line option to stack), or would stack just "know where to look" (e.g., registry entry, env vars, "well known location" such as ghc/../../msys)?

[gbaz]

I like the idea of adding HASKELL_MSYS2=... to the env, as @snoyberg proposed above. Ideally if we agree now on that, I guess the stack we ship can already be one which is aware of this.

I would like to teach cabal to be aware of this and do the right thing too, but since I don't bet on patches landing there fast, I wouldn't want to hinge anything on cabal having learned about this prior to the platform shipping, and so making sure we provide the switcher scripts makes sure we're in good shape no matter what.

[gbaz]

Actually, what about CABAL_PATH_PREFIX= so this isn't OS dependent? (even though of course it'll probably only be needed / get set on windows, certainly at first). That would make me more comfortable submitting a patch to Cabal to respect it...

[snoyberg]

shrug I don't care too much what color the bikeshed is. However, stack does more than just set the PATH based on MSYS, it also sets include and lib dirs. See:

https://github.com/commercialhaskell/stack/blob/35aaaee88b0f7086d3dafd6c0b746e6445b1368e/src/Stack/Setup.hs#L585

[gbaz]

ok, that's important to know. those can be set with the extra-include-dirs and extra-lib-dirs in cabal/config directly though...

which leads to a question... I guess we could add a bin-dir to cabal/config as well, right? And then for uniformity set the msys bin directory there too. That seems more in keeping with how cabal works typically.

And then of course, for the convenience of other tools that might want it, we could then have the platform add HASKELL_MSYS2=... to the env anyway, so that this information could be shared with stack or whoever.

[gbaz]

Another thing pointed out to me -- we can use extra-prog-path in cabal config to add extra info to its search path.

I think this is insufficient for some reason, but it would be good to figure out why and if we can't just improve that instead.

@randen -- thoughts?

[gbaz]

Ok, I think I have this approach working, in this PR for cabal

haskell/cabal#2840

[gbaz]

Here are the steps I've figured out to allow network, etc. (and generally build-type:configure) packages to work with the windows platform.

1. Add to the cabal config file the following lines:

extra-prog-path: C:\Program Files\Haskell Platform\7.10.3\msys\usr\bin
extra-lib-dirs: C:\Program Files\Haskell Platform\7.10.3\mingw\lib
extra-include-dirs: C:\Program Files\Haskell Platform\7.10.3\mingw\include

1. In the program-location subsection of the cabal config file add

   gcc-location: C:\PROGRA1\HASKEL1\7.10.3\mingw\bin\gcc.exe

At a minimum we can just have instructions to add these lines.

However, it would be nice if the platform installer on windows added these lines by default, I think. (I know I was on the fence about this before).

That said, in order to add these lines by default, we first need a cabal config file, and one may not yet exist (we would need to run cabal to initialize one on a fresh system, I think).

At this point, if we can augment the windows installer to do this, I think that would be great. But I'd like to defer to Randy to see if he has a good idea on how to add this, since the windows build system isn't my wheelhouse...

[gbaz]

This is resolved in 8.0.1 -- I've tested the addition of only the top 3 lines in fact suffices to allow network to build on a fresh install. And I've added instructions to the hp website. I'd like to make this more streamlined and require less from end users, but I feel that can close out this ticket.