On making Windows Compilation less convoluted

[frnco]

Currently, the process of compiling Crystal code into a windows binary is quite convoluted. For context, I'm mostly a Linux user, and since so long ago that it may as well be considered forever, windows is something that other people use, which I have to interact with to help friends and family and tolerate when playing a few games. And since Valve pushed game developers to support Linux, it's actually been years since I last had windows installed on a Machine at all. But recently I came across Tauri and started using it, which got me thinking it'd be nice to share some of the stuff I make with friends and family (who use windows). Fast forward a couple months, I wrote quite a few small utilities using Tauri (Audio Transcriber, eBook reader, Manga reader, small side-projects that replace other software which I wasn't too happy with but had no better alternative to replace it, basically), but I'm not the biggest fan of Rust, and Tauri has an interesting feature called "Sidecar", which allows you to include binaries with your packages, and that gave me the idea of using Crystal to deal with some of the more hardcore stuff, since I don't like Rust all that much and front-end javascript is just not enough in some situations.

Upon trying to compile Crystal binaries for windows, though, I realized it wouldn't be as easy as I had hoped. After reading and re-reading https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows I started trying stuff on Wine, which didn't go too well because microsoft (Their build tools, more specifically). Then I created a VM to do it, which did solve the build tools part, but then I came across a jumbled mess of libraries and having to compile from the same folder where the libraries are located and it's just unreasonably, painfully messy.

There's the possibility of writing code just to deal with those steps but doing that on a per-project basis seems counterproductive. Ideally crystal should provide a easy, reasonable way to deal with all those messy implementation details, providing a straightforward, objective sequence of steps for compiling a project in windows. Something like "Install Crystal in Windows" -> "Run crystal.exe --compile-shared-libs" -> "Run crystal.exe compile whatever.obj" or something like that.

I started to write a few scripts to deal with that, but as soon as I began in occurred to me that this deserves being discussed by more people and possibly being part of the official crystal release for windows, so I decided to open this issue.

[konovod]

but then I came across a jumbled mess of libraries and having to compile from the same folder where the libraries are located and it's just unreasonably, painfully messy.

I just have this
LIB=f:\projects\crystal\crystal\libs\;c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib\amd64\;
in my environment, and then place all lib files into crystal\libs folder. I think that's not ideal (lib files literally doesn't have any information that isn't present in corresponding dll), but that's requirement of linker.

[straight-shoota]

I think the experience for using Crystal on windows should eventually be similar to that on other platforms. You download an archive, extract it and get a crystal.exe which just works. Obviously you need MSVC installed for that. Maybe we can avoid the developer command prompt using vswhere (see #5430 (comment)). All dependencies except system libraries should be provided in the archive.

Maybe we can avoid the developer command prompt using vswhere (see #5430 (comment)). An alternative could be https://gist.github.com/Kalinovcic/b4d9cc55a37f929cb62320763e8fbb47

As a later step, we might want to add a Windows installer as well. But a plain archive should be good enough for a start. You can either download it manually or it can be used by package managers like scoop.

[HertzDevil]

The Zig locator mentioned in that gist, which supports multiple architectures: https://github.com/ziglang/zig/blob/master/src/windows_sdk.cpp

Note that relying on the developer command prompt has its own advantages because the locators only extract the minimum dependencies needed to invoke the MSVC build tools properly, but sometimes the extra MSVC customizations and PATH entries are desirable on their own, e.g. Vcpkg integration or accessing MSVC's own installation of CMake.

I just have this LIB=f:\projects\crystal\crystal\libs\;c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib\amd64\; in my environment, and then place all lib files into crystal\libs folder. I think that's not ideal (lib files literally doesn't have any information that isn't present in corresponding dll), but that's requirement of linker.

After #11424 there should be no need to override the CRYSTAL_PATH and LIB environment variables; if we set CRYSTAL_CONFIG_PATH and CRYSTAL_CONFIG_LIBRARY_PATH to the same values we use on our portable packages at cross-compilation time, everything should just work apart from passing an extra /LIBPATH for the first linking.

Crystal on Windows does not use DLLs; even if it does, load-time dynamic linking still needs the LIB files and is different from run-time dynamic linking (which is the job of something like Crystal::Loader).

[oprypin]

This issue seems to start with the point of talking about the experience of a "normal" user trying to compile some program for Windows.
It is unfortunate that you followed https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows for that purpose, because the guide isn't for that purpose.
For a normal user right now, the process is "install msbuild; download prebuilt crystal.exe+libs; download crystal source; set some environment variables; crystal.exe build whatever.cr".

So, what is this issue trying to discuss? I'm just saying that I'm finding it hard to reuse this issue for any particular purpose, because I don't understand it.

[straight-shoota]

I understand the content of the OP as providing some background information on @frnco's story. The topic as I see it is how to provide a distribution package for Windows. So it basically continues the discussion starting from #5430 (comment)

[frnco]

I just have this LIB=f:\projects\crystal\crystal\libs\;c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib\amd64\; in my environment, and then place all lib files into crystal\libs folder. I think that's not ideal (lib files literally doesn't have any information that isn't present in corresponding dll), but that's requirement of linker.

Yeah I'm precisely trying to figure out what'd be the best generalized solution to small things like that.

So, what is this issue trying to discuss? I'm just saying that I'm finding it hard to reuse this issue for any particular purpose, because I don't understand it.

Sorry I wasn't sufficiently clear. Crystal does not officially support Windows, so normal user trying to compile something for windows isn't something that should happen. However, as Crystal gets closer and closer to fully functional in Windows, it seemed like a good idea to start reviewing the current process. It makes sense to work from crystal repo's root folder if the only thing we're trying to do in windows is test the crystal compiler, but windows support reached a point where, in my humble opinion, it's time to start checking how other stuff will behave in windows. In this scenario the crystal compiler is merely a tool, not the protagonist. The current process revolves around the root folder of the crystal repo, and it seems like a good time to start the discussion on what we want to change and how we might go about it. Sure, ultimately we'll just have crystal.exe build something.cr do all the hard work, but before we get to that point we need to decide what that hard work should actually be, and this thread was intended as a place to discuss that, as opposed to me writing some stuff that would solve my problem but would not contribute anything towards supporting windows.

Using PATH/ENV is not a bad solution, but there should be some sort of script or something to deal with setting up Crystal and its libraries, preferably also checking whether MS build tools are installed and if possible even starting the process of installing build tools in case they're not available in the system. To make something like that it's necessary to discuss how to deal with different versions of Crystal, whether the script-thinggie will deal with updating Crystal if it detects an older version of crystal already installed, that kind of thing, and that's only after deciding what'd be the suggested installation path, how the libraries would be made available to the user etc.

[HertzDevil]

That much is solved by CRYSTAL_CONFIG_PATH and CRYSTAL_CONFIG_LIBRARY_PATH which ensure a portable installation works, the only requirements being that Crystal is added to PATH and that currently the MSVC developer command prompt must be active.

What isn't solved right now is the self-hosting part of Crystal; the Makefile works on neither CMD nor Powershell (even with MSVC nmake), and the same goes for the bin/crystal wrapper script (this is the one used for compiler development, not one we distribute to releases). I ported the latter to a Powershell script, but it remains to be seen whether the Makefile should be similarly ported.

[oprypin]

A fully convenient solution (via Scoop) has been offered in #11489. It already fully works and further improvements to it are being discussed there. I intend to promote it as the main option, at least for the time being.

[frnco]

A fully convenient solution (via Scoop) has been offered in #11489. It already fully works and further improvements to it are being discussed there. I intend to promote it as the main option, at least for the time being.

I do have some reservations when considering Windows package managers in general, but Scoop does seem quite reasonable, at least for stuff that requires the .NET framework. Still, having something like a standard MSI installer, which is the de-facto standard for windows software, seems like something that should be done at some point in time.

What isn't solved right now is the self-hosting part of Crystal; the Makefile works on neither CMD nor Powershell (even with MSVC nmake), and the same goes for the bin/crystal wrapper script (this is the one used for compiler development, not one we distribute to releases). I ported the latter to a Powershell script, but it remains to be seen whether the Makefile should be similarly ported.

The first thing that comes to mind is: Considering sooner or later Crystal will probably be packaged and distributed using an MSI or similar installer, how should things be organized in that installer? I'm not a windows person, at least in the sense that, while I did use Windows a lot when I was younger and for gaming and when fixing computers of pretty much everyone I know (Programmers' perks, yay! -_-), and I actually do have some experience managing active directory and stuff, but I don't write windows software, and I avoid it like the plague whenever doing any actual programming, so I have no idea how I'd make something "install" in windows, and I'm not knowledgeable enough about what should go on a Makefile (or anywhere else for that matter) when packaging windows stuff. Thus, I can't help with stuff that is too windows-specific. One thing I do understand, however, is that installing stuff can be painful, and Makefiles will most likely be nowhere near enough to deal with all the windows-specific stuff. Thus, I'd start by considering how much windows-specific-stuff should be done by a Makefile, and what should be refactored to some other location. Setting the path is something I think will probably be better left to some other thing, script, installer or whatever, instead of the Makefile.

Honestly, I see nothing wrong with having a completely separate process for building in Windows, I'm more concerned about making it work in a way that is reasonable for windows and its users, meaning it has, for instance, to properly clear up when being uninstalled (Which is one of the things I hate about Windows, by the way).

This is something that deserves some serious consideration, I believe.

[rishavs]

I do not think a msi packaging is required. Crystal is a dev tool and the people using it will be familiar with command line usage.
Supporting Scoop, Winget and Chocolatey would be sufficient. Maybe Windows Store in the future (now that it has started supporting non-gui apps).

Tools like Crystal need to be updated every now and then and this will anyway need integration with a windows package manager.

[straight-shoota]

I'm sure we should provide a Windows archive which you can extract and just use, without installation. It should be easy to put that into an installer. But when the installer literally just downloads (or already includes) and extracts a zip archive, yeah, it might not be that useful. It's definitely an option though, and not hard to do based on having an archive anyways.

@frnco Makefile and bin/crystal wrapper are tools for compiler and stdlib developers. They are not made available to regular users who just want to build Crystal software. So this has nothing to do with an installer setup.

[frnco]

I do not think a msi packaging is required. Crystal is a dev tool and the people using it will be familiar with command line usage. Supporting Scoop, Winget and Chocolatey would be sufficient. Maybe Windows Store in the future (now that it has started supporting non-gui apps).

Absolutely agree with the command line bit. As someone used to version managers for languages I'm not sure I agree with the point about package managers though. I used msi as an example, but I should have phrased it as "a self-contained package that can be easily installed by anyone", my point here is that stuff like asdf, rvm, nvm etc ends up becoming a necessity for people working on a contract-basis, since we can't always afford to update everything from every client. Ruby is pretty okay, but NodeJS is a painfully painful pain in the rear to deal with just because anything you do with it take tons of disk space, Crystal will, hopefully, at some point be in the same boat, with devs needing more than one version of Crystal. Unless Scoop or whatever can properly deal with that, one or ten versioning-things for Crystal will pop-up, and the people working on that probably deserve to have some self-packaged-thing they can work with. Of course, it's not absolutely necessary, but then again the same can be said for Windows Support, or even for the language itself, but we're here because we want Crystal to succeed, and every time I decide to put Crystal aside and use Haskell or Rust or whatever pains me. I'd rather do everything in Crystal, but to do that I need to help push Crystal forward so that it doesn't hinder me. Currently, Crystal packages are so-so (A bit better than Haskell I guess, still behind Rust, nowhere near something like Ruby though), but compiling to Windows is, in my opinion, a game-changer. If this is done well, it can be an amazing thing, but it's also easy to screw this up.

Windows worries me mostly because more compilation targets mean more people can use Crystal, which in turn means more people writing Shards. In my opinion Crystal is already a lot better than Rust, but Rust has crates in greater number and often more polished than crystal shards. My goal here is to give some contribution to make it as reasonable as possible to use Crystal in Windows, which will hopefully let me use Crystal to write stuff that can be used in Windows, in turn allowing me to spend more time writing Crystal code and publishing such code as shards. And hopefully enough people are in the same boat that we can make this work.

[straight-shoota]

I think this issue can be considered resolved.

We're building a self-contained and portable distribution package in CI now (since #11494). It is accessible as https://nightly.link/crystal-lang/crystal/workflows/win/master/crystal.zip: just download and extract the archive and there you have a working Crystal compiler. We can publish it as a build artifact on the next release.

scoop is an alternative method for installing Crystal on windows (#11489): https://github.com/neatorobito/scoop-crystal

For the current status of Windows support, these distribution methods should be sufficient. We can consider further improvements later.