Release a Crystal preview on Windows with Scoop

[neatorobito]

What

This issue serves to capture, continue, and hopefully conclude the discussion in #5430 and elsewhere around packaging and distribution for Windows by publishing an official scoop bucket. I've tried to address multiple concerns and add actionable steps.

Why

This discussion has been had across many threads, but to summarize:

Releasing the Crystal bits that work on Windows right now would unblock users across a number of use cases as @rishavs has pointed out in many threads. It would also bring some positive attention, focus, manpower, and potentially new users to the language. Releasing an MVP would start a process to help us finish that last 20% of Windows support. It's a similar decision as going to the 1.x releases.

I'm not sure if is there some consternation around this, but I want to call out that an MVP does not mean low quality. The work that the Crystal team and contributors have done to get Windows support this far is awesome! Again the majority (80%) of the language works on Windows. There should be no hard feelings about releasing it in it's current state. I've already been hacking away with Windows internals using it and I'm more than satisfied with it's current state. And we think you're going to love it too.

How

I've created a scoop repo here with documentation that builds on @oprypin's work and allow users to run Crystal on Windows. Right now I manually include an artifact drop. In order to make this solution more robust we need to:

1. Publish the artifacts publicly

   • This would allow us to include the artifact binaries in a package manager such as scoop or winget.
     • Updating from a package manager would simply mean checking out the last successfully built artifact and source.
   • We'll need to warn users on those pages that only package manager based installations of Crystal are supported on Windows. Downloading the drops won't do anything.

2. Ensure consistent messaging that this is a Preview release.

   • On all social media channels, never forget to add that word and explain that this is a nightly release. We want to set expectations correctly.
   • Use the checklist in Coordinate porting to Windows #5430 to make a simple table graphic that can be copypasta'd everywhere showing what works/doesn't work with the Windows preview release.

3. Make sure the GitHub, Gitter, Twitter, Reddit, etc. are ready for a potential influx of folks.

   • Do we need Windows preview specific issue labels on GitHub?

4. Publish or point to the scoop repo in an official capacity.

5. Tell everyone about it.

[neatorobito]

Starting with the first action item. I see there's an stale issue tracking distribution scripts on Windows. I know that's more related to CI infrastructure, though. Something way simpler could work here.

Is there a GitHub Action we can install or install nightly.link to unblock publishing artifacts? I would again throw in a vote for adding nightly.link.

• Artifacts expire after 90 days, but this shouldn't affect someone who has already downloaded a particular release with scoop. The binary would still be on their machine.
• Nightly.link would let us curtail access to the drops by default rather using another method and having users confused when they find artifact drops for Windows in the releases page.

[oprypin]

Thanks for continuing the discussion here. I've been writing a response but didn't have a place to put it 😅

I have finally tried out this Scoop package you created. It does indeed make things easy. It seems to be what we need now.
https://github.com/neatorobito/scoop-crystal

Some steps remain before I could fully recommend it:

• It should have no references remaining to /raw/main/ such as here https://github.com/neatorobito/scoop-crystal/blob/711438572a44b96e104fb195c248b1ff84d17587/bucket/crystal.json#L6 (it is possible to refer to full commit hashes even though it's within the same repo)
• It shouldn't have self-provided binaries of Crystal.
  • Right now, https://nightly.link/ can be used. There's nothing that needed to be done to "enable" it.
    https://nightly.link/crystal-lang/crystal/actions/runs/1496112588/crystal points to artifact https://nightly.link/crystal-lang/crystal/actions/artifacts/117879911.zip (actual permalink, so can be checked with a hash) and also points to commit b10165e. Sure, this will expire after a few months, but I think these can be refreshed sometimes. Automatically, even.
  • For the next release, there will likely be an official download of Crystal available. Even if it's not fully usable on its own. There is initial agreement on this in the core team.
• Eventually the wrapper script should be dropped - I think that putting its exact functionality into crystal.exe (as @HertzDevil suggested) is the right thing to work on now.
• License should be changed to permissive. This is not actually a strong requirement, but I'd like that. For now I'm abstaining from direct contributions. Maybe eventually it will make sense to put this repo under crystal-lang org on GitHub.
• As a mostly unrelated suggestion, the way you accomplish the source code download can actually be changed to a zip download from GitHub, rather than git clone. E.g. https://github.com/crystal-lang/crystal/archive/b10165ef37605ecb384b114e1062f558fd388a35.zip

[faustinoaq]

Hi, I just tried instructions on comment #5430 (comment) to install crystal and it works like a charm for some examples 👍

[neatorobito]

@faustinoaq Thanks for testing it out! Glad to see it's working for you. I have been actively developing in the bucket though so there's probably a breaking change. You may want to uninstall, run scoop update, scoop cache rm all, and then reinstall.

@oprypin Thanks for the testing and all the feedback. I've pushed some changes to address your comments. I also realized that packaging nightlies is trickier than I though. My first instinct was to version them like 1.2.2@commit_hash. But scoop gets angry if you use anything other than numbers. Here's what I came up with:

• In the vanilla crystal package everything is downloaded using Nightly.link or GitHub links with no self provided binaries and tied to a specific commit. Upon the next official release I'll manually bump the version and update the links/hashes. I can figure out an official automated solution for this in the coming weeks using GitHub actions. The only script here is the shim.

• If you absolutely need the latest nightly, install crystal-nightly which includes a grab_latest_nightly script that downloads the Crystal source using git, and also grabs the latest matching nightly using Nightly.link.

  • Ideally this goes away and we use a GitHub action to update the vanilla package with the correct links on a push event.

• Glad to see there is some agreement within the core team about a release.

• Agreed the shim should go away.

• I switched the license to the permissive Apache 2.0 License, same as Crystal.

Please let me know if you have any more feedback. I will check back in this weekend or early next week after the holiday.

Edit: Also let me know if you encounter any issues as I was actively working on the scripts/manifests so you may need the steps up above if you installed before my latest commit.

[oprypin]

@neatorobito Awesome, thanks much!

I do fully recommend your repo now 😁

Ideally this goes away and we use a GitHub action to update the vanilla package with the correct links on a push event.

Yes...
I have a very clear idea of all aspects of how to do that, so I'd like to contribute that part 😊

[oprypin]

As I was expecting, I did find a bad side effect of relying on a wrapper script.

git clone https://github.com/crystal-lang/shards
cd shards
git checkout v0.16.0
make

That works with a raw crystal.exe in PATH, but not when installed as here via Scoop.

The system cannot find the path specified.
process_begin: CreateProcess(NULL, cat VERSION, ...) failed.
Makefile:17: pipe: Bad file descriptor
process_begin: CreateProcess(NULL, cat VERSION, ...) failed.
docs.mk:8: pipe: No error
File not found - *.cr
The system cannot find the path specified.
A subdirectory or file bin already exists.
Error occurred while processing: bin.
make: *** [Makefile:37: bin/shards] Error 1

[oprypin]

In any case, it would be nice to provide Shards via this solution. Right now perhaps it'd need to be compiled from source.
And if someone is using this Scoop solution and wants to build Shards, the required Bash commands instead of make would be the following (I obtained them by running the makefile for real):

git checkout v0.16.0
mkdir -p lib/molinillo
shards install || (curl -L "crystal-lang/crystal-molinillo@v0.2.0.tar.gz" | tar -xzf - -C lib/molinillo --strip-components=1)
SHARDS_CONFIG_BUILD_COMMIT="ff94dd2" SOURCE_DATE_EPOCH="1633554420" crystal build  src/shards.cr -o bin/shards

[neatorobito]

Hm, once #11496 is merged we could:

1. Get rid of the shim.
2. Have the scoop package depend on make.
3. Grab the shards package, and then compile from source as apart of the post install script.

That would get the ball rolling while we work out a CI setup that publishes these binaries later.

And then the only other part of this issue I'd like to tackle is some kind of graphic or table I can post in the scoop repo showing what works/doesn't work. I could make a prettier/friendlier version of the table in the #5430, but I wanted to see if anyone felt strongly about that action item.

[oprypin]

Yes, the 1 and 3 steps were exactly what I had on my mind :) 👍

[oprypin]

@neatorobito A good summary of what missing is needed but it shouldn't be specific to the Scoop repo. At this point we're also targeting extract-and-run (without a package manager) so it would ideally be shared.

[straight-shoota]

I would actually prefer to build shards here in CI and include it in the build artifact. We ship shards in all other distribution packages already. So there wouldn't be anything specific to be done in scoop.

If we can't set that up quickly, it's still possible to rig it up in scoop.

[HertzDevil]

The Crystal compiler is a debug build, and we don't distribute its PDB file yet. Should this change?

[neatorobito]

I would actually prefer to build shards here in CI and include it in the build artifact. We ship shards in all other distribution packages already. So there wouldn't be anything specific to be done in scoop.

If we can't set that up quickly, it's still possible to rig it up in scoop.

@straight-shoota I'll work on adding shards to the artifact drop.

@HertzDevil Yes, I think it'd be helpful to include the PDB.

[straight-shoota]

Is there anything actionable left here?

[neatorobito]

Nope, all of my concerns have been addressed. I'm excited to see an accessible version of Crystal on Windows!

[straight-shoota]

That's great.

Just bear in mind that Windows support is still incomplete and only experimental 😉