Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coordinate porting to Windows #5430

Open
22 tasks done
straight-shoota opened this issue Dec 21, 2017 · 204 comments
Open
22 tasks done

Coordinate porting to Windows #5430

straight-shoota opened this issue Dec 21, 2017 · 204 comments
Assignees
Labels
community:in-progress help wanted This issue is generally accepted and needs someone to pick it up kind:feature platform:windows
Projects

Comments

@straight-shoota
Copy link
Member

straight-shoota commented Dec 21, 2017

UPDATE 2023-04-24:

All the main platform features for Windows are finished! 🚀

There are still some smaller stories pending. Project board: https://github.com/orgs/crystal-lang/projects/11/views/5

You can support the ongoing development financially: Windows suppport project on Open Collective.

UPDATE 2021-11-18:

By now, the compiler works pretty well on windows and most of the standard library has been ported as well. Besides a couple other smaller missing pieces, the major gap is in concurrency-related features held back by the missing event loop implementation.


With #5339 being merged, the first step of bringing Crystal to Windows has been accomplished!
With current master branch it is possible to cross-compile and run simple programs (alá Hello, World!) on native Windows.

Obviously we're still far away from porting the compiler or entire standard library to windows. This issue is supposed to coordinate and keep track of ongoing efforts to add support for more and more of the crystal standard library.

TODO

The primary focus should be on the core library (somewhat odered by priority):

And of course a CI infrastructure once specs are running on windows.

Proceedings

The general course of action is to extract platform-specific implementations and then add the implementation for win32 platform.

A lot of work has already been done in #3582 but it needs to be updated and chopped into smaller pieces of changes.

The first goal should probably be to get specs running on windows to be able to verify the windows implementations. Essential for this are as far as I can see file, dir, exceptions (backtrace!), process and signal.

If you want to help with that effort, or want to "take" a piece of that work to work on, please comment below.

Useful References

Please feel free to add and edit this post as appropriate (core team).

@RX14
Copy link
Contributor

RX14 commented Dec 21, 2017

@straight-shoota that's a good point. We need to figure out the basics of getting a blank spec suite working on windows (we should do this first without unwind support, just the raise==exit we have now). Then we need to create some kind of win32_spec.cr which requires only the parts of spec/std that work. I'll take on this task (tomorrow).

@RX14
Copy link
Contributor

RX14 commented Dec 21, 2017

Another area of priority is looking at c168035 and working out ways of reducing that stubbing. For example: get iconv working so we can remove the stubbing in src/io/encoding.cr. The modified parts of the corelib are actually I think worth focus much more than the missing parts (for now).

@RX14 RX14 added community:in-progress help wanted This issue is generally accepted and needs someone to pick it up kind:feature platform:windows labels Dec 21, 2017
@straight-shoota
Copy link
Member Author

A very basic working spec shouldn't be too hard I think. I've tried it earlier to figure out what might be needed for that. It seems to be missing mostly some basic file and dir methods. After disabling all feature options I even got a simple spec to compile, allthough it resulted a LLVM error at codegen.

@s5bug
Copy link

s5bug commented Dec 22, 2017

I'll take Fibers, I'll be able to start work on them after January 2nd.

@RX14
Copy link
Contributor

RX14 commented Dec 22, 2017

@straight-shoota looks like my spec PR has turned into a Dir PR first.

@RX14
Copy link
Contributor

RX14 commented Dec 22, 2017

Also of importance: deciding what to do about windows's UTF-16 use. There should definitely be a really quick and easy way of going between String -> UTF-16 WCHAR* and the reverse. We should also have a policy of no A functions, only W functions even this early in the port.

This is an important issue: we shouldn't rely on iconv for this (I presume it's slow for small strings, and we don't have it in windows anyway). We need to write UTF-16 infrastructure for all platforms first.

@RX14
Copy link
Contributor

RX14 commented Dec 24, 2017

Good news: spec compiles and works on my fork.

Bad news: it only works sometimes.

From this spec failure, looks like we have a codegen or ABI bug somewhere:

@RX14
Copy link
Contributor

RX14 commented Dec 24, 2017

Here's my branch porting spec: https://github.com/RX14/crystal/commits/feature/windows-spec

I'll turn "Remove c/winbase" and "Add Crystal::System::Dir" into PRs.

@RX14
Copy link
Contributor

RX14 commented Dec 24, 2017

Another idea before I forget it:

Consolidate all stubbing out into one huge windows_stubs.cr file so that we can keep track of them all.

@straight-shoota
Copy link
Member Author

You mean to remove winapi.cr, not winbase.
And this commit should also rename DWord.

@RX14
Copy link
Contributor

RX14 commented Dec 24, 2017

@straight-shoota yep, thanks, commit message is wrong. Shouldn't be working at 1am...

@RX14
Copy link
Contributor

RX14 commented Dec 24, 2017

Made those PRs: #5447 and #5448. I hope you don't mind me editing the checklist in your issue @straight-shoota.

@straight-shoota
Copy link
Member Author

Looks great!

I've deliberatly added a note at the end of the OP for that purpose ;)

@felipetesc
Copy link

Great news! Thank you for this.

@drhuffman12
Copy link

In https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows, should:

Ongoing efforts will be tracked and coordinated in #

... be:

Ongoing efforts will be tracked and coordinated in #5430?

@straight-shoota
Copy link
Member Author

@drhuffman12 thanks, fixed!

@drhuffman12
Copy link

@straight-shoota , yw

@s5bug
Copy link

s5bug commented Jan 2, 2018

Due to time constraints and last minute events I won't be able to work on Fibers as expected :(. If someone else wants to claim them, that would be amazing.

@RX14
Copy link
Contributor

RX14 commented Jan 4, 2018

If you can get openssh server working on WSL (hint: you need to edit /etc/ssh/sshd_conf to set UsePrivilegeSeparation no and change Port 2222), you can use the following script to make the windows testing process seamless:

#!/bin/sh
set -euo pipefail

windows=192.168.xxx.xxx

bin/crystal build --progress --cross-compile --target x86_64-unknown-windows-msvc -o .build/cross "${@}"
scp -P 2222 .build/cross.o $windows:/mnt/c/crystal/cross.o

cat <<EOF | ssh -p 2222 $windows bash
export PATH="$PATH:/mnt/c/Windows/System32"
cmd.exe /S /c "C:\Program^ Files^ ^(x86^)\Microsoft^ Visual^ Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x64 && cd \crystal && cl cross.o /Fecross pcre.lib gc.lib libcmt.lib"

cd /mnt/c/crystal

echo
./cross.exe
EOF

This script assumes you have gc.lib and pcre.lib in C:\crystal and that you're using the visual studio 2017 build tools. MSVC 2015 can surely just change the path to vcvarsall.bat.

@RX14
Copy link
Contributor

RX14 commented Jan 4, 2018

A script to automatically generate a compilable manifest of the entire spec suite, with broken tests commented out with a vague reason:

cd spec
for spec in ./std/**/*_spec.cr; do
  (cd .. && crystal-windows spec/$spec) >/dev/null 2>/dev/null && echo "require \"$spec\"" ||
    (../bin/crystal build --no-codegen --target x86_64--windows-msvc $spec >/dev/null 2>/dev/null && echo "# require \"$spec\" (failed to run)" || echo "# require \"$spec\" (failed to compile)")
done

crystal-windows is the script from above.

@straight-shoota
Copy link
Member Author

The script posted by @RX14 is intended to be run on an external system which connects to WSL through SSH to upload the object file and run the linker through Windows interoperability feature.
If you want to run the compiler locally on WSL, you don't need to go through SSH.
I have put together a crystal-windows wrapper for this: https://gist.github.com/straight-shoota/e4393c9e3f0febb7dc290470f92f5c0d

@txe
Copy link
Contributor

txe commented Jan 12, 2018

As I see, there is no dir.cr in crystal/system/win32, so I'm going to make it.
Is there any idea what to do with UTF-16?

@txe
Copy link
Contributor

txe commented Jan 12, 2018

About UTF-16: I suppose we need just a couple of functions to encode\decode utf-16 without any len, find and others functions and can keep utf-16 string as array of wchar.
FYI, Go has this code for that and Nim this

@asterite
Copy link
Member

@txe Please coordinate with @RX14 , as far as I know he was porting File to Windows (not sure about Dir). For UTF16 we now have this: #5541 . But check the PR's comments, we might need to change it a bit to include a zero at the end when encoding to Slice(UInt16).

@RX14
Copy link
Contributor

RX14 commented Jan 12, 2018

@txe Fantastic to see you back!

I'm working on File and Dir right now, they're both mostly complete but the new code isn't in my branch (uncomitted on my PC). UTF-16 also has an added tweak in my branch.

I have a lot of work not in master, perhaps committing my branch would help you have more of a base to work off of?

Namely, working in my branch but not master: File, Dir, Spec. I also have a spec/windows_spec.cr which lists all working specs.

@straight-shoota
Copy link
Member Author

Some personal comments:

  • I don't think non-blocking file IO has a high priority. This seems to be rather uncommon on Windows, actually. That's similar on most POSIX systems btw., where it's probably easier to do, but still often not used by default (File in Crystal is blocking unless you configure it to be non-blocking).
  • Non-blocking pipes are certainly important. I think this shouldn't be hard to implement, using the proper Win32 API for that. This just wasn't necessary (or even possible to test) before we got the event loop working.
  • Unix domain sockets in Windows are a rather recent feature overall. Adding support for that in Crystal is nice, but IMO not high priority.
  • Multithreading support shouldn't be far out I suppose?
  • IMO the playground has not much priority. If we can get it to work on Windows, that's great. But the interpreter is likely more important.

So overall, I think there's not much standing in the way of proclaiming Windows as a supported platform.

My must-have list for that is rather short (maybe I'm missing something, though).

  • Signals (or equivalent)
  • non-blocking pipe
  • Confirm concurrent IO operations in practice. The event loop port is very fresh and we should test it with some real work scenarios and stress tests to iron out the edge cases.

@HertzDevil
Copy link
Contributor

HertzDevil commented Jul 9, 2022

Regular file I/O could stay blocking, but pipes are IO::FileDescriptors and I don't think there is any non-blocking API specific to pipes that wouldn't generally work on other files. These examples suggest ReadFile / WriteFile with overlapped I/O is enough.

@konovod
Copy link
Contributor

konovod commented Jul 9, 2022

  • I don't think non-blocking file IO has a high priority. This seems to be rather uncommon on Windows, actually.

Just my 2 cents, but it's useful to, say, work with (virtual) COM Port.
But if general concurrency will work, i think i can go with WinAPI or own dll to emulate non-blocking api.

@jgaskins
Copy link
Contributor

jgaskins commented Jul 9, 2022

pipes are IO::FileDescriptors

I was actually surprised IO.pipe was implemented in terms of file descriptors. I assumed it was basically an IO::Memory that used separate read/write cursors until I ran out of file descriptors a while back with NATS object store.

Is there any reason not to implement IO.pipe in user space so it would work on all platforms?

@asterite
Copy link
Member

asterite commented Jul 9, 2022

Right, pipes in Go are implemented in pure Go, no file descriptors at all. We wanted to do that eventually with Juan but we never got to it.

@jgaskins
Copy link
Contributor

jgaskins commented Jul 9, 2022

Yeah, when I realized, I assumed you used POSIX FDs simply because that was quicker at the time and didn’t intend that to be part of the semantics of the pipe.

Is there an open issue for this? I have some terrible code written for it that mostly works.

@HertzDevil
Copy link
Contributor

The most relevant discussion seems to be in #8152.

But if we are passing pipes across processes, possibly ones that aren't written in Crystal, wouldn't that require an OS-backed pipe anyway?

@straight-shoota
Copy link
Member Author

Let's continue the discussion about re-implementing pipes in Crystal in #12238

@Blacksmoke16
Copy link
Member

Blacksmoke16 commented Sep 2, 2022

@89z I think he means from a stdlib usability point of view. I.e. you can actually use Crystal on windows fairly stably, even if it requires Visual Studio. Having something working and usable is more important to the end user compared to which backend is used when the latter is something that can be added in after the former is working.

See #5430 (comment).

@asterite
Copy link
Member

asterite commented Sep 2, 2022

Does it require Visual Studio? I though we were binding to standard Windows functions

@mjblack
Copy link

mjblack commented Sep 23, 2022

look like the command is set here:

cmd = "#{cl} #{args}"

where cl is defined here:

and CL is hardcoded here:

CC = ENV["CC"]? || "cc"
CL = "cl.exe"

if Crystal could change CL to try an environment variable first, simlar to CC, then it should be possible to set that environment variable to clang-cl, in order to use LLVM instead of Visual Studio.

A little further down in the file it is ignoring what CL is set to for msvc due to hard coding the executable as a method argument:

cl = Process.quote_windows(msvc_path.join("bin", "Host#{host_bits}", target_bits, "cl.exe").to_s)

It probably should be updated to:

cl = Process.quote_windows(msvc_path.join("bin", "Host#{host_bits}", target_bits, cl).to_s)

@mdwagner
Copy link
Contributor

I just ran into the issue where Windows can't find cl.exe. I was using crystal-1.7.1-windows-x86_64-msvc-unsupported.zip, but will try using Scoop instead.

@neatorobito
Copy link
Contributor

The scoop package has the same issue for the moment, still trying to coordinate a fix.

@mdwagner
Copy link
Contributor

@neatorobito Ah, you're right. Same issue.

@neatorobito
Copy link
Contributor

@mdwagner I've tested and pushed a fix for this in the scoop repo, please give it a try when you get a chance.

@mdwagner
Copy link
Contributor

mdwagner commented Feb 1, 2023

@neatorobito I updated to the new version in Scoop, but still have the same issue.
image

@straight-shoota
Copy link
Member Author

@mdwagner Can you please open a dedicated issue for this? It's out of scope for this meta issue. Please add details on your system setup (like install location of VS Build Tools).

@mominshaikhdevs
Copy link

Isn't it high time this + #26 + #7339 be closed ? (since as per this comment, #7339 is completed as #12224 + #12284 being completed) and move focus to other issues?

@straight-shoota
Copy link
Member Author

straight-shoota commented Feb 17, 2023

No, it's not.
I'm not sure if the signal story is entirely finished. We just recently merged a bunch of changes. If that's all, yes we can close #7339. But we'll need to confirm that.

That would then mean the bulk of the Windows port is completed as per the TODO list in this issue. But there are still a lot of very rough edges, so #26 is still a way to go.

@zw963
Copy link
Contributor

zw963 commented May 7, 2023

Hi, from the TODO list, it's seem like Signal should works on windows, but it build failed on my github action platform is ms-windows 2022.

Dependencies are satisfied
Building: translater
[1/13] Parse                             
[1/13] Parse                             
[2/13] Semantic (top level)              
[2/13] Semantic (top level)              
[3/13] Semantic (new)                    
[3/13] Semantic (new)                    
[4/13] Semantic (type declarations)      
[4/13] Semantic (type declarations)      
[5/13] Semantic (abstract def check)     
[5/13] Semantic (abstract def check)     
[6/13] Semantic (restrictions augmenter) 
[6/13] Semantic (restrictions augmenter) 
[7/13] Semantic (ivars initializers)     
[7/13] Semantic (ivars initializers)     
[8/13] Semantic (cvars initializers)     
[8/13] Semantic (cvars initializers)     
[9/13] Semantic (main)                   
Error target translater failed to compile:
Showing last frame. Use --error-trace for full trace.

In lib\selenium\src\selenium\service.cr:92:20
 92 | process.signal(Signal::KILL)

                     ^-----------
Error: undefined constant Signal::KILL
Did you mean 'Signal::ILL'?

Please have a look, following is log.

https://github.com/crystal-china/translater/actions/runs/4906504148/jobs/8761004602

@HertzDevil
Copy link
Contributor

They should use Process#terminate, because Signal is not portable, and MSVC never defines Signal::KILL. This is by design

@Neltherion
Copy link

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

@mominshaikhdevs
Copy link

mominshaikhdevs commented Jun 16, 2024

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

All I have seen here for the last 4 years is, New "Windows Related" Issues are piling up everyday. Instead of resolving the old ones. 😂

@cloewen8
Copy link

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

Not surprised, checked the funding and it just was not there. I've moved on to other languuages with better marketing. The ecosystem is moving on, without Crystal.

@Blacksmoke16
Copy link
Member

Blacksmoke16 commented Jun 16, 2024

I mean have any of you actually tried it? Windows support is, and has been for a while, at the point where you can pretty much install via the provided installer on the GH release and use it as you would on any other OS.

There are still some loose ends that need to be figured out, but I don't think it's fair to say "there is no windows support."

@mjblack
Copy link

mjblack commented Jun 16, 2024

9 Years passed and still no Windows support. This is why Crystal will unfortunately always be a niche language for a small minority.

You can compile on windows, you can create windows gui applications, you can create directx applications. What do you think is still missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community:in-progress help wanted This issue is generally accepted and needs someone to pick it up kind:feature platform:windows
Projects
Windows
  
Todo
Status: Metaissue
Development

No branches or pull requests