Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

windows-arm64 support #8995

Merged
merged 4 commits into from Nov 23, 2023
Merged

windows-arm64 support #8995

merged 4 commits into from Nov 23, 2023

Conversation

pbo-linaro
Copy link
Contributor

@pbo-linaro pbo-linaro commented Nov 22, 2023

This series allows to build and run Ruby natively on windows-arm64.

It contains fixes for build system (some of them taken from MSYS2 Ruby package).
Most of the work is detection of __pioinfo pointer, which was the blocking step. Details are available in commit message. This was tested on windows-arm64 11 retail build (not insiders).

You can find a build log here, that uses this script from an MSYS2 clangarm64 shell.

I'm not sure if Ruby community accepts a serie of patches, or if several PR are needed. Let me know.
Note: This is an effort part of my work at Linaro.

Credits to MSYS2 Ruby package using this patch.
Fix compilation error when using MSYS2 environment.
Credits to MSYS2 Ruby package using this patch.
When adding preprocessor option for llvm-windres (using clang as
parameter), it fails. Thus, do not add this.

It's needed to be able to compile windows-arm64 version, because MSYS2
toolchain is LLVM based (instead of GCC/binutils).
This fixes "unexpected ucrtbase.dll" for native windows-arm64 ruby
binary. It does not solve issue with x64 version emulated on this
platform.

Value of pioinfo pointer can be found in ucrtbase.dll at latest adrp/add
sequence before return of _isatty function. This works for both release
and debug ucrt.

Due to the nature of aarch64 ISA (vs x86 or x64), it's needed to
disassemble instructions to retrieve offset value, which is a bit more
complicated than matching specific string patterns.

Details about adrp/add usage can be found in this blog post:
https://devblogs.microsoft.com/oldnewthing/20220809-00/?p=106955
For instruction decoding, the Arm documentation was used as a reference.
@pbo-linaro pbo-linaro changed the title Windows-arm64 support windows-arm64 support Nov 22, 2023
@pbo-linaro
Copy link
Contributor Author

For now, 2 tests are failing. I can investigate and fix this, once this PR will be merged.

#209 test_fiber.rb:37: 
     Fiber.new(&Object.method(:class_eval)).resume("foo")
  #=> killed by SIGSEGV (signal 11)  [ruby-dev:34128]
#1524 test_thread.rb:336: 
     g = enum_for(:binding)
     loop { g.next }
  #=> killed by SIGSEGV (signal 11)  [ruby-dev:34128]
FAIL 2/0 tests failed

@pbo-linaro
Copy link
Contributor Author

@dennisameling @lazka You could be interested by some of those patches for MSYS2 Ruby package. It would enable clangarm64 version of it (I didn't try yet to build additional gems).

@pbo-linaro
Copy link
Contributor Author

@nurse You're welcome to review it, as this __pioinfo stuff should be familiar to you ;).

@Biswa96
Copy link

Biswa96 commented Nov 23, 2023

Most of the work is detection of __pioinfo pointer, which was the blocking step.

Out of curiosity, would it be possible to remove/replace that __pioinfo mess entirely? The assembly instructions in ucrtbase may change anytime in any Windows version.

@lazka
Copy link

lazka commented Nov 23, 2023

@larskanis did something in that area last year: https://bugs.ruby-lang.org/issues/18605#note-6 I don't know what the status there is.

@pbo-linaro
Copy link
Contributor Author

pbo-linaro commented Nov 23, 2023

I completely agree that the best solution would be to get rid of pioinfo pointer. Python, Perl, and MinGW (at least) got rid of it from what I investigated. As @lazka mentioned, @larskanis started an implementation here. However, it seems it has been untouched for a while.

For now, it's a tentative to have a pragmatic solution, and hopefully, @ArminG-MSFT can make sure ucrtbase _isatty function will keep the same pattern for windows-arm64 too.

@nurse nurse merged commit 784fdec into ruby:master Nov 23, 2023
99 checks passed
@nurse
Copy link
Member

nurse commented Nov 23, 2023

The patch looks good. Thank you for your contribution!

@nurse
Copy link
Member

nurse commented Nov 23, 2023

As lazka quoted and larskanis said, to remove __pioninfo it needs to reimplement C standard library I/O functions to use our HANDLE/SOCKET mapping table. Why the table is needed is to emulate Unix-like behavior which unifies file handle and socket. In other words to provide IO.select. The strategy larskanis wrote in https://bugs.ruby-lang.org/issues/18605#note-6 is aligned our (I and usa) understanding. The most tough part is implement CRLF handling (text mode).

I completely agree that the best solution would be to get rid of pioinfo pointer. Python, Perl, and MinGW (at least) got rid of it from what I investigated.

As far as I understand Python is under slightly different situation because they don't provide HANDLE/SOCKET unification. (their select doesn't handle files on Windows) https://docs.python.org/3/library/select.html

Perl seems similar situation, but they introduced another hack this year (!). Perl/perl5@8552f09
It looks interesting and it's worth investigating how stable the strategy is.

@lazka
Copy link

lazka commented Dec 24, 2023

fyi, the USE_LLVM_WINDRES workaround in this PR should no longer be needed starting with llvm v18, and the relevant llvm changes are backported in MSYS2 as of today -> msys2/MINGW-packages#19524

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants