windows-arm64 support

[p-b-o]

This series allows to build and run Ruby natively on windows-arm64.

It contains fixes for build system (some of them taken from MSYS2 Ruby package).
Most of the work is detection of __pioinfo pointer, which was the blocking step. Details are available in commit message. This was tested on windows-arm64 11 retail build (not insiders).

You can find a build log here, that uses this script from an MSYS2 clangarm64 shell.

I'm not sure if Ruby community accepts a serie of patches, or if several PR are needed. Let me know.
Note: This is an effort part of my work at Linaro.

[p-b-o]

For now, 2 tests are failing. I can investigate and fix this, once this PR will be merged.

#209 test_fiber.rb:37: 
     Fiber.new(&Object.method(:class_eval)).resume("foo")
  #=> killed by SIGSEGV (signal 11)  [ruby-dev:34128]
#1524 test_thread.rb:336: 
     g = enum_for(:binding)
     loop { g.next }
  #=> killed by SIGSEGV (signal 11)  [ruby-dev:34128]
FAIL 2/0 tests failed

[p-b-o]

@dennisameling @lazka You could be interested by some of those patches for MSYS2 Ruby package. It would enable clangarm64 version of it (I didn't try yet to build additional gems).

[p-b-o]

@nurse You're welcome to review it, as this __pioinfo stuff should be familiar to you ;).

[Biswa96]

Most of the work is detection of __pioinfo pointer, which was the blocking step.

Out of curiosity, would it be possible to remove/replace that __pioinfo mess entirely? The assembly instructions in ucrtbase may change anytime in any Windows version.

[lazka]

@larskanis did something in that area last year: https://bugs.ruby-lang.org/issues/18605#note-6 I don't know what the status there is.

[p-b-o]

I completely agree that the best solution would be to get rid of pioinfo pointer. Python, Perl, and MinGW (at least) got rid of it from what I investigated. As @lazka mentioned, @larskanis started an implementation here. However, it seems it has been untouched for a while.

For now, it's a tentative to have a pragmatic solution, and hopefully, @ArminG-MSFT can make sure ucrtbase _isatty function will keep the same pattern for windows-arm64 too.

[nurse]

The patch looks good. Thank you for your contribution!

[nurse]

As lazka quoted and larskanis said, to remove __pioninfo it needs to reimplement C standard library I/O functions to use our HANDLE/SOCKET mapping table. Why the table is needed is to emulate Unix-like behavior which unifies file handle and socket. In other words to provide IO.select. The strategy larskanis wrote in https://bugs.ruby-lang.org/issues/18605#note-6 is aligned our (I and usa) understanding. The most tough part is implement CRLF handling (text mode).

I completely agree that the best solution would be to get rid of pioinfo pointer. Python, Perl, and MinGW (at least) got rid of it from what I investigated.

As far as I understand Python is under slightly different situation because they don't provide HANDLE/SOCKET unification. (their select doesn't handle files on Windows) https://docs.python.org/3/library/select.html

Perl seems similar situation, but they introduced another hack this year (!). Perl/perl5@8552f09
It looks interesting and it's worth investigating how stable the strategy is.

[lazka]

fyi, the USE_LLVM_WINDRES workaround in this PR should no longer be needed starting with llvm v18, and the relevant llvm changes are backported in MSYS2 as of today -> msys2/MINGW-packages#19524

[jeremyd2019]

I'm messing with some similar code, and noticed what I think is a bug here: adrp's immediate should be sign-extended. I think something like

Suggested change

	/* imm = immhi:immlo:Zeros(12), 64 */
	const uint64_t adrp_imm = ((adrp_immhi << 2) | adrp_immlo) << 12;
	/* imm = SignExtend(immhi:immlo:Zeros(12), 64) */
	int64_t adrp_imm = (adrp_immhi << 2) | adrp_immlo;
	/* sign extend */
	if (adrp_imm & (1 << 20))
	adrp_imm |= ~((1 << 21) - 1);
	adrp_imm <<= 12;

if you really want that to be const I suppose you could do that in a ternary but I don't know if it's more legible that way:

Suggested change

	/* imm = immhi:immlo:Zeros(12), 64 */
	const uint64_t adrp_imm = ((adrp_immhi << 2) | adrp_immlo) << 12;
	/* imm = SignExtend(immhi:immlo:Zeros(12), 64) */
	const int64_t adrp_imm = (((adrp_immhi & (1 << 18)) ? ~((1 << 21) - 1) : 0) | (adrp_immhi << 2) | adrp_immlo) << 12;

[p-b-o]

You're right, I missed that, and luckily, we didn't hit the case where adrp immhi sign matters.
I won't push a fix, but you're welcome to contribute it.

[jeremyd2019]

Maybe... The code I'm messing with didn't hit a negative offset either, but I was thinking it wouldn't be very useful if it wasn't signed, so checked the docs instead of just relying on what this code was doing... I didn't check the specifics of the add instruction either, the code I was looking at didn't use that, instead an ldr with an immediate unsigned offset to add in the low-order bits.

[p-b-o]

By curiosity, I checked the QEMU source as well, and sign is carefully extended for the immediate when decoding instruction.

[jeremyd2019]

I'm having trouble understanding the add id/mask. I think this is trying to match https://developer.arm.com/documentation/ddi0602/2024-09/Index-by-Encoding/Data-Processing----Immediate?lang=en#addsub_imm.

id   0  0  0  1  0  0  0  1  0  0
mask 0  0  1  1  1  1  1  1  1  1
desc sf op S  1  0  0  0  1  0  sh

So you mask off sf (size, 32 or 64 bit), and op (apparently add vs subtract), and match S must be off (not setting flags) and sh must be off (not shifting by 12 bits). You then later implement checking and handling the sh bit left shifting by 12, but never check or implement the op bit turning it into a subtract. Perhaps you meant mask 0x7f800000 instead? Or am I off by one somewhere?

Anyway, sh should not be set, the adrp sets everything but the least-significant 12 bits, so the purpose of the following add in the pair is to set the lower 12 bits, and shifting the add's immediate by 12 bits would result in it not setting those 12 bits either. So a mask of 0x7fc00000 would probably be fine, and then drop the add_sh handling completely.

[p-b-o]

I used doc pages for instructions encoding: add - vs sub.

_mask are the bits that matters to identify encoding (same between sub and add).
_id are the bits expected, i.e. the concrete encoding. (bit 30 allow to differentiate sub vs add).

To identify an add, we need to check bits 23 to 30, which represent the instruction (fixed) encoding.
add_id: 0x11000000 (bits 23 to 30 are 0b00100010, sub_id would be 0b10100010)
add_mask: 0x3fc00000 (0b0111111110000000000000000000000, bits 23 to 30), sub_mask would be the same.

Note: The code included here is not supposed to be a complete disassembler, just a simple way to find a couple of sequences expected (it's supposed that only immediates may be changed in the future). So it makes assumptions (S bit at 0) and is "enough" to work, no more.

[jeremyd2019]

_mask are the bits that matters to identify encoding (same between sub and add). _id are the bits expected, i.e. the concrete encoding. (bit 30 allow to differentiate sub vs add).

To identify an add, we need to check bits 23 to 30, which represent the instruction (fixed) encoding. add_id: 0x11000000 (bits 23 to 30 are 0b00100010, sub_id would be 0b10100010) add_mask: 0x3fc00000 (0b0111111110000000000000000000000, bits 23 to 30), sub_mask would be the same.

sub x1, x2, #0x159 is:
0xd1056441 is 0b11010001000001010110010001000001
your mask is:
0x3fc00000 is 0b00111111110000000000000000000000
anding them results in:
0x11000000 is the same as add_id.

I think you may be off-by-one in your bits due to them numbering from 0 to 31, so counting from MSB you need to start at 31 not 32.

I still recommend mask of 0x7fc00000 and removing add_sh. I could open a PR for that, if you agree with my reasoning.

[p-b-o]

oops, you're right, it seems that the mask is shifted to the right, and missing the op bit which is wrong.
And counting again yesterday, I did the same mistake...

The correct mask I get is 0x7f800000 (not including the sh bit). (0xff << 23, bits 23 to 30). Unluckily, since bit 30 is 0 for add, we missed that because mask exclude it. I would have catch the problem if it was a sub instead.

[p-b-o]

Feel free to make a PR for it (and ping me, I can give you a +1).