Initial support for ARM64 Windows #8142
Conversation
|
|
CT Test Results 10 files 248 suites 3h 34m 33s ⏱️ Results for commit 551751f. ♻️ This comment has been updated with latest results. To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass. See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally. Artifacts
// Erlang/OTP Github Action Bot |
cocoa-xu
force-pushed
the
cx/windows-arm64
branch
4 times, most recently
from
1b08767
to
73bc87e
Compare
|
Hi, I have also managed to craft a GitHub workflow that can build it natively on an ARM64 Windows machine, and here is the workflow log, the corresponding CI yaml file and the final installer! This build was based on 6c6b409. There're still some issues when cross-compiling on x86_64 machine, which I'll look into later. Other than that, all libraries except wx and odbc build without any issue, ARM64 JIT also seem to work. Although I haven't got time to do any thorough tests yet. And one more thing is to adjust nsis scripts where relevant, such as defaulting to |
| elif [ X"$CONFIG_SUBTYPE" = X"arm64" ]; then | ||
| GCC="aarch64-w64-mingw32-gcc -m64" | ||
| elif [ X"$CONFIG_SUBTYPE" = X"x64_arm64" ]; then | ||
| GCC="aarch64-w64-mingw32-clang -m64" |
There was a problem hiding this comment.
I'm having a hard time following what x64_arm64 is supposed to mean and how it differs from plain arm64, and an even harder time seeing why one uses gcc and the other clang. Can you elaborate a bit?
There was a problem hiding this comment.
Hi @jhogberg, x64_arm64 is used when cross-compiling to arm64 windows on an x86_64 machine because we need to call vcvarsall.bat in erts/etc/win32/wsl_tools/SetupWSLcross.bat with different parameters to get right msvc toolchain, for example,
- call
vcvarsall.bat arm64if compile natively - call
vcvarsall.bat amd64_arm64if cross-compile on x86_64 machines
And I think that it's somewhat awkward naming on Windows because win32 and win64 traditionally only stands for 32-bit and 64-bit respectively, but they actually means x86 and x86_64. I wonder if we should standardise these naming while providing "aliases" so that build script won't break. I'm think that adding a universal entry point for setting up environment variables:
./otp_build env_win [<BUILD_ARCH>-]<TARGET_ARCH>
where BUILD_ARCH and TARGET_ARCH each can be one of x86, x86_64 and aarch64.
For example,
x86 target
For compiling natively on x86 windows, env_win32 should be the alias of env_win x86 (or vice-versa). The following commands should behave the same
./otp_build env_win x86, universal entry point./otp_build env_win32, alias for backward compatibility
For cross-compiling, we will have
./otp_build env_win x86_64-x86, universal entry point./otp_build env_win aarch64-x86, universal entry point
x86_64 target
Similarly, the following commands should behave the same
./otp_build env_win x86_64, universal entry point./otp_build env_win32 x64, alias for backward compatibility./otp_build env_win32 x64, alias for backward compatibility./otp_build env_win64 x64, alias for backward compatibility
For cross-compiling, we will have
./otp_build env_win x86-x86_64, universal entry point./otp_build env_win aarch64-x86_64, universal entry point
aarch64 target
For native compiling,
./otp_build env_win aarch64, universal entry point
Maybe we can have these aliases which would follow the convention:
./otp_build env_win32 x86-aarch64./otp_build env_win32 x64-aarch64./otp_build env_win64 x86-aarch64./otp_build env_win64 x64-aarch64
And for cross-compiling, we will have
./otp_build env_win x86-aarch64, universal entry point./otp_build env_win x86_64-aarch64, universal entry point
WDYT?
There was a problem hiding this comment.
The reason why using clang for cross-compiling is that aarch64-w64-mingw32-gcc seems to crash the windows machine😅 I'll double verify this on another x86_64 windows machine.
There was a problem hiding this comment.
What GCC version is aarch64-w64-mingw32-gcc? There's a known bug in GCC 7 (maybe 8 or 9 too?) where it segfaults while compiling the JIT. If so, I don't think we want this workaround.
As for the other stuff, I think @dgud is more equipped to answer than me :)
There was a problem hiding this comment.
What GCC version is
aarch64-w64-mingw32-gcc? There's a known bug in GCC 7 (maybe 8 or 9 too?) where it segfaults while compiling the JIT. If so, I don't think we want this workaround.
It's actually LLVM with a wrapper script, https://github.com/mstorsjo/llvm-mingw/releases. I tested aarch64-w64-mingw32-gcc on a physical x86_64 machine, and it seems to be okay now (as I'm not able to do a full build -- there're still some issues when cross-compiling).
I think the reason was that I was testing on a virtual machine, and perhaps there were some bugs in the virtualization layer/software.
cocoa-xu
force-pushed
the
cx/windows-arm64
branch
7 times, most recently
from
b590118
to
becdd6d
Compare
cocoa-xu
force-pushed
the
cx/windows-arm64
branch
from
becdd6d
to
08bebe0
Compare
cocoa-xu
force-pushed
the
cx/windows-arm64
branch
from
08bebe0
to
0dc9a4c
Compare
updated installation guide build with native-ethr-impls enable updated installation guide
arm64 jit now works on windows
fix cross-compile on x86_64 for aarch64
Oh I didn't know that before. Thanks for the information! Gonna update the nsis script file! |
Co-authored-by: Dan Gudmundsson <dangud@gmail.com>
|
And here we have the test results. I'm not sure which of these files should I archive and upload here (and also because the test result directory is about ~1.5 GB, and 259MB after compressing), so I only did a screenshot for now. Please let me know if you need me to upload the test results. tests results below is outdated
|
|
It seems that the |
Hi, I decided to upload the test result here, https://github.com/cocoa-xu/otp-build/releases/download/v27.0-rc1/test_server-otp_win_arm64_27.0-rc1.tar.xz, and using xz reduces its size to 75MB. It will be easier to check all relevant logs, as well as for other people to look and maybe give any feedbacks. It corresponds to this installer, https://github.com/cocoa-xu/otp-build/releases/download/v27.0-rc1/otp_win_arm64_27.0-rc1.exe. Also please let me know if any tests need a re-run, and I'll update the test results. :) |
As for the ones that cause crashes, they're
|
new tests results are available in the reply below outdated test resultsUpdates for the `emulator` tests. I looked and removed the lines that cause hard-crashes, and now the test results ishard-crashescommonIn
{system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(id(MaxSmall))/unit:32,0:64>>),
{system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(id(MaxSmall))/unit:32,(id(0)):64>>),
|
|
Updates for all tests. I did a clean compile and removed the lines that cause hard-crashes in For the {system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(id(MaxSmall))/unit:32,0:64>>),
{system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(id(MaxSmall))/unit:32,(id(0)):64>>),In these test cases:
In addition to the lines above, I also have to remove these two lines in {system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(MaxSmall)/unit:32,0:64>>),
{system_limit, {1,integer,size,MaxSmall}, _} = ?ERROR_INFO(<<0:(MaxSmall)/unit:32,(id(0)):64>>),For the disk_log_full(Config) ->
ok.
% ...
disk_log_events(_Config) ->
ok.After the above changes, I ran the tests in WSL (which was done in Powershell before), and the latest test results are:
Also, sorry that I'm not sure how I should properly remove/truncate successful tests to save space, so I still have to upload the full test results as a GitHub release file here, as the attachment file cannot exceed 25MBs in PR comments. |
|
And I was wondering if and which ones of these failed test cases should we address in this PR? Perhaps the ones in the |
| @@ -6,6 +6,7 @@ | |||
| "installation_guide/INSTALL.md": [source: "../../HOWTO/INSTALL.md"], | |||
| "installation_guide/INSTALL-CROSS.md": [source: "../../HOWTO/INSTALL-CROSS.md"], | |||
| "installation_guide/INSTALL-WIN32.md": [source: "../../HOWTO/INSTALL-WIN32.md"], | |||
| "installation_guide/INSTALL-WIN32-ARM64.md": [source: "../../HOWTO/INSTALL-WIN32-ARM64.md"], | |||
There was a problem hiding this comment.
This line was causing an issue when testing documentation in CI (although the file should be there?), should I remove this line for now?
The hard crashes must be fixed before we can include this in the nightly test runs, and the rest need to be investigated and fixed before we merge it. :-) |
|
I found that a few of these hard-crashed ones were related to #ifdef __WIN32__
printf("[debug] we're in display_string_2 win32\r\n");
HANDLE fd;
if (ERTS_IS_ATOM_STR("stdout", BIF_ARG_1)) {
fd = GetStdHandle(STD_OUTPUT_HANDLE);
printf("[debug] and the BIF_ARG_1 is stdout, fd=%d\r\n", fd);
} else if (ERTS_IS_ATOM_STR("stderr", BIF_ARG_1)) {
fd = GetStdHandle(STD_ERROR_HANDLE);
printf("[debug] and the BIF_ARG_1 is stderr, fd=%d\r\n", fd);
if (fd == INVALID_HANDLE_VALUE) {
printf("[debug] and fd == INVALID_HANDLE_VALUE\r\n");
}
}
#else
// ...
#endif
#ifdef __WIN32__
Uint32 w;
if (!WriteFile(fd, str, len, &w, NULL)) {
DWORD last_error = GetLastError();
LPTSTR error_text = NULL;
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, HRESULT_FROM_WIN32(last_error), MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPTSTR)&error_text, 0, NULL);
if (error_text != NULL) {
printf("[debug] error msg: %s\r\n", error_text);
LocalFree(error_text);
} else {
printf("[debug] error msg: cannot get formatted error message\r\n");
}
goto error;
}
written = (Sint)w;
#else
// ...
#endifThe coresponding test case was changed to: display_string(Config) when is_list(Config) ->
# using `erlang:display` for comparison and check that we can display to console
true = erlang:display("hej"),
true = erlang:display_string("hej"),
ok.And I got this output when running this test case: And I also attempted to use #ifdef __WIN32__
int fd;
if (ERTS_IS_ATOM_STR("stdout", BIF_ARG_1)) {
fd = _fileno(stdout);
} else if (ERTS_IS_ATOM_STR("stderr", BIF_ARG_1)) {
fd = _fileno(stderr);
}
#else
// ...
#endif
#ifdef __WIN32__
written = 0;
do {
res = _write(fd, str+written, len-written);
if (res < 0 && errno != ERRNO_BLOCK && errno != EINTR) {
printf("[debug] _write went wrong, errno=%d\r\n", errno);
goto error;
}
written += res;
} while (written < len);
#else
// ...
#endifThe test outputted: Yet this function seems to be fine when using I wonder if we should skip this test for ARM64 Windows or using some alternatives? |
And for this particular function, if [ X"$X64" = X"true" ]; then
echo_setenv OVERRIDE_TARGET win32 ';'
echo_setenv CONFIG_SUBTYPE win64 ';'
elif [ X"$ARM64" = X"arm64" -o X"$ARM64" = X"x64_arm64" ]; then
echo_setenv OVERRIDE_TARGET aarch64-pc-windows ';'
echo_setenv CONFIG_SUBTYPE "$ARM64" ';'
else
echo_setenv OVERRIDE_TARGET win32 ';'
fi |
|
Isn't win32 used as the name of the used API and we still use that on arm for windows? |
Oh it's probably just a minor thing. I was trying to find a way to detect the CPU architecture using erlang:system_info(system_architecture).For other systems, the result can usually be used to check the CPU architecture, for example, os:getenv("PROCESSOR_ARCHITECTURE").Besides that, it also doesn't follow the official document: Maybe it's too late and would definitely break a lot of things for x86 and x86_64 Windows, but perhaps we can do it for ARM64 Windows. WDYT? |
|
I still can't figure out why Windows think the file handle we get from emulator
kernel
I'll upload a screenshot for the |
|
Some explanation for commit 892019a During my tests, I found that erts_time_sup__.r.o.start_offset.nsec = (ErtsMonotonicTime)
erts_time_unit_conversion((Uint64) abs_native_offset,
(Uint32) ERTS_MONOTONIC_TIME_UNIT,
(Uint32) 1000*1000*1000);The value of #include <stdio.h>
#include <stdint.h>
uint64_t
erts_time_unit_conversion(uint64_t value,
uint32_t from_time_unit,
uint32_t to_time_unit)
{
uint64_t high, low, result;
if (value <= ~((uint64_t) 0)/to_time_unit)
return (value*to_time_unit)/from_time_unit;
low = value & ((uint64_t) 0xffffffff);
high = (value >> 32) & ((uint64_t) 0xffffffff);
low *= to_time_unit;
high *= to_time_unit;
high += (low >> 32) & ((uint64_t) 0xffffffff);
low &= ((uint64_t) 0xffffffff);
result = high % from_time_unit;
high /= from_time_unit;
printf("[!] high=%llu\n", high);
printf("[!] high will overflow=%s\n", (high >> 32) != 0 ? "yes" : "no");
high <<= 32;
printf("[!] high=%llu, result=%llu\n", high, result);
result <<= 32;
result += low;
result /= from_time_unit;
result += high;
return result;
}
int main(int argc, char *argv[]) {
printf("%lld\n\n", erts_time_unit_conversion(576460752312000000, 24000000, 1000*1000*1000));
printf("%lld\n", erts_time_unit_conversion(576460752312000000, 24000000, 1000*1000));
}The output of the above minimal code is The correct value of the final result should be And this overflow will cause all subsequent calculations depending on As for the fix, I think one possible (and perhaps the easiest and the most striaghtforwad) way is that we can simply do the calculations in erlang code. I tried to detect if there's any overflow in the C code and if so, we handle it in erlang code. But after I've done that, I feel that it just adds some unnecessary complexity to the C code. I hope all this would make sense, but I'm happy to make any other changes to the C code if there's a better solution for it. |
|
I was wondering if I should try to use |
|
Hi all! As this is kinda stale, I wonder if it would be easier for us if I break this PR into smaller PRs? I can start with PRs that definitely won't break any existing things. |
Hi, this PR includes initial patches and instructions to bring Erlang/OTP to ARM64 Windows. These changes should only affect the behaviour when running in WSL and user executes one of the following command to set up the build environment
./otp_build env_win32 arm64, set up build environment for compiling ARM64 Erlang/OTP natively,./otp_build env_win32 x64_arm64, set up build environment for cross-compiling ARM64 Erlang/OTP on an x86_64 machineAlthough I understand that ARM64 Windows might not be anywhere close to the main tier of support yet, this PR can serve as an initial guide for people in the community who're also interested in running Erlang/OTP natively on ARM64 Windows.