Add Aarch64 support for macOS #10066

RomainFranceschini · 2020-12-10T00:19:01Z

Since Apple now ships ARM processors on their computers, it would be nice to have this platform supported by the Crystal compiler.

Crystal already supports AArch64 for Linux and Darwin on x86_64, so achieve Tier 3 support as a first step should be feasible ~~without too much work~~.
EDIT: Actually, an ABI for Apple devices should be adapted from the aarch64 one.

A target triple such as aarch64-apple-darwin can already be provided to the --target argument, but the compiler will fail as C bindings for this target are missing.

This commit adds the LibC bindings for the target. It is basically a copy-paste-modify from the x86_64-darwin target, where the modify part is inspired by what the rust team did (rust-lang/libc#1817) for the same purpose.

It seems to be enough to produce simple binaries, maybe it could be the basis for a PR if you think it goes into the right direction.

Cross-compiling

To test whether this slight change is enough for cross-compilation, you need (besides the hardware):

A side-by-side installation of homebrew (one for ARM, one for x86_64). I will refer to them as brew and ibrew respectively.
Install Crystal runtime requirements for ARM brew install gmp libevent libyaml openssl@1.1 pcre pkg-config bdw-gc. While homebrew on ARM is not officially supported yet, those compiles fine from sources.
Install the x86_64 official release of Crystal as well as LLVM 9 since Crystal does not support LLVM 11 yet. ibrew install crystal-lang llvm@9

The next step is to build the modified compiler for x86_64 first (using Homebrew's crystal running under Rosetta):

LLVM_CONFIG="$(ibrew --prefix)/opt/llvm@9/bin/llvm-config make"

Now we can try to cross-compile programs to ARM using our built from source Crystal.

First, we need to build an ARM version of libcrystal:

cc -c -o sigfault.o src/ext/sigfault.c/sigfault.c
ar -rcs libcrystal.a sigfault.o

And finally:

./bin/crystal build samples/binary-trees.cr --cross-compile --target aarch64-apple-darwin --release
cc binary-trees.o -o binary-trees -rdynamic -L$(brew --prefix)/opt/bdw-gc/lib -L$(brew --prefix)/opt/pcre/lib -lpcre -lgc -lpthread libcrystalarm.a -L$(brew --prefix)/opt/libevent/lib -levent -liconv -ldl

We can check the architecture with file ./binary-trees. My console shows ./binary-trees: Mach-O 64-bit executable arm64

The next step would be to cross compile the compiler itself, but it is more difficult as:

I was not able to compile LLVM 9 for ARM.
LLVM 11 for ARM compiles, but Crystal does not support this version yet (see #9809).

The text was updated successfully, but these errors were encountered:

jhass · 2020-12-10T07:26:24Z

Huh, I thought Apple did their own ABI instead of following AAPCS?

n-rodriguez · 2020-12-10T08:03:29Z

Before or after 1.0?

RomainFranceschini · 2020-12-10T10:18:39Z

@jhass Yes, it seems their arm64 ABI differs a bit from AAPCS, based on https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms.

There's also, the arm64e ABI for ARMV8.3 which could be considered for another issue.

jhass · 2020-12-10T10:20:31Z

So that needs implementation too :) https://github.com/crystal-lang/crystal/tree/master/src/llvm/abi

n-rodriguez · 2020-12-10T10:27:13Z

@jhass don't be so confused it's just a question

jhass · 2020-12-10T10:28:29Z

It's gonna be done when somebody finds the motivation and time for it, like everything else in a dominantly open source project ;)

RomainFranceschini · 2020-12-10T10:28:31Z

I will definitely try to implement this since I have access to the hardware.
The extra benefit would be supporting iOS devices :-)

bcardiff · 2020-12-10T15:16:50Z

@RomainFranceschini did you check if LLVM 10 can be built for ARM? Even if it's not available on brew you can point LLVM_CONFIG to whatever llvm-config binary of your choice.

It seems to be enough to produce simple binaries, maybe it could be the basis for a PR if you think it goes into the right direction.

I think it does. The next binary to try would be the std_spec. If LLVM 10 happens to work we could be in a better shape and check if compiler_spec pass.

Whatever you are able to accomplish it will be valuable.

Ideally we need to find a way / wait for AArch64 macOS CI to avoid going backwards with its support.

jhass · 2020-12-10T15:28:26Z

Maybe somebody will go through the trouble of doing a Linux distribution using Apple's ABI, then we could spin up VMs of that on our existing arm64 machine and it should be close enough, at least for a temporary solution.

RomainFranceschini · 2020-12-16T17:56:46Z

@bcardiff I did several attempts at building LLVM 10 from source, unsuccessfully. It seems checking whether std_spec and compiler_spec pass will have to wait.

Regarding the ABI, I tried to review what would need an update from the existing aarch64 implementation, and AFAIU:

The reserved x18 register should not be a concern when generating LLVM IR, but it is a concern for manually written assembly. It is not used in src/fiber/context/aarch64.cr.
Data types seems already correct, both in bytes size and alignment.

I still need to review more thoroughly how arguments passed on the stack should be aligned.

bcardiff · 2020-12-21T20:47:36Z

@RomainFranceschini I was able to move forward with llvm 11 support at #9829

maxfierke · 2021-01-28T04:59:03Z

Was able to get the compiler building on the M1 Air with @RomainFranceschini's patch using the following steps:

LLVM 11 on both ARM and Intel (both from Brew), though compiled bdw-gc, sigfault.o/libcrystal.a w/ just Clang from Xcode 12

Environment

export INTEL_BREW_PREFIX="$(ibrew --prefix)"
export INTEL_LLVM_ROOT="$INTEL_BREW_PREFIX/opt/llvm"
export INTEL_LLVM_CONFIG="$INTEL_LLVM_ROOT/bin/llvm-config"

export ARM_BREW_PREFIX="$(brew --prefix)"
export ARM_LLVM_ROOT="$ARM_BREW_PREFIX/opt/llvm"
export ARM_LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config"

Compiling libcrystal.a

cc -c -o sigfault.o src/ext/sigfault.c
ar -rcs libcrystal.a sigfault.o

Compiling bdw-gc

$ git clone https://github.com/ivmai/bdwgc.git
$ cd bdwgc
$ git clone https://github.com/ivmai/libatomic_ops.git
$ autoreconf -vif
$ ./configure --enable-static --disable-shared
$ make -j
$ make check

Then copy .libs/libgc.a into the Crystal source root

Compiling Crystal for ARM

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   CPPFLAGS="-I$INTEL_LLVM_ROOT/include" \
   CC="$INTEL_LLVM_ROOT/bin/clang" \
   AR="$INTEL_LLVM_ROOT/bin/llvm-ar" \
   arch -x86_64 make

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   arch -x86_64 ./bin/crystal build src/compiler/crystal.cr --cross-compile --target aarch64-apple-darwin -Dwithout_playground

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   $ARM_LLVM_ROOT/bin/clang -I$ARM_LLVM_ROOT/include -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   $ARM_LLVM_ROOT/bin/clang crystal.o -o crystal  -rdynamic -L$ARM_BREW_PREFIX/lib src/llvm/ext/llvm_ext.o `"$ARM_LLVM_ROOT/bin/llvm-config" --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre libgc.a -lpthread libcrystal.a -L$(brew --prefix libevent)/lib -levent -liconv -ldl

$ file ./crystal
./crystal: Mach-O 64-bit executable arm64

Unfortunately, I think there's still a missing piece with maybe how the LLVM target triple is being presented or something in the ARM host environment, ~~as it doesn't seem to be loading the libc bindings properly:~~

crystal eval output

$ CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib ./crystal eval 'puts "Hello ARM!"' --error-trace
error in line 1
Error: while requiring "prelude"


In src/prelude.cr:19:1

 19 | require "exception"
      ^
Error: while requiring "exception"


In src/exception.cr:1:1

 1 | require "./exception/call_stack"
     ^
Error: while requiring "./exception/call_stack"


In src/exception/call_stack.cr:3:1

 3 | require "c/dlfcn"
     ^
Error: can't find file 'c/dlfcn'

If you're trying to require a shard:
- Did you remember to run `shards install`?
- Did you make sure you're running the compiler in the same directory as your shard.yml?

~~This seems odd, because $CRYSTAL_PATH/lib_c/aarch64-darwin/c/dlfcn.cr, does exist.~~

~~I've tried setting up some symlinks w/ various other permutations, aarch64-apple-darwin, arm64-darwin, unknown-darwin, and -darwin, but no luck~~.

EDIT: The answer seems to be yes, the target is being presented differently, as arm-darwin when the compile is running with ARM64 as host. Adding a symlink for arm-darwin -> aarch64-darwin resolves the libc bindings issue.

However, the fun stops with an LLVM error.

LLVM error on eval

CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib ./crystal eval 'puts "Hello ARM!"' --error-trace --debug
"arm-darwin"
LLVM ERROR: Cannot select: 0x13080b828: ch,glue = ARMISD::CALL 0x13080b6f0, 0x13080b688, Register:i32 $r0, RegisterMask:Untyped, 0x13080b6f0:1, pthread.cr:62:3
  0x13080b688: i32,ch = load<(non-temporal dereferenceable invariant load 4 from got)> 0x117e35c38, 0x13080b008, undef:i32, pthread.cr:62:3
    0x13080b008: i32,ch = load<(load 4 from got)> 0x117e35c38, 0x13080b620, undef:i32, pthread.cr:62:3
      0x13080b620: i32 = ARMISD::WrapperPIC TargetGlobalTLSAddress:i32<%Thread** @"Thread::current"> 0 [TF=128], pthread.cr:62:3
        0x13080ae68: i32 = TargetGlobalTLSAddress<%Thread** @"Thread::current"> 0 [TF=128], pthread.cr:62:3
      0x13080b348: i32 = undef
    0x13080b348: i32 = undef
  0x13080b070: i32 = Register $r0
  0x13080b758: Untyped = RegisterMask
  0x13080b6f0: ch,glue = CopyToReg 0x13080b688:1, Register:i32 $r0, 0x13080b008, pthread.cr:62:3
    0x13080b070: i32 = Register $r0
    0x13080b008: i32,ch = load<(load 4 from got)> 0x117e35c38, 0x13080b620, undef:i32, pthread.cr:62:3
      0x13080b620: i32 = ARMISD::WrapperPIC TargetGlobalTLSAddress:i32<%Thread** @"Thread::current"> 0 [TF=128], pthread.cr:62:3
        0x13080ae68: i32 = TargetGlobalTLSAddress<%Thread** @"Thread::current"> 0 [TF=128], pthread.cr:62:3
      0x13080b348: i32 = undef
In function: *Thread::current:Thread
LLVM ERROR: Cannot select: 0x130193970: ch,glue = ARMISD::CALL 0x1301a1e10, 0x13019db38, Register:i32 $r0, RegisterMask:Untyped, 0x1301a1e10:1
  0x13019db38: i32,ch = load<(non-temporal dereferenceable invariant load 4 from got)> 0x127fba3e8, 0x130180c78, undef:i32
    0x130180c78: i32 = ARMISD::WrapperPIC TargetGlobalTLSAddress:i32<%Thread** @"Thread::current"> 0 [TF=128]
      0x13019b150: i32 = TargetGlobalTLSAddress<%Thread** @"Thread::current"> 0 [TF=128]
    0x1300a38d0: i32 = undef
  0x1301a14f8: i32 = Register $r0
  0x130198838: Untyped = RegisterMask
  0x1301a1e10: ch,glue = CopyToReg 0x13019db38:1, Register:i32 $r0, 0x130180c78
    0x1301a14f8: i32 = Register $r0
    0x130180c78: i32 = ARMISD::WrapperPIC TargetGlobalTLSAddress:i32<%Thread** @"Thread::current"> 0 [TF=128]
      0x13019b150: i32 = TargetGlobalTLSAddress<%Thread** @"Thread::current"> 0 [TF=128]
In function: *Thread::current
clang: error: no such file or directory: '_main.o'
clang: error: no such file or directory: 'D-ir.o'
clang: error: no such file or directory: 'C-rystal5858S-ystem5858D-ir.o'
clang: error: no such file or directory: 'F-ile5858E-rror.o'
clang: error: no such file or directory: 'E-rrno.o'
clang: error: no such file or directory: 'S-tring.o'
clang: error: no such file or directory: 'S-tring5858B-uilder.o'
clang: error: no such file or directory: 'A-rgumentE-rror.o'
clang: error: no such file or directory: 'E-xception5858C-allS-tack.o'
clang: error: no such file or directory: 'A-rray40P-ointer40V-oid4141.o'
clang: error: no such file or directory: 'P-ointer40P-ointer40V-oid4141.o'
clang: error: no such file or directory: 'P-ointer40V-oid41.o'
clang: error: no such file or directory: 'I-nt32.o'
clang: error: no such file or directory: 'F-loat64.o'
clang: error: no such file or directory: 'I-nt64.o'
clang: error: no such file or directory: 'E-xception.o'
clang: error: no such file or directory: 'P-ointer40L-ibU-nwind5858C-ontrolB-lock41.o'
clang: error: no such file or directory: 'U-I-nt32.o'
clang: error: no such file or directory: 'G-C-.o'
clang: error: no such file or directory: 'C-har5858R-eader.o'
clang: error: no such file or directory: 'P-ointer40U-I-nt841.o'
clang: error: no such file or directory: 'C-har.o'
clang: error: no such file or directory: 'S-lice40T-41.o'
clang: error: no such file or directory: 'S-lice40U-I-nt841.o'
clang: error: no such file or directory: 'I-O-5858E-ncoder.o'
clang: error: no such file or directory: 'C-rystal5858I-conv.o'
clang: error: no such file or directory: 'T-uple40S-tring4432S-tring41.o'
clang: error: no such file or directory: 'E-numerable5858R-eflect40I-nt3241.o'
clang: error: no such file or directory: 'T-uple40S-tring4432S-tring4432S-tring4432S-tring41.o'
clang: error: no such file or directory: 'R-untimeE-rror.o'
clang: error: no such file or directory: 'T-hread.o'
clang: error: no such file or directory: 'A-tomic40U-I-nt841.o'
clang: error: no such file or directory: 'A-tomic40T-hread3212432N-il41.o'
clang: error: no such file or directory: 'F-iber5858C-ontext.o'
clang: error: no such file or directory: 'T-hread5858L-inkedL-ist40T-hread41.o'
clang: error: no such file or directory: 'H-ash5858E-ntry40T-hread4432D-eque40F-iber4141.o'
clang: error: no such file or directory: 'U-I-nt64.o'
clang: error: no such file or directory: 'C-rystal5858H-asher.o'
clang: error: no such file or directory: 'S-taticA-rray40U-I-nt644432241.o'
clang: error: no such file or directory: 'P-ointer40U-I-nt6441.o'
clang: error: no such file or directory: 'P-ointer40U-I-nt1641.o'
clang: error: no such file or directory: 'P-ointer40U-I-nt3241.o'
clang: error: no such file or directory: 'D-eque40F-iber41.o'
clang: error: no such file or directory: 'H-ash5858E-ntry40T-hread4432C-rystal5858E-vent41.o'
clang: error: no such file or directory: 'C-rystal5858E-ventL-oop.o'
clang: error: no such file or directory: 'L-ibE-vent25858E-ventF-lags.o'
clang: error: no such file or directory: 'C-rystal5858E-vent5858B-ase.o'
clang: error: no such file or directory: 'P-ointer40F-iber41.o'
clang: error: no such file or directory: 'C-rystal5858S-cheduler.o'
clang: error: no such file or directory: 'N-il.o'
clang: error: no such file or directory: 'N-ilA-ssertionE-rror.o'
clang: error: no such file or directory: 'C-rystal5858E-vent.o'
Error: execution of command failed with code: 1: `cc "${@}" -o $HOME/.cache/crystal/crystal-run-eval.tmp  -rdynamic -L/opt/homebrew/lib -lpcre -lgc -lpthread $HOME/src/crystal/src/ext/libcrystal.a -L/opt/homebrew/Cellar/libevent/2.1.12/lib -levent -liconv -ldl`

Perhaps someone with a good eye for LLVM IR or ARM asm can spot the issue immediately.

bcardiff · 2021-01-28T16:10:09Z

FTR at

crystal/src/compiler/crystal/crystal_path.cr

Lines 36 to 44 in be5de4a

    
           target = "#{codegen_target.architecture}-#{codegen_target.os_name}" 
        
           @entries.each do |path| 
        
             path = File.join(path, "lib_c", target) 
        
             if Dir.exists?(path) 
        
               @entries << path unless @entries.includes?(path) 
        
               return 
        
             end 
        
           end

you can find how lib_c are lookup based on the target.

maxfierke · 2021-01-28T16:13:54Z

FTR at

crystal/src/compiler/crystal/crystal_path.cr

Lines 36 to 44 in be5de4a

target = "#{codegen_target.architecture}-#{codegen_target.os_name}"

@entries.each do |path|

path = File.join(path, "lib_c", target)

if Dir.exists?(path)

@entries << path unless @entries.includes?(path)

return

end

end

you can find how lib_c are lookup based on the target.

Yep, adding a print there pointed to the issue. But still unclear as to why the target is "arm-darwin" instead of something expected like "arm64-darwin" or "aarch64-darwin".

maxfierke · 2021-01-29T06:58:23Z

Looked into it further. It all seems to stem from LLVMGetDefaultTargetTriple (from LLVM 11 via brew) reporting arm-darwin, which consequently enables the :arm flag, instead of the :aarch64 flag. If you compile for --target aarch64-apple-darwin specifically, the correct flags get set and it's able to move past the LLVM error in my earlier post, by doing the proper TLS setup for aarch64 and not 32-bit arm

@bcardiff I see we already do some massaging of the default target triple for macOS here:

crystal/src/llvm.cr

Lines 76 to 84 in 90bbb0a

    
           def self.default_target_triple : String 
        
             chars = LibLLVM.get_default_target_triple 
        
             triple = string_and_dispose(chars) 
        
             if triple =~ /x86_64-apple-macosx|x86_64-apple-darwin/ 
        
               "x86_64-apple-macosx" 
        
             else 
        
               triple 
        
             end 
        
           end

Would it be reasonable to handle arm-darwin similarly, by treating it as aarch64-apple-darwin? Unless there's a secret 32-bit ARM Mac I'm not aware of, I can't see any problems with that outside of potentially papering over an LLVM bug.

RomainFranceschini · 2021-01-29T07:30:43Z

On my side I was able to compile std_spec using LLVM 11.

Finished in 34.79 seconds
10768 examples, 25 failures, 0 errors, 7 pending

Failed examples:

crystal spec spec/std/complex_spec.cr:82 # Complex cis
crystal spec spec/std/exception_spec.cr:38 # Exception collect memory within ensure block
crystal spec spec/std/kernel_spec.cr:5 # exit exits normally with status 0
crystal spec spec/std/kernel_spec.cr:10 # exit exits with given error code
crystal spec spec/std/kernel_spec.cr:18 # at_exit runs handlers on normal program ending
crystal spec spec/std/kernel_spec.cr:29 # at_exit runs handlers on explicit program ending
crystal spec spec/std/kernel_spec.cr:42 # at_exit runs handlers in reverse order
crystal spec spec/std/kernel_spec.cr:61 # at_exit runs all handlers maximum once
crystal spec spec/std/kernel_spec.cr:88 # at_exit allows handlers to change the exit code with explicit `exit` call
crystal spec spec/std/kernel_spec.cr:116 # at_exit allows handlers to change the exit code with explicit `exit` call (2)
crystal spec spec/std/kernel_spec.cr:146 # at_exit changes final exit code when an handler raises an error
crystal spec spec/std/kernel_spec.cr:175 # at_exit shows unhandled exceptions after at_exit handlers
crystal spec spec/std/kernel_spec.cr:196 # at_exit can get unhandled exception in at_exit handler
crystal spec spec/std/kernel_spec.cr:212 # at_exit allows at_exit inside at_exit
crystal spec spec/std/kernel_spec.cr:235 # seg fault reports SIGSEGV
crystal spec spec/std/kernel_spec.cr:245 # seg fault detects stack overflow on the main stack
crystal spec spec/std/kernel_spec.cr:267 # seg fault detects stack overflow on a fiber stack
crystal spec spec/std/process_spec.cr:167 # Process chroot raises when unprivileged
crystal spec spec/std/va_list_spec.cr:4 # VaList works with C code
crystal spec spec/std/exception/call_stack_spec.cr:4 # Backtrace prints file line:column
crystal spec spec/std/exception/call_stack_spec.cr:25 # Backtrace doesn't relativize paths outside of current dir (#10169)
crystal spec spec/std/exception/call_stack_spec.cr:43 # Backtrace prints exception backtrace to stderr
crystal spec spec/std/exception/call_stack_spec.cr:52 # Backtrace prints crash backtrace to stderr
crystal spec spec/std/io/file_descriptor_spec.cr:4 # IO::FileDescriptor reopen STDIN with the right mode
crystal spec spec/std/spec/hooks_spec.cr:5 # Spec hooks runs in correct order

I was not able to move forward with the ABI yet, due to lack of time (and knowledge).
AFAIK, some errors such as those related to variadic arguments are directly linked to this missing patch.

maxfierke · 2021-01-29T07:35:46Z

Interesting... after massaging the LLVM target triple to aarch64-apple-darwin, std_spec and compiler_spec completed with only two errors each for me, which I think are mostly environmental

std_spec

Pending:
  Math Functions for computing quotient and remainder
  Path #expand converts a pathname to an absolute pathname, using a complete path assert
  Spec matchers pending block is not compiled pending has block with valid syntax, but invalid semantics
  URI .parse unescaped @ in user/password should not confuse host
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers
  UDPSocket using IPv6 joins and transmits to multicast groups
  UDPSocket sends broadcast message

Failures:

  1) Complex cis
     Failure/Error: 2.4.cis.should eq(Complex.new(-0.7373937155412454, 0.675463180551151))

       Expected: (-0.7373937155412454 + 0.675463180551151i)
            got: (-0.7373937155412454 + 0.6754631805511511i)

     # spec/std/complex_spec.cr:83

  2) VaList works with C code
     Failure/Error: File.exists?(executable_file).should be_true

       Expected: true
            got: false

     # spec/std/spec_helper.cr:81

Finished in 1:34 minutes
10887 examples, 2 failures, 0 errors, 7 pending

Failed examples:

crystal spec spec/std/complex_spec.cr:82 # Complex cis
crystal spec spec/std/va_list_spec.cr:4 # VaList works with C code

for compiler_spec, just two as well (first one because I had an amd64 build sitting in .build/crystal):

compiler_spec

Pending:
Code gen: lib codegens lib var set and get
Code gen: primitives codegens pointer of int
Code gen: primitives sums two numbers out of an [] of Number
Code gen: C ABI accepts large struct in a callback (for real) (#9533)
Crystal::Doc::Markdown::DocRenderer expand_code_links doesn't find wrong kind of sibling methods
Crystal::Doc::Markdown::DocRenderer expand_code_links doesn't find wrong kind of methods
Semantic: cast casts from union to incompatible union gives error
Semantic: def overload restricts on generic type with free type arg
Semantic: def overload restricts on generic type without type arg
Semantic: instance var doesn't infer type to be nilable if using self.class in call in assign
Semantic: pointer allows using pointer with subclass
Semantic: primitives types pointer of int

Failures:

1) Crystal::Config .host_target
  Failure/Error: Crystal::Config.host_target.should eq Crystal::Codegen::Target.new({{ `bin/crystal --version`.lines[-1] }}.lchop("Default target: "))

    Expected: #<Crystal::Codegen::Target:0x113f54b70 @architecture="x86_64", @vendor="apple", @environment="macosx">
         got: #<Crystal::Codegen::Target:0x109cded80 @architecture="aarch64", @vendor="apple", @environment="darwin">

  # spec/compiler/config_spec.cr:6

2) Compiler treats all arguments post-filename as program arguments

    Error opening file with mode 'r': '/var/folders/76/vgfv152n0z30zllkfx7c9ppw0000gr/T/cr-spec-39755aaf/compiler/args_test': No such file or directory (File::NotFoundError)
      from src/crystal/system/unix/file.cr:11:7 in 'open'
      from src/file.cr:111:5 in 'new'
      from src/file.cr:627:5 in 'read'
      from src/file.cr:641:3 in 'read'
      from spec/compiler/compiler_spec.cr:35:7 in '->'
      from src/primitives.cr:255:3 in 'internal_run'
      from src/spec/example.cr:33:16 in 'run'
      from src/spec/context.cr:18:23 in 'internal_run'
      from src/spec/context.cr:330:7 in 'run'
      from src/spec/context.cr:18:23 in 'internal_run'
      from src/spec/context.cr:147:7 in 'run'
      from src/spec/dsl.cr:274:7 in '->'
      from src/primitives.cr:255:3 in 'run'
      from src/crystal/main.cr:45:14 in 'main'
      from src/crystal/main.cr:119:3 in 'main'


Finished in 16:04 minutes
9867 examples, 1 failures, 1 errors, 12 pending

Failed examples:

crystal spec spec/compiler/config_spec.cr:5 # Crystal::Config .host_target
crystal spec spec/compiler/compiler_spec.cr:31 # Compiler treats all arguments post-filename as program arguments

RomainFranceschini · 2021-01-29T07:55:00Z

Hm.. after reading your compiling steps more closely, mine differs a bit. I'll try yours later (probably this week-end) and keep you posted.

As for the compiler_spec, compilation fails altogether on my side.

bcardiff · 2021-01-29T12:25:22Z

@maxfierke so, llvm on brew reports arm- , but outside brew reports aarch64-? If so, yes, we hand handle that difference in default_target_triple.

maxfierke · 2021-01-29T18:28:02Z

@maxfierke so, llvm on brew reports arm- , but outside brew reports aarch64-? If so, yes, we hand handle that difference in default_target_triple.

A little bit weirder than that... llc -version (from LLVM 11 from brew) reports default target of arm64-apple-darwin20.2.0 just fine, but LLVMGetDefaultTargetTriple (from LLVM 11 from brew) returns arm-darwin

bcardiff · 2021-01-29T18:50:15Z

Hm, ok.... then let's go wiith the symlink in lib_c to support arm-darwin and aarch64-darwin . In i686 and i386 we do that. And the rewrite of x86_64-apple-darwin to x86_64-apple-macosx is to deal with darwin/macosx ambiguity and not arch aliases. Maybe is better to use the same solution as in i686/i386 and that's it.

maxfierke · 2021-01-29T19:03:54Z

@bcardiff regardless of the libc issue, the problem with using arm-darwin is that Crystal will then treat it like 32-bit ARM instead of AArch64, so we'd need to deal with that elsewhere (maybe when determining flags). Either approach deals with it, just sort of a question where it makes the most sense to live.

bcardiff · 2021-01-29T20:18:56Z

You mean de abi that is used, right?. That can be handled in src/llvm/target_machine.cr LLVM::TargetMachine#abi. There is already a special case for windows there. WDYT?

maxfierke · 2021-01-29T23:19:43Z

You mean de abi that is used, right?. That can be handled in src/llvm/target_machine.cr LLVM::TargetMachine#abi. There is already a special case for windows there. WDYT?

Yes and no. ABI but also architecture flags. However, I did find the source of arm-darwin, and it's not LLVM. I think I was focusing too much on the call to LLVMGetDefaultTargetTriple and not noticing that Crystal::Codegen::Target sets the architecture for anything starting with arm to arm, so arm64 (from the LLVM default) was getting set to arm, which is how arm-darwin ended up in the libc lookup.

crystal/src/compiler/crystal/codegen/target.cr

Lines 23 to 32 in b632525

    
           case @architecture 
        
           when "i486", "i586", "i686" 
        
             @architecture = "i386" 
        
           when "amd64" 
        
             @architecture = "x86_64" 
        
           when .starts_with?("arm") 
        
             @architecture = "arm" 
        
           else 
        
             # no need to tweak the architecture 
        
           end

Adding a case there for turning arm64 into aarch64 and then updating the regex in TargetMachine as well, to handle both arm64 and aarch64 works well

Re-ran the test suite, and it mostly agrees 😄 .

compiler_spec

$ CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib CC="$ARM_LLVM_ROOT/bin/clang" LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" LDFLAGS="-L$ARM_LLVM_ROOT/lib" CPPFLAGS="-I$ARM_LLVM_ROOT/include" PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" ./crystal build -o spec/compiler_spec spec/compiler_spec.cr --link-flags "$PWD/libcrystal.a" -Di_know_what_im_doing
$ CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib CC="$ARM_LLVM_ROOT/bin/clang" LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" LDFLAGS="-L$ARM_LLVM_ROOT/lib $(pwd)/libgc.a $(pwd)/libcrystal.a" CPPFLAGS="-I$ARM_LLVM_ROOT/include" PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" spec/compiler_spec

[ ... ]

Pending:
  Code gen: lib codegens lib var set and get
  Code gen: primitives codegens pointer of int
  Code gen: primitives sums two numbers out of an [] of Number
  Code gen: C ABI accepts large struct in a callback (for real) (#9533)
  Crystal::Doc::Markdown::DocRenderer expand_code_links doesn't find wrong kind of sibling methods
  Crystal::Doc::Markdown::DocRenderer expand_code_links doesn't find wrong kind of methods
  Semantic: cast casts from union to incompatible union gives error
  Semantic: def overload restricts on generic type with free type arg
  Semantic: def overload restricts on generic type without type arg
  Semantic: instance var doesn't infer type to be nilable if using self.class in call in assign
  Semantic: pointer allows using pointer with subclass
  Semantic: primitives types pointer of int

Finished in 19:28 minutes
9867 examples, 0 failures, 0 errors, 12 pending

std_spec

$ CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib CC="$ARM_LLVM_ROOT/bin/clang" LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" LDFLAGS="-L$ARM_LLVM_ROOT/lib" CPPFLAGS="-I$ARM_LLVM_ROOT/include" PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" ./crystal build -o spec/std_spec spec/std_spec.cr --target aarch64-apple-darwin --link-flags "$PWD/libcrystal.a" -Di_know_what_im_doing
$ CRYSTAL_PATH=$(pwd)/src CRYSTAL_LIBRARY_PATH=/opt/homebrew/lib CC="$ARM_LLVM_ROOT/bin/clang" LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" LDFLAGS="-L$ARM_LLVM_ROOT/lib $(pwd)/libgc.a $(pwd)/libcrystal.a" CPPFLAGS="-I$ARM_LLVM_ROOT/include" PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig" spec/std_spec

Pending:
  Math Functions for computing quotient and remainder
  Path #expand converts a pathname to an absolute pathname, using a complete path assert
  Spec matchers pending block is not compiled pending has block with valid syntax, but invalid semantics
  URI .parse unescaped @ in user/password should not confuse host
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers
  UDPSocket using IPv6 joins and transmits to multicast groups
  UDPSocket sends broadcast message

Failures:

  1) Complex cis
     Failure/Error: 2.4.cis.should eq(Complex.new(-0.7373937155412454, 0.675463180551151))

       Expected: (-0.7373937155412454 + 0.675463180551151i)
            got: (-0.7373937155412454 + 0.6754631805511511i)

     # spec/std/complex_spec.cr:83

Finished in 1:41 minutes
10887 examples, 1 failures, 0 errors, 7 pending

Failed examples:

crystal spec spec/std/complex_spec.cr:82 # Complex cis

So, one spec failure all-told, and it's due to Complex being slightly too precise. Hopefully I'll get a chance this weekend to clean this up a bit, and maybe do some stuff to avoid having to pass around all those environment variables. Not supplying them leads to confusing things where a tool or library from Intel-land will be used instead, etc. and it's just not a great experience.

But documented here, if folks with the hardware want to try. Patch (additive to what @RomainFranceschini provided for libc):

0001-Interpret-arm64-target-triple-for-Apple-Silicon-prop.patch

From 9535bb30bdb924638e70138ae1a9bde4e818687b Mon Sep 17 00:00:00 2001
From: Max Fierke <max@maxfierke.com>
Date: Fri, 29 Jan 2021 17:17:42 -0600
Subject: [PATCH] Interpret arm64 target triple for Apple Silicon properly

---
 spec/compiler/codegen/target_spec.cr   | 1 +
 src/compiler/crystal/codegen/target.cr | 2 ++
 src/llvm/target_machine.cr             | 2 +-
 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/spec/compiler/codegen/target_spec.cr b/spec/compiler/codegen/target_spec.cr
index fea72b21e..e6cd8f192 100644
--- a/spec/compiler/codegen/target_spec.cr
+++ b/spec/compiler/codegen/target_spec.cr
@@ -16,6 +16,7 @@ describe Crystal::Codegen::Target do
   it "normalizes triples" do
     Target.new("i686-unknown-linux-gnu").to_s.should eq("i386-unknown-linux-gnu")
     Target.new("amd64-unknown-openbsd").to_s.should eq("x86_64-unknown-openbsd")
+    Target.new("arm64-apple-darwin20.2.0").to_s.should eq("aarch64-apple-darwin20.2.0")
   end

   it "parses freebsd version" do
diff --git a/src/compiler/crystal/codegen/target.cr b/src/compiler/crystal/codegen/target.cr
index 60d9f3b44..4ab099e1a 100644
--- a/src/compiler/crystal/codegen/target.cr
+++ b/src/compiler/crystal/codegen/target.cr
@@ -25,6 +25,8 @@ class Crystal::Codegen::Target
       @architecture = "i386"
     when "amd64"
       @architecture = "x86_64"
+    when "arm64"
+      @architecture = "aarch64"
     when .starts_with?("arm")
       @architecture = "arm"
     else
diff --git a/src/llvm/target_machine.cr b/src/llvm/target_machine.cr
index b2b214f53..015b57fd7 100644
--- a/src/llvm/target_machine.cr
+++ b/src/llvm/target_machine.cr
@@ -53,7 +53,7 @@ class LLVM::TargetMachine
       ABI::X86_64.new(self)
     when /i386|i486|i586|i686/
       ABI::X86.new(self)
-    when /aarch64/
+    when /aarch64|arm64/
       ABI::AArch64.new(self)
     when /arm/
       ABI::ARM.new(self)
--
2.30.0

alin23 · 2021-01-30T12:50:14Z

Awesome work! With @maxfierke 's instructions and @bcardiff 's patch I was able to get working crystal, shards and crystalline binaries, and I was able to recompile all my previous Crystal projects.

RomainFranceschini · 2021-01-31T21:36:06Z

@maxfierke nice work! I was able to get same results. Thanks for picking this up.

Does that mean we don't have to tweak the ABI after all? I'm confused now 😄

maxfierke · 2021-01-31T21:39:40Z

@maxfierke nice work! I was able to get same results. Thanks for picking this up.

Does that mean we don't have to tweak the ABI after all? I'm confused now 😄

I think that's right! AFAICT the arm64 / aarch64 backends and ABI are the same and mean the same thing (aarch64 being then official ARM terminology and arm64 being Apple's terminology)

jhass · 2021-02-01T20:52:20Z

We're probably just lucky so far...

Apple platforms diverge from the standard 64-bit ARM architecture in a few specific ways. Apart from these small differences, iOS, tvOS, and macOS adhere to the rest of the 64-bit ARM specification

https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms

maxfierke · 2021-02-04T01:42:24Z

We're probably just lucky so far...

Apple platforms diverge from the standard 64-bit ARM architecture in a few specific ways. Apart from these small differences, iOS, tvOS, and macOS adhere to the rest of the 64-bit ARM specification

https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms

Yeah it seems that way:

"Respect the Purpose of Specific CPU Registers"
- We don't touch x18, and we populate x29 in our fiber switching inline asm
"Function arguments may consume slots on the stack that are not multiples of 8 bytes. If the total number of bytes for stack-based arguments is not a multiple of 8 bytes, insert padding on the stack to maintain the 8-byte alignment requirements."
- I think we do this:
  
  crystal/src/compiler/crystal/codegen/call.cr
  
  Line 256 in f0901e1
  
  align = @abi.align(abi_arg_type.type)
"When passing an argument with 16-byte alignment in integer registers, Apple platforms allow the argument to start in an odd-numbered xN register. The standard ABI requires it to begin in an even-numbered xN register."
- This is a looser restriction than AArch64. Not sure if this is handled automatically by LLVM or not but maybe we could support this later.
The caller of a function is responsible for signing or zero-extending any argument with fewer than 32 bits. The standard ABI expects the callee to sign or zero-extend those arguments.
- We already do this:
  
  crystal/src/compiler/crystal/codegen/call.cr
  
  Line 237 in f0901e1
  
  call_arg = extend_int arg_type, @program.int32, call_arg
Functions may ignore parameters that contain empty struct types. This behavior applies to the GNU extension in C and, where permitted by the language, in C++. The AArch64 documentation doesn’t address the issue of empty structures as parameters, but Apple chose this path for its implementation.
- This is a restriction relaxation from AArch64. Not sure if we do this or not, but doesn't seem to matter either way (maybe a later optimization if we don't.)
Update Code that Passes Arguments to Variadic Functions
- We use the va_start and va_end LLVM intrinsics, so I think we're covered here.

jhass · 2021-02-04T07:27:44Z

Did LLVM fix up the variadic intrinsics? Last time I looked at them they produced complete garbage for AArch64, clang was sporting its own fun implementation and the feedback I got from LLVM devs was "don't use". So for now variadics are disabled in Crystal for AArch64.

maxfierke · 2021-02-04T15:01:26Z

Yeah we have the va_arg primitive disabled for AArch64, but we do still the LLVM intrinsics for va_start and va_end (though maybe not actively 😄 ) Not sure if it's been fixed upstream or not.

straight-shoota added kind:feature platform:aarch64 platform:darwin labels Dec 10, 2020

Blacksmoke16 mentioned this issue Jan 11, 2021

Can't install on Apple M1 Silicon #10237

Closed

maxfierke mentioned this issue Jan 28, 2021

[req] Apple M1 Silicon support maxfierke/mstrap#21

Closed

6 tasks

maxfierke mentioned this issue Feb 1, 2021

Add aarch64 support for macOS/darwin targets #10348

Merged

bcardiff closed this as completed in #10348 Feb 12, 2021

Xanders mentioned this issue Mar 30, 2021

Apple Silicon support Xanders/make-help#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Aarch64 support for macOS #10066

Add Aarch64 support for macOS #10066

RomainFranceschini commented Dec 10, 2020 •

edited

Loading

jhass commented Dec 10, 2020

n-rodriguez commented Dec 10, 2020

RomainFranceschini commented Dec 10, 2020

jhass commented Dec 10, 2020

n-rodriguez commented Dec 10, 2020

jhass commented Dec 10, 2020

RomainFranceschini commented Dec 10, 2020

bcardiff commented Dec 10, 2020

jhass commented Dec 10, 2020 •

edited

Loading

RomainFranceschini commented Dec 16, 2020

bcardiff commented Dec 21, 2020

maxfierke commented Jan 28, 2021 •

edited

Loading

bcardiff commented Jan 28, 2021

maxfierke commented Jan 28, 2021

maxfierke commented Jan 29, 2021

RomainFranceschini commented Jan 29, 2021

maxfierke commented Jan 29, 2021 •

edited

Loading

RomainFranceschini commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

alin23 commented Jan 30, 2021

RomainFranceschini commented Jan 31, 2021

maxfierke commented Jan 31, 2021

jhass commented Feb 1, 2021

maxfierke commented Feb 4, 2021

jhass commented Feb 4, 2021

maxfierke commented Feb 4, 2021

Add Aarch64 support for macOS #10066

Add Aarch64 support for macOS #10066

Comments

RomainFranceschini commented Dec 10, 2020 • edited Loading

Cross-compiling

jhass commented Dec 10, 2020

n-rodriguez commented Dec 10, 2020

RomainFranceschini commented Dec 10, 2020

jhass commented Dec 10, 2020

n-rodriguez commented Dec 10, 2020

jhass commented Dec 10, 2020

RomainFranceschini commented Dec 10, 2020

bcardiff commented Dec 10, 2020

jhass commented Dec 10, 2020 • edited Loading

RomainFranceschini commented Dec 16, 2020

bcardiff commented Dec 21, 2020

maxfierke commented Jan 28, 2021 • edited Loading

Environment

Compiling libcrystal.a

Compiling bdw-gc

Compiling Crystal for ARM

bcardiff commented Jan 28, 2021

maxfierke commented Jan 28, 2021

maxfierke commented Jan 29, 2021

RomainFranceschini commented Jan 29, 2021

maxfierke commented Jan 29, 2021 • edited Loading

RomainFranceschini commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

bcardiff commented Jan 29, 2021

maxfierke commented Jan 29, 2021

alin23 commented Jan 30, 2021

RomainFranceschini commented Jan 31, 2021

maxfierke commented Jan 31, 2021

jhass commented Feb 1, 2021

maxfierke commented Feb 4, 2021

jhass commented Feb 4, 2021

maxfierke commented Feb 4, 2021

RomainFranceschini commented Dec 10, 2020 •

edited

Loading

jhass commented Dec 10, 2020 •

edited

Loading

maxfierke commented Jan 28, 2021 •

edited

Loading

maxfierke commented Jan 29, 2021 •

edited

Loading