Apple M1 build

[magnumripper]

I got my hands on a Macbook M1 for a while and thought I'd sort out JtR on it. This issue will contains a bunch of notes and likely end up with a PR or two, if needed.

So, starting with a 100% pristine Macbook. Before anything else, I installed xcode and command-line tools. After that (maybe before it as well, I dunno) I had git(1) so could clone the repo. Then,

First try: Just run ./configure and see what happens. As expected, it failed due to lack of OpenSSL.

Second try: Install Homebrew OpenSSL and only that. This time, (passing LDFLAGS and CPPFLAGS as recommended from Homebrew) we passed configure stage but build failed.

It appears configure detected ARM and 64-bit but used arm32le.h and NEON as opposed to arm64le.h and ASIMD.

https://en.wikipedia.org/wiki/Mac_transition_to_Apple_Silicon

[solardiz]

Thank you, @magnumripper!

This is only subtly related, but FWIW as far as I'm aware (from user feedback) our builds made on/for Intel Macs with SSE4 do work on M1 via Rosetta 2, however those with AVX2 don't (print the "AVX2 is required" message).

[magnumripper]

FWIW as far as I'm aware (from user feedback) our builds made on/for Intel Macs with SSE4 do work on M1 via Rosetta 2

That's interesting, I'll try things like that as well (and comparing benchmarks).

I haven't googled anywhere near what I need to yet, but I was also under the impression even running an ARM build wouldn't quite be "native" Silicon M1 (but "more native" than intel binaries). Lots of things to learn here.

[magnumripper]

Simply re-linking arch.h to arm64le.h didn't help at all - same build error.

[magnumripper]

Trying --disable-simd for a starter. That works as-is.

Using arm32le.h, I get DEScrypt speeds of 5334K (many salts) and 5060K (one salt). NT-opencl benchmarks at 958253K.

Using arm64le.h, nothing much changed and I realized even with arm32le.h we got 64-bit bitslice. Not sure exactly what's going on there.

I also noticed OpenCL figures vary a fair bit between runs, even with same auto-tune results.

Umm I also see now that I didn't get an OpenMP build (native clang compiler). Even given that, the DEScrypt figures are mediocre.

[magnumripper]

the DEScrypt figures are mediocre.

Seems I was wrong. Disabling SIMD and OpenMP on my Intel Macbook, I only get 3355K "many salts" and 3233K "one salt"

[magnumripper]

I haven't googled anywhere near what I need to yet, but I was also under the impression even running an ARM build wouldn't quite be "native" Silicon M1 (but "more native" than intel binaries). Lots of things to learn here.

The native gcc is just arm64 though, as is Homebrew's real gcc (v10).

Using the latter, all build problems went away and I got OpenMP, and I ended up with DES 128/128 NEON (still not sure why it's not ASIMD though) and speeds for 8 threads are 17704K many salts, 16530K one salt.

Manually relinking arch.h to arm64le.h fixes the SIMD confusion, rendering DES 128/128 ASIMD and DEScrypt speeds of 37060K (many salts) and 30909K (one salt).

My 8 core, 16 threads 2.3 GHz intel i9 Macbook gets 88227K (many salts) and 57212K (one salt). And a whole lot better NT speeds but it does have a "discrete" AMD Vega GPU so it's not comparable. Using the "Intel UHD Graphics 630" gets me 454800K for NT-opencl, just half the speed of the M1.

[magnumripper]

So far, just this needed

Install Xcode via App Store, then install "command-line tools":

$ xcode-select --install

Clone repo:

$ mkdir ~/src
$ cd ~/src
$ git clone https://github.com/openwall/john.git

Install Homebrew, then install OpenSSL and gcc with it:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> ~/.zprofile
$ eval $(/opt/homebrew/bin/brew shellenv)
$ brew install openssl gcc

Build JtR

$ cd ~/src/john/src
$ ./configure CC=gcc-10 LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"  CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include" && make -s clean && make -sj8

Until #4587 is merged, you also need ln -fs arm64le.h arch.h right after ./configure and before make in order to get proper SIMD speeds.

[magnumripper]

Native gcc (clang)

$ gcc -dM -E -x c /dev/null | grep -iE 'neon|simd|aarch'
#define __AARCH64EL__ 1
#define __AARCH64_SIMD__ 1
#define __ARM_NEON 1
#define __ARM_NEON_FP 0xE
#define __ARM_NEON__ 1
#define __aarch64__ 1

So it defaults to 64-bit and to having support for NEON and/or ASIMD. It doesn't seem to need -mfpu=neon or march=armv8-a+simd but it doesn't complain with them either. On a side note it seems to support -m32 and that turns off even NEON.

Homebrew gcc-10

$ /opt/homebrew/bin/gcc-10 -dM -E -x c /dev/null | grep -iE 'neon|simd|aarch'
#define __AARCH64_CMODEL_SMALL__ 1
#define __aarch64__ 1
#define __AARCH64EL__ 1
#define __ARM_NEON 1

$ /opt/homebrew/bin/gcc-10 -dM -E -x c /dev/null -mfpu=neon | grep -iE 'neon|simd|aarch'
gcc-10: error: unrecognized command-line option '-mfpu=neon'

$ /opt/homebrew/bin/gcc-10 -dM -E -x c /dev/null -march=armv8-a+simd | grep -iE 'neon|simd|aarch'
#define __AARCH64_CMODEL_SMALL__ 1
#define __aarch64__ 1
#define __AARCH64EL__ 1
#define __ARM_NEON 1

It defaults to 64-bit and doesn't even allow -m32. It seems to support NEON without any option but doesn't support -mfpu=neon. It doesn't complain with -march=armv8-a+simd but that option doesn't change any macro it defines (even bar that grep shown above).

I'm not yet sure if there's something in our arm64le.h that actually triggers ASIMD and makes those build twice as fast, or if it's the -march=armv8-a+simd option. Perhaps a combo of both?

[magnumripper]

I'm not yet sure if there's something in our arm64le.h that actually triggers ASIMD and makes those build twice as fast, or if it's the -march=armv8-a+simd option. Perhaps a combo of both?

From what I can tell, us calling it "ASIMD" instead of "NEON" is blindly from that arch.h change, but the speed difference might be from the -march=armv8-a+simd option. I need to verify that.

Our platform-agnostic intrinsics triggers on __ARM_NEON || __aarch64__ everywhere I can see so it will trigger on any of the compilers without any extra option. But the native compiler fails to build it then...

[magnumripper]

The native gcc also doesn't change its macro output with -mfpu=neon at all. But when using -march=armv8-a+simd the following defines go away compared to default output:

$ diff native-bare.txt native-march.txt | grep \<
< #define __ARM_ARCH_8_3__ 1
< #define __ARM_FEATURE_COMPLEX 1
< #define __ARM_FEATURE_CRC32 1
< #define __ARM_FEATURE_CRYPTO 1
< #define __ARM_FEATURE_FP16_SCALAR_ARITHMETIC 1
< #define __ARM_FEATURE_FP16_VECTOR_ARITHMETIC 1
< #define __ARM_FEATURE_JCVT 1
< #define __ARM_FEATURE_QRDMX 1

[magnumripper]

Regardless of which arch.h is used, here's the first error for native gcc

simd-intrinsics.c:234:2: error: argument value 39 is outside the valid range [0, 31]
        MD5_STEP(MD5_F, a, b, c, d, 0, 0xd76aa478, 7)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
simd-intrinsics.c:90:16: note: expanded from macro 'MD5_STEP'
        a[i] = vroti_epi32( a[i], (s) );            \
               ^~~~~~~~~~~~~~~~~~~~~~~~
./pseudo_intrinsics.h:70:61: note: expanded from macro 'vroti_epi32'
                                         (vtype)vsriq_n_u32(vshlq_n_u32((x).v32, 32 + (i)), (x).v32, -(i)))
                                                ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/arm_neon.h:23561:24: note: 
      expanded from macro 'vshlq_n_u32'
  __ret = (uint32x4_t) __builtin_neon_vshlq_n_v((int8x16_t)__s0, __p1, 50); \
                       ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/arm_neon.h:25162:21: note: 
      expanded from macro 'vsriq_n_u32'
  uint32x4_t __s0 = __p0; \
                    ^~~~

[magnumripper]

Back to real gcc: I now confirmed using -march=armv8-a+simd make no difference at all. The nearly 2x speed difference come from differences between arm32le.h and arm64le.h.

Things that differ for DES is DES_MASK (0 for arm32, 1 for arm64) and DES_COPY (the opposite values). And perhaps most importantly DES_BS (1 for arm32 and 3 for arm64) and DES_BS_VECTOR (4 for arm32 and 2 for arm64). I shouldn't even give more thought about this - obviously we should use arm64le.h and be done with it. Perhaps the current #4587 is fine as-is then.

[magnumripper]

On a side note, Apple seem to reuse their universal binary thing:

$ file /bin/sh
/bin/sh: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/bin/sh (for architecture x86_64):	Mach-O 64-bit executable x86_64
/bin/sh (for architecture arm64e):	Mach-O 64-bit executable arm64e

We could consider that when distributing John binaries.

BTW the john binary created with gcc-10 isn't arm64e:

$ file ../run/john
../run/john: Mach-O 64-bit executable arm64

It's just arm64. Same goes for a native gcc build:

$ file ../run/john
../run/john: Mach-O 64-bit executable arm64

[magnumripper]

From https://en.wikipedia.org/wiki/Mach-O

ARMv8-A A64 also known as arm64 (for iPhone 5S to iPhone X), ARMv8.3-A A64 also known as arm64e (for iPhone XS and MacBook Air (M1))

See also https://en.wikipedia.org/wiki/AArch64#ARMv8-A

[claudioandre-br]

BTW the john binary created with gcc-10 isn't arm64e:

Have you tried things like as -arch arm64e "$@" clang --target=arm64e-apple-darwin20 "$@", ld -arch arm64e "$@", ...

Web are talking about this. PS (not clear if you are able to compile using clang).

[magnumripper]

Snorting the gcc-10 man page, there are options applicable for this arch such as -march=armv8.3-a or -march=armv8.3-a+simd (why would that be needed though and not default?) that I need to test for any difference in performance.

Using -march=armv8.3-a to Homebrew's gcc-10 adds these compiler macros:

$ diff gcc10-bare.txt gcc10-83.txt | grep -E '^[<>]'
> #define __ARM_FEATURE_ATOMICS 1
> #define __ARM_FEATURE_JCVT 1
> #define __ARM_FEATURE_QRDMX 1
> #define __ARM_FEATURE_COMPLEX 1
> #define __ARM_FEATURE_CRC32 1

Using -march=armv8.3-a+simd with gcc-10 just adds the same ones.

These were some of the ones that were dropped when using native compiler with -march=armv8-a+simd which now makes sense to me - it actually disabled stuff from that compiler's default... Very obvious in hindsight.

Anyway my first tests (Only comparing single-run benchmarks) of -march=armv8.3-a+simd compared to defaults showed a slight boost but it might well be random noise - I need to test that more thoroughly.

Anyway, what made me do all the above - the arm64 vs arm64e did not change. We still produce "Mach-O 64-bit executable arm64".

On some other note (of now many) I believe this CPU can access unaligned data - we might want to tweak some #ifdef's such as the one disabling the RAR format.

I got the idea to try this:

$ grep -i align gcc10-bare.txt 
#define __ARM_ALIGN_MAX_PWR 28
#define __BIGGEST_ALIGNMENT__ 16
#define __ARM_FEATURE_UNALIGNED 1
#define __ARM_ALIGN_MAX_STACK_PWR 16

Score! So I reckon we can look for __ARM_FEATURE_UNALIGNED to set/override our ARCH_ALLOWS_UNALIGNED for ARM.

[magnumripper]

BTW the john binary created with gcc-10 isn't arm64e:

Have you tried things like as -arch arm64e "$@" clang --target=arm64e-apple-darwin20 "$@", ld -arch arm64e "$@", ...

Web are talking about this. PS (not clear if you are able to compile using clang).

Thanks, that'll be yet another thing to my to-do list :-)

[magnumripper]

Score! So I reckon we can look for __ARM_FEATURE_UNALIGNED to set/override our ARCH_ALLOWS_UNALIGNED for ARM.

Tried the following

diff --git a/src/arm32le.h b/src/arm32le.h
index 9a8ff9995..a0c60c0d2 100644
--- a/src/arm32le.h
+++ b/src/arm32le.h
@@ -27,7 +27,7 @@
 #define ARCH_INT_GT_32                 0
 #endif
 
-#define ARCH_ALLOWS_UNALIGNED          0
+#define ARCH_ALLOWS_UNALIGNED          __ARM_FEATURE_UNALIGNED
 #define ARCH_INDEX(x)                  ((unsigned int)(unsigned char)(x))
 
 #define CPU_DETECT                     0
diff --git a/src/arm64le.h b/src/arm64le.h
index a916cc053..63b1ed932 100644
--- a/src/arm64le.h
+++ b/src/arm64le.h
@@ -28,7 +28,7 @@
 #define ARCH_INT_GT_32                 0
 #endif
 
-#define ARCH_ALLOWS_UNALIGNED          0
+#define ARCH_ALLOWS_UNALIGNED          __ARM_FEATURE_UNALIGNED
 #define ARCH_INDEX(x)                  ((unsigned int)(unsigned char)(x))
 
 #define CPU_DETECT                     0

The RAR format now compiles, and works fine.

$ ../run/john -test -form:rar
Will run 8 OpenMP threads
Benchmarking: rar, RAR3 (length 5) [SHA1 128/128 ASIMD 4x AES]... (8xOMP) DONE
Raw:	592 c/s real, 98.7 c/s virtual

Just half of intel speed though, comparing M1 @3.2 Ghz (well it doesn't say! Google indicates it might be 3.2 GHz) against intel core i9 @2.4 Ghz and same number of (real) cores. Come to think of it, a core or two might have been totally hogged by other things during that benchmark though. Anyway, new to-do entry would be to test penalty for unaligned...

Oh and BTW I was comparing 128-bit wide ASIMD against 256 bit AVX2...

[magnumripper]

Mixing all sorts of info in this issue, this M1 chip allegedly has four high-efficiency (as in low-power) cores and four high-performance cores.
https://www.macworld.com/article/3596796/apple-m1-chip-specs-performance-features-power-efficiency.html

Running just one thread

Benchmarking: rar, RAR3 (length 5) [SHA1 128/128 ASIMD 4x AES]... DONE
Raw:	132 c/s real, 132 c/s virtual

versus

Benchmarking: rar, RAR3 (length 5) [SHA1 256/256 AVX2 8x AES]... DONE
Raw:	154 c/s real, 154 c/s virtual

Clock for clock and width for width it's not too bad, I think. But I might have screwed that calculation up royally.

[magnumripper]

To-do: Have configure try -march=armv8.3-a+simd as well. Perhaps also --target=arm64e-apple-darwin20, I need to try that.

[magnumripper]

Regardless of which arch.h is used, here's the first error for native gcc

simd-intrinsics.c:234:2: error: argument value 39 is outside the valid range [0, 31]
        MD5_STEP(MD5_F, a, b, c, d, 0, 0xd76aa478, 7)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
simd-intrinsics.c:90:16: note: expanded from macro 'MD5_STEP'
        a[i] = vroti_epi32( a[i], (s) );            \
               ^~~~~~~~~~~~~~~~~~~~~~~~

Trivial fix:

diff --git a/src/pseudo_intrinsics.h b/src/pseudo_intrinsics.h
index 5fabbdbb8..450e2477b 100644
--- a/src/pseudo_intrinsics.h
+++ b/src/pseudo_intrinsics.h
@@ -66,10 +66,10 @@ typedef union {
 #define VLOADU_EMULATED         1
 #define vor(x, y)               (vtype)vorrq_u32((x).v32, (y).v32)
 #define vorn(x, y)              (vtype)vornq_u32((x).v32, (y).v32)
-#define vroti_epi32(x, i)       (i > 0 ? (vtype)vsliq_n_u32(vshrq_n_u32((x).v32, 32 - (i)), (x).v32, i) : \
-                                         (vtype)vsriq_n_u32(vshlq_n_u32((x).v32, 32 + (i)), (x).v32, -(i)))
-#define vroti_epi64(x, i)       (i > 0 ? (vtype)vsliq_n_u64(vshrq_n_u64((x).v64, 64 - (i)), (x).v64, i) : \
-                                         (vtype)vsriq_n_u64(vshlq_n_u64((x).v64, 64 + (i)), (x).v64, -(i)))
+#define vroti_epi32(x, i)       (i > 0 ? (vtype)vsliq_n_u32(vshrq_n_u32((x).v32, 32 - ((i) & 31)), (x).v32, (i) & 31) : \
+                                 (vtype)vsriq_n_u32(vshlq_n_u32((x).v32, (32 + (i)) & 31), (x).v32, (-(i)) & 31))
+#define vroti_epi64(x, i)       (i > 0 ? (vtype)vsliq_n_u64(vshrq_n_u64((x).v64, 64 - ((i) & 63)), (x).v64, (i) & 63) : \
+                                 (vtype)vsriq_n_u64(vshlq_n_u64((x).v64, (64 + (i)) & 63), (x).v64, (-(i)) & 63))
 #define vroti16_epi32           vroti_epi32
 #define vset1_epi32(i)          (vtype)vdupq_n_u32(i)
 #define vset1_epi64(i)          (vtype)vdupq_n_u64(i)

This solves all problems with native gcc.

Native gcc (doesn't support OpenMP):

Benchmarking: descrypt, traditional crypt(3) [DES 128/128 ASIMD]... DONE
Many salts:	9358K c/s real, 9358K c/s virtual
Only one salt:	8727K c/s real, 8727K c/s virtual

Benchmarking: NT [MD4 128/128 ASIMD 4x2]... DONE
Raw:	47300K c/s real, 47065K c/s virtual

Benchmarking: Raw-MD5 [MD5 128/128 ASIMD 4x2]... DONE
Raw:	31346K c/s real, 31346K c/s virtual

Benchmarking: Raw-SHA1 [SHA1 128/128 ASIMD 4x]... DONE
Raw:	15445K c/s real, 15445K c/s virtual

Benchmarking: Raw-SHA256 [SHA256 128/128 ASIMD 4x]... DONE
Raw:	12025K c/s real, 12025K c/s virtual

Benchmarking: Raw-SHA512 [SHA512 128/128 ASIMD 2x]... DONE
Raw:	5385K c/s real, 5385K c/s virtual

Real gcc (with OpenMP disabled):

Benchmarking: descrypt, traditional crypt(3) [DES 128/128 ASIMD]... DONE
Many salts:	10411K c/s real, 10359K c/s virtual
Only one salt:	9687K c/s real, 9736K c/s virtual

Benchmarking: NT [MD4 128/128 ASIMD 4x2]... DONE
Raw:	63540K c/s real, 63858K c/s virtual

Benchmarking: Raw-MD5 [MD5 128/128 ASIMD 4x2]... DONE
Raw:	34588K c/s real, 34588K c/s virtual

Benchmarking: Raw-SHA1 [SHA1 128/128 ASIMD 4x]... DONE
Raw:	21388K c/s real, 21388K c/s virtual

Benchmarking: Raw-SHA256 [SHA256 128/128 ASIMD 4x]... DONE
Raw:	13788K c/s real, 13788K c/s virtual

Benchmarking: Raw-SHA512 [SHA512 128/128 ASIMD 2x]... DONE
Raw:	5530K c/s real, 5530K c/s virtual

[magnumripper]

BTW the john binary created with gcc-10 isn't arm64e:

Have you tried things like as -arch arm64e "$@" clang --target=arm64e-apple-darwin20 "$@", ld -arch arm64e "$@", ...

Tried it now, it fails because Homebrew's OpenSSL is just arm64:

ld: warning: ignoring file /opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib, building for macOS-arm64e but attempting to link with file built for macOS-arm64

I could compile OpenSSL too myself of course, but I don't think there will be much (if any) difference.

Anyway, basically I just manually set these lines in the generated Makefile:

CC = clang --target=arm64e-apple-darwin20
AR = ar
AS = as -arch arm64e
LD = ld -arch arm64e
CPP = $(CC)

[magnumripper]

And just for reference, John 1.8.0.9-jumbo-1-bleeding SSE-4.1 benchmark, using Rosetta2:

$ ./john -test -form:descrypt
Will run 8 OpenMP threads
Benchmarking: descrypt, traditional crypt(3) [DES 128/128 SSE2-16]... (8xOMP) DONE
Many salts:	27057K c/s real, 5070K c/s virtual
Only one salt:	23494K c/s real, 4814K c/s virtual

Native (current bleeding)

Will run 8 OpenMP threads
Benchmarking: descrypt, traditional crypt(3) [DES 128/128 ASIMD]... (8xOMP) DONE
Many salts:	37194K c/s real, 7446K c/s virtual
Only one salt:	32292K c/s real, 6644K c/s virtual