Mac: fix arm64 build

[sguo35]

Fixes the build to compile for arm64, but obviously nothing works yet. Current next step is to implement W^X which Apple enforces on arm64, then dealing with the virtual memory situation.

Build (make sure your brew is arm64):

brew install molten-vk vulkan-headers sdl2 nasm qt@5 ninja cmake glew ffmpeg
export Qt5_DIR=$(brew --prefix)/opt/qt5
export VULKAN_SDK=$(brew --prefix)/opt/molten-vk
# taken from build-mac.sh
ln -s "$VULKAN_SDK/lib/libMoltenVK.dylib" "$VULKAN_SDK/lib/libvulkan.dylib"
export VK_ICD_FILENAMES=$VULKAN_SDK/share/vulkan/icd.d/MoltenVK_icd.json

git clone git@github.com:sguo35/rpcs3.git
cd rpcs3
# also taken from build-mac.sh
sed -i '' "s/extern const double NSAppKitVersionNumber;/const double NSAppKitVersionNumber = 1343;/g" 3rdparty/hidapi/hidapi/mac/hid.c

mkdir build
cd build
cmake .. -DPNG_ARM_NEON=on -DUSE_ALSA=OFF -DUSE_PULSE=OFF -DUSE_AUDIOUNIT=ON -DUSE_NATIVE_INSTRUCTIONS=OFF -DUSE_SYSTEM_FFMPEG=on -DCMAKE_OSX_ARCHITECTURES="arm64" -DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM" -GNinja
ninja

# Apple enforces codesigning on Apple Silicon
codesign --force --deep --sign - bin/rpcs3.app/Contents/MacOS/rpcs3
# will fail verification on memory_commit
./bin/rpcs3.app/Contents/MacOS/rpcs3

[kd-11]

This patch will allow rpcs3 to actually load the GUI, but ofc nothing runs successfully.
kd-11@9b014a5#diff-37645c22f08af879d11df33551addfb65b4df9568bc3e02c956c7df5e020d759R273-R289
The #ifndef APPLE check should read #if !defined(APPLE) || defined(ARCH_X64) of course. The hacky set of patches there was from a few weeks ago when we were working on getting it working to see how far it would get.

[sguo35]

@Nekotekina @Megamouse fixed

@kd-11 On Apple Silicon you have to invalidate the icache after toggling permissions, also in the future if we want to distribute Mac binaries then we need to sign the binary -> we need to pass MAP_JIT in mmap calls, see Apple docs.

[kd-11]

@Nekotekina @Megamouse fixed

@kd-11 On Apple Silicon you have to invalidate the icache after toggling permissions, also in the future if we want to distribute Mac binaries then we need to sign the binary -> we need to pass MAP_JIT in mmap calls, see Apple docs.

I know all of that, honestly I'm hoping someone else handles it. This isn't a mac-first product afterall, the design needs to not break the other 3 operating systems. Do not read too much into hacky "get me something working quickly" code on the repo, I was merely pointing out why it doesn't get to the GUI, in case you want to verify that the build works.

[sguo35]

I know all of that, honestly I'm hoping someone else handles it. This isn't a mac-first product afterall, the design needs to not break the other 3 operating systems. Do not read too much into hacky "get me something working quickly" code on the repo, I was merely pointing out why it doesn't get to the GUI, in case you want to verify that the build works.

Ah, I thought you were saying to integrate that patch, my bad. I was planning on incrementally putting up PRs for each issue I ran into as I fixed stuff so I expected the gui to fail to start.