Fix compilation on Apple Silicon #7714

adamretter · 2020-11-25T13:13:56Z

Closes - #7710

I tested this on an Apple DTK (Developer Transition Kit) with an Apple A12Z Bionic CPU and macOS Big Sur (11.0.1).

Previously the arm64 specific CRC optimisations were limited to Linux only OS... Well now Apple Silicon is also arm64 but runs macOS ;-)

jurmous · 2020-11-29T16:17:05Z

@adamretter I looked into the branch with my M1 machine. It compiled ok. I ran make check and it failed with the following output:

Computers / CPU cores / Max jobs to run
1:local / 8 / 8

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 1371s Left: 4389 AVG: 0.31s  local:8/162/100%/0.3s /bin/bash: line 1: 41486 Abort trap: 6           ./listener_test >&t/log-listener_test
ETA: 0s Left: 0 AVG: 0.41s  local:0/4551/100%/0.4s      
Seq	Host	Starttime	JobRuntime	Send	Receive	Exitval	Signal	Command
76	:	1606654824.876	     0.005	0	0	1	0	./block_cache_trace_analyzer_test >& t/log-block_cache_trace_analyzer_test
154	:	1606654844.230	     0.027	0	0	1	0	./io_tracer_parser_test >& t/log-io_tracer_parser_test
158	:	1606654844.668	     4.640	0	0	134	0	./listener_test >& t/log-listener_test
206	:	1606654857.916	     0.005	0	0	1	0	./trace_analyzer_test >& t/log-trace_analyzer_test
make[1]: *** [check_0] Error 1
make: *** [check] Error 2

The ones with Exit val 1 did fail because they could not find gflags.

Please install gflags to run trace_analyzer test

I tried to install gflags with a fresh checkout with make install and later brew install gflags but RocksDB make check did not pick it up. But could also be that I did something wrong.

The actual failing test was outputting this:

[==========] Running 14 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 14 tests from EventListenerTest
[ RUN      ] EventListenerTest.OnSingleDBCompactionTest
[       OK ] EventListenerTest.OnSingleDBCompactionTest (38 ms)
[ RUN      ] EventListenerTest.OnSingleDBFlushTest
[       OK ] EventListenerTest.OnSingleDBFlushTest (44 ms)
[ RUN      ] EventListenerTest.MultiCF
[       OK ] EventListenerTest.MultiCF (37 ms)
[ RUN      ] EventListenerTest.MultiDBMultiListeners
[       OK ] EventListenerTest.MultiDBMultiListeners (107 ms)
[ RUN      ] EventListenerTest.DisableBGCompaction
[       OK ] EventListenerTest.DisableBGCompaction (121 ms)
[ RUN      ] EventListenerTest.CompactionReasonLevel
[       OK ] EventListenerTest.CompactionReasonLevel (37 ms)
[ RUN      ] EventListenerTest.CompactionReasonUniversal
[       OK ] EventListenerTest.CompactionReasonUniversal (65 ms)
[ RUN      ] EventListenerTest.CompactionReasonFIFO
[       OK ] EventListenerTest.CompactionReasonFIFO (18 ms)
[ RUN      ] EventListenerTest.TableFileCreationListenersTest
[       OK ] EventListenerTest.TableFileCreationListenersTest (2025 ms)
[ RUN      ] EventListenerTest.MemTableSealedListenerTest
[       OK ] EventListenerTest.MemTableSealedListenerTest (14 ms)
[ RUN      ] EventListenerTest.ColumnFamilyHandleDeletionStartedListenerTest
[       OK ] EventListenerTest.ColumnFamilyHandleDeletionStartedListenerTest (14 ms)
[ RUN      ] EventListenerTest.BackgroundErrorListenerFailedFlushTest
[       OK ] EventListenerTest.BackgroundErrorListenerFailedFlushTest (1023 ms)
[ RUN      ] EventListenerTest.BackgroundErrorListenerFailedCompactionTest
[       OK ] EventListenerTest.BackgroundErrorListenerFailedCompactionTest (1019 ms)
[ RUN      ] EventListenerTest.OnFileOperationTest
db/listener_test.cc:1058: Failure
Expected: (info.duration.count()) > (0), actual: 0 vs 0
libc++abi.dylib: terminating with uncaught exception of type testing::internal::GoogleTestFailureException: db/listener_test.cc:1058: Failure
Expected: (info.duration.count()) > (0), actual: 0 vs 0

If you have any further questions you are always welcome to ask!

adamretter · 2020-11-29T16:43:46Z

@jurmous Excellent thanks for testing. Looks good :-)
I think we can get this merged first to fix compilation, and then address any failing tests.

hayesgm · 2020-12-02T03:27:16Z

Are there steps here to get this merged to master? Is there an area to focus on we can help with?

pdillinger

👍

facebook-github-bot

@pdillinger has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@pdillinger has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-12-04T19:47:10Z

@adamretter has updated the pull request. You must reimport the pull request before landing.

pdillinger · 2020-12-04T19:47:26Z

FB infra was choking on this diff. Hopefully works with a merge.

facebook-github-bot

@pdillinger has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-12-04T23:43:03Z

@pdillinger merged this pull request in ee4bd47.

This patch updates rust-rocksdb to correctly build on Apple Silicon (using all native tools). We include patches to rustdb found here: facebook/rocksdb#7714. Additionally, we upgrade the build script and rocksdb libraries to fix compilation. See issue here: rust-rocksdb#482

Summary: Closes - facebook#7710 I tested this on an Apple DTK (Developer Transition Kit) with an Apple A12Z Bionic CPU and macOS Big Sur (11.0.1). Previously the arm64 specific CRC optimisations were limited to Linux only OS... Well now Apple Silicon is also arm64 but runs macOS ;-) Pull Request resolved: facebook#7714 Reviewed By: ltamasi Differential Revision: D25287349 Pulled By: pdillinger fbshipit-source-id: 639b168bf0ac2652907531e9604936ac4974b577

1. Backport facebook#7714 from upstream. 2. `uname -m` on the new Apple Silicon Mac outputs `arm64`. The ARMv8 CRC check in the cmake file is changed. 3. Do not treat arm darwin as iOS. With this PR, ` cmake .. -DWITH_GFLAGS=OFF -DWITH_TESTS=OFF -DWITH_TOOLS=OFF` outputs: ``` -- The C compiler identification is AppleClang 12.0.0.12000032 -- The CXX compiler identification is AppleClang 12.0.0.12000032 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- The ASM compiler identification is Clang -- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- Found Git: /usr/bin/git (found version "2.24.3 (Apple Git-128)") -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER - Success -- Performing Test HAS_ARMV8_CRC -- Performing Test HAS_ARMV8_CRC - Success -- HAS_ARMV8_CRC yes -- Performing Test HAVE_SSE42 -- Performing Test HAVE_SSE42 - Failed -- Performing Test HAVE_THREAD_LOCAL -- Performing Test HAVE_THREAD_LOCAL - Success -- Enabling RTTI in Debug builds only (default) -- Performing Test HAVE_FALLOCATE -- Performing Test HAVE_FALLOCATE - Failed -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE - Failed -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP - Failed -- Looking for malloc_usable_size -- Looking for malloc_usable_size - not found -- Looking for sched_getcpu -- Looking for sched_getcpu - not found -- Looking for getauxval -- Looking for getauxval - not found -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- JNI library is disabled -- Configuring done -- Generating done ``` And then, `make rocksdb -j4` succeeds. Signed-off-by: Yilin CHen <sticnarf@gmail.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>

Summary: Closes - facebook#7710 I tested this on an Apple DTK (Developer Transition Kit) with an Apple A12Z Bionic CPU and macOS Big Sur (11.0.1). Previously the arm64 specific CRC optimisations were limited to Linux only OS... Well now Apple Silicon is also arm64 but runs macOS ;-) Pull Request resolved: facebook#7714 Reviewed By: ltamasi Differential Revision: D25287349 Pulled By: pdillinger fbshipit-source-id: 639b168bf0ac2652907531e9604936ac4974b577

1. Backport facebook#7714 from upstream. 2. `uname -m` on the new Apple Silicon Mac outputs `arm64`. The ARMv8 CRC check in the cmake file is changed. 3. Do not treat arm darwin as iOS. With this PR, ` cmake .. -DWITH_GFLAGS=OFF -DWITH_TESTS=OFF -DWITH_TOOLS=OFF` outputs: ``` -- The C compiler identification is AppleClang 12.0.0.12000032 -- The CXX compiler identification is AppleClang 12.0.0.12000032 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- The ASM compiler identification is Clang -- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- Found Git: /usr/bin/git (found version "2.24.3 (Apple Git-128)") -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER - Success -- Performing Test HAS_ARMV8_CRC -- Performing Test HAS_ARMV8_CRC - Success -- HAS_ARMV8_CRC yes -- Performing Test HAVE_SSE42 -- Performing Test HAVE_SSE42 - Failed -- Performing Test HAVE_THREAD_LOCAL -- Performing Test HAVE_THREAD_LOCAL - Success -- Enabling RTTI in Debug builds only (default) -- Performing Test HAVE_FALLOCATE -- Performing Test HAVE_FALLOCATE - Failed -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE - Failed -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP - Failed -- Looking for malloc_usable_size -- Looking for malloc_usable_size - not found -- Looking for sched_getcpu -- Looking for sched_getcpu - not found -- Looking for getauxval -- Looking for getauxval - not found -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- JNI library is disabled -- Configuring done -- Generating done ``` And then, `make rocksdb -j4` succeeds. Signed-off-by: Yilin CHen <sticnarf@gmail.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>

In order to build on Apple Silicon (e.g. M1 processors) we need RocksDB v6.16.3 or higher. facebook/rocksdb#7714

adamretter added mac arm labels Nov 25, 2020

adamretter requested review from pdillinger and ramvadiv November 25, 2020 13:13

facebook-github-bot added the CLA Signed label Nov 25, 2020

Fix compilation on Apple Silicon (Tested on Apple DTK (A12Z))

f8f5b58

adamretter force-pushed the hotfix/apple-silicon-build branch from 0b7ef20 to f8f5b58 Compare November 25, 2020 15:02

adamretter added 3 commits November 25, 2020 16:07

Try and fix the Circle x64 builds

35d0d99

fixup! Try and fix the Circle x64 builds

be56a22

fixup! Try and fix the Circle x64 builds

67c9f16

adamretter force-pushed the hotfix/apple-silicon-build branch from f57f19c to 67c9f16 Compare November 25, 2020 15:28

hayesgm mentioned this pull request Nov 25, 2020

Failed to build on Apple Silicon rust-rocksdb/rust-rocksdb#482

Closed

valeriansaliou mentioned this pull request Nov 26, 2020

Fails to build on Apple Silicon valeriansaliou/sonic#234

Closed

hayesgm mentioned this pull request Nov 28, 2020

Fix for Apple Silicon rust-rocksdb/rust-rocksdb#483

Closed

adamretter mentioned this pull request Nov 28, 2020

Java: Support Apple Silicon/M1 machines #7720

Closed

pdillinger approved these changes Dec 2, 2020

View reviewed changes

facebook-github-bot reviewed Dec 2, 2020

View reviewed changes

facebook-github-bot reviewed Dec 4, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into pr/7714

6cb21e4

facebook-github-bot reviewed Dec 4, 2020

View reviewed changes

facebook-github-bot closed this in ee4bd47 Dec 4, 2020

facebook-github-bot added the Merged label Dec 4, 2020

sticnarf mentioned this pull request Dec 10, 2020

Fix build on Apple Silicon Mac tikv/rocksdb#219

Merged

adamretter mentioned this pull request Dec 11, 2020

RocksDB fails to build on M1 Apple Silicon due to incorrect ifdefs #7710

Closed

ccampbell mentioned this pull request Jan 29, 2021

Build fails on ARM64 Mac cockroachdb/cockroach#59569

Closed

lonnylot mentioned this pull request Jan 30, 2021

Mac M1 build support proposal (minus rocksdb option) MariaDB/server#1743

Merged

jak-pan mentioned this pull request Feb 5, 2021

Consider Upgrading wasmtime paritytech/substrate#7989

Closed

wezrule mentioned this pull request Feb 24, 2021

Update RocksDB to support building on M1 Mac when a new stable release supports it nanocurrency/nano-node#3105

Open

changvvb mentioned this pull request Mar 12, 2021

Fix compilation on Apple Silicon (#7714) ClickHouse/rocksdb#5

Closed

siying mentioned this pull request Jun 1, 2021

Docker Apple M1 Error while trying to install in Docker #8349

Closed

intelliot added a commit to intelliot/rippled that referenced this pull request Jun 16, 2021

Update to RocksDB 6.20.3

cbebc71

In order to build on Apple Silicon (e.g. M1 processors) we need RocksDB v6.16.3 or higher. facebook/rocksdb#7714

intelliot mentioned this pull request Jun 16, 2021

Update to RocksDB 6.20.3 XRPLF/rippled#3870

Closed

1 task

gr0vity-dev mentioned this pull request Jan 10, 2023

Update RocksDB to support building on M1 Mac when a new stable release supports it gr0vity-dev/nano-node-tracker#189

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix compilation on Apple Silicon #7714

Fix compilation on Apple Silicon #7714

adamretter commented Nov 25, 2020

jurmous commented Nov 29, 2020 •

edited

Loading

adamretter commented Nov 29, 2020 •

edited

Loading

hayesgm commented Dec 2, 2020

pdillinger left a comment

facebook-github-bot left a comment

facebook-github-bot left a comment

facebook-github-bot commented Dec 4, 2020

pdillinger commented Dec 4, 2020

facebook-github-bot left a comment

facebook-github-bot commented Dec 4, 2020

Fix compilation on Apple Silicon #7714

Fix compilation on Apple Silicon #7714

Conversation

adamretter commented Nov 25, 2020

jurmous commented Nov 29, 2020 • edited Loading

adamretter commented Nov 29, 2020 • edited Loading

hayesgm commented Dec 2, 2020

pdillinger left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 4, 2020

pdillinger commented Dec 4, 2020

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 4, 2020

jurmous commented Nov 29, 2020 •

edited

Loading

adamretter commented Nov 29, 2020 •

edited

Loading