Error building cpymad on M1 Apple Silicon - "-march=core2" not supported

[GuillaumeRD]

I followed the new installation procedure for Apple Silicon (M1 in this case) and successfully built MAD-X, however when running:

python setup.py build_ext -lblas -llapack

I get the following error message:

running build_ext
building 'cpymad.libmadx' extension
clang-14 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/[$username]/miniforge3/envs/xsuite-arm/include -arch arm64 -fPIC -O2 -isystem /Users/[$username]/miniforge3/envs/xsuite-arm/include -arch arm64 -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/[$username]/miniforge3/include -D_FORTIFY_SOURCE=2 -isystem /Users/[$username]/miniforge3/include -I/[$MADXDIR]/include -I/Users/[$username]/miniforge3/envs/xsuite-arm/include/python3.10 -c src/cpymad/libmadx.c -o build/temp.macosx-11.0-arm64-cpython-310/src/cpymad/libmadx.o -std=gnu99
clang-14: error: the clang compiler does not support '-march=core2'
error: command '/Users/[$username]/miniforge3/envs/xsuite-arm/bin/clang-14' failed with exit code 1

Some context:

• [$username] and [$MADXDIR] are generic names I am using here instead of the actual paths on the disk;
• I have tried different compilers as CC (see installation instructions): gcc, clang, and clang-14 - but they all return the same error message above;
• I put in bold a part of the error message that I find weird: if the call to clang knows that the architecture of the machine is Silicon as seen with the -arch arm64, why is the compilation still calling for the -march and -mtune flags? Any chance I can edit those out somewhere and replace them both with -mcpu=apple-m1 as I was able to do when building MAD-X? So far I have not been able to find where those flags are available...

[coldfix]

Hey,

I followed the new installation procedure for Apple Silicon

you mean the one outlined in #114?

I'm not good with Apple, but regarding the last part of the question:

Any chance I can edit those out somewhere and replace them both with -mcpu=apple-m1 as I was able to do when building MAD-X?

You can try simply invoking the compiler manually, just copy the whole line and replace what you think needs to be changed. If you succeed in building the .so file and place in the correct location under build/ as given by the command line above, setup.py will automatically pick up the file without trying to rebuild (if it's newer than the .c file and so on). That's also the procedure we're currently using for makig the windows wheels.

I'm currently also working on getting M1 wheels via cross-compilation as recommended in #113.

[GuillaumeRD]

I actually followed the instructions listed here:

https://xsuite.readthedocs.io/en/latest/installation.html#install-on-apple-silicon

along with what can be found here:

https://github.com/hibtc/cpymad/blob/master/doc/installation/macos.rst

since the mention of running pip install -U cython wheel setuptools delocate could not be found at the first link and made one of the scripts look for libmadx.c - which is not available through git clone, only libmadx.pyx can be found.

I tried your suggestion above and ran the compiler manually: I got 6 warning messages related to CYTHON_FALLTHROUGH and it generated a libmadx.o file, not libmadx.so. I moved it into the temporary build/ directory regardless, but upon running setup.py it still gives me the -march error message as it seems to not see the file I just built...

[coldfix]

I'm not sure I can help here, however I have succeeded in cross-compiling for M1 now. You may try the M1 wheels that will be arriving shortly with cpymad 1.12.0.

The main thing that needed to be changed with respect to the x86_64 build was to change the relevant argument to

- -DCMAKE_OSX_ARCHITECTURES=x86_64
+ -DCMAKE_OSX_ARCHITECTURES=arm64

in the cmake command. Other than that, make to install a suitable gfortran compiler, set FC accordingly. In the GitHub Actions cross-compilation environment I also had to setup a bit of environment, but some of these settings may not be necessary for a native build on a well-configured platform:

sudo xcode-select -switch /Applications/Xcode_12.5.1.app
export SDKROOT="$(xcrun --show-sdk-path)"

export ARCHFLAGS="-arch arm64"
export _PYTHON_HOST_PLATFORM="macosx-11.0-arm64"
export MACOSX_DEPLOYMENT_TARGET="11.0"

LIBDIR="path to folder that contains libgfortran.so"
export LDFLAGS="-L$LIBDIR -Wl,-rpath,$LIBDIR"

Some care needs to be taken to use compatible versions of deployment target, xcode version and gfortran.

See also the updated github action starting here:

cpymad/.github/workflows/build.yml

Lines 185 to 223 in c8156cb

	# Select matching Xcode and SDK, see: https://xcodereleases.com/
	- run: |
	sudo xcode-select -switch /Applications/Xcode_12.5.1.app
	echo "SDKROOT=$(xcrun --show-sdk-path)" >> $GITHUB_ENV
	- name: Setup gfortran
	run: |
	set -xeo pipefail
	if [[ ${{ matrix.arch }} == "x86_64" ]]; then
	FC=gfortran-11
	LIBDIR=$(dirname $(find /usr/local/Cellar/gcc@11 -name libgfortran.dylib))
	echo "MACOSX_DEPLOYMENT_TARGET=10.14" >> $GITHUB_ENV
	elif [[ ${{ matrix.arch }} == "arm64" ]]; then
	# See: https://github.com/MacPython/gfortran-install
	name=gfortran-darwin-arm64-cross
	curl -L -O https://github.com/isuruf/gcc/releases/download/gcc-11.3.0-2/${name}.tar.gz
	sudo tar -f ${name}.tar.gz -C /opt -xzv
	rm ${name}.tar.gz
	FC=$(find /opt/${name}/bin -name "*-gfortran")
	LIBDIR=$(dirname $(find /opt/${name}/lib -name libgfortran.dylib))
	# See: https://github.com/pypa/cibuildwheel/discussions/997
	echo "ARCHFLAGS=-arch arm64" >> $GITHUB_ENV
	echo "_PYTHON_HOST_PLATFORM=macosx-11.0-arm64" >> $GITHUB_ENV
	echo "MACOSX_DEPLOYMENT_TARGET=11.0" >> $GITHUB_ENV
	fi
	LDFLAGS="-L$LIBDIR -Wl,-rpath,$LIBDIR"
	echo "FC=$FC" >> $GITHUB_ENV # for cmake
	echo "LDFLAGS=$LDFLAGS" >> $GITHUB_ENV # for setuptools
	echo "LIBDIR=$LIBDIR" >> $GITHUB_ENV # for delocate step
	- name: Build MAD-X
	if: steps.madx-build-cache.outputs.cache-hit != 'true'
	run: .github/build/macos/madx.sh src/MAD-X ${{ matrix.arch }}

I tried your suggestion above and ran the compiler manually: I got 6 warning messages related to CYTHON_FALLTHROUGH and it generated a libmadx.o file, not libmadx.so. I moved it into the temporary build/ directory regardless, but upon running setup.py it still gives me the -march error message as it seems to not see the file I just built...

If you take the compiler line from above and just modify the parts that need changing, it should place the .o file in the correct location (see -o argument) if it succeeds. An additional link command is required afterwards. Have you tried simply setting architecture to arm64 instead?

Anyway, this route is not really recommended if it can be avoided in any way. Otherwise, see how we do it for the windows build:

cpymad/.github/build/windows/cpymad.sh

Lines 64 to 86 in c8156cb

	gcc -mdll -O -Wall -flto $CFLAGS \
	-I$MADXDIR/include \
	-I$pythondir/include \
	-c src/cpymad/libmadx.c \
	-o $tempdir/libmadx.obj \
	-std=gnu99
	# Linking directly against the `pythonXX.dll` is the only way I found to
	# satisfy the linker in a conda python environment. The conventional
	# command line `-L$pythondir/libs -lpython$py_ver` used to work fine on
	# WinPython, but fails on conda with large number of complaints about
	# about undefined references, such as `__imp__Py_NoneStruct`,
	gcc -shared -s -flto \
	$tempdir/libmadx.obj \
	-L$MADXDIR/lib \
	-static \
	-lmadx -lDISTlib -lptc -lgc-lib \
	-lstdc++ -lgfortran -lquadmath \
	$pythondir/python$py_ver.dll \
	-o $libdir/libmadx$file_tag.pyd
	# Turn target python environment back on, see above:
	python setup.py bdist_wheel

[GuillaumeRD]

Thanks for the detailed reply; turns out that it had to do with the environment the conda install had set up: when checking on env, I noticed that I had to manually update the following variables

CFLAGS=-march=core2 -mtune=haswell [...]
CONDA_BACKUP_DEBUG_CFLAGS=-march=core2 -mtune=haswell [...]
DEBUG_CFLAGS=-march=core2 -mtune=haswell [...]
CONDA_BACKUP_CFLAGS=-march=core2 -mtune=haswell [...]

by replacing -march=core2 -mtune=haswell with -mcpu=apple-m1. I re-did the full cpymad install from scratch following your updated commands, and as soon as I changed the 4 CFLAGS above the installation proceeded flawlessly. Running the test via python -m pytest test returned 67 passed, 2 xpassed, 4 warnings.

[coldfix]

Ok, so I assume this is resolved (except maybe improved installation instructions)?