How to run / develop for Feast on M1 Macs

[woop]

Overview

This guide summarizes how both end users and Feast developers can get their M1 Macs set up to use or develop for Feast. pyenv is used in this guide, but isn't absolutely necessary.

Installation Guide

1. Install pyenv

https://github.com/pyenv/pyenv#homebrew-in-macos

2. Install dependencies

brew install xz protobuf openssl zlib

3. Install Python
   (only tested with 3.8.12)

CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.8.12

4. Upgrade Pip and install dependencies

pip install --upgrade pip
pip install cryptography -U

export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"

You can stop here if you are just a Feast user (not a developer). At this point you should be able to pip install feast. Steps (5) and (6) are for developers only

5. Build and install Feast repository (run from Feast project root)

export LDFLAGS="-L$(brew --prefix freetds)/lib -L$(brew --prefix protobuf)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix openssl)/lib"
export CPPFLAGS="-I$(brew --prefix protobuf)/include -I$(brew --prefix zlib)/include"
export MYSQLCLIENT_LDFLAGS=$(pkg-config --libs mysqlclient)
export MYSQLCLIENT_CFLAGS=$(pkg-config --cflags mysqlclient)
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

pip install pip-tools
brew install mysql
make install-python-ci-dependencies PYTHON=3.8

6. Run Feast tests (all tests should pass)

FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 sdk/python/tests

Problems & Solutions

Problem

Lack of _ctypes

Solution

pyenv/pyenv#1768 (comment)

Problem

Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

Solution

pip install --upgrade pip
pip install cryptography -U

Problem

distutils.errors.DistutilsExecError: command 'clang' failed with exit status 1

Solution

export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Problem

ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/opt/homebrew/Caskroom/miniforge/base/envs/intel_py39/lib/python3.9/site-packages/google/protobuf/pyext/__init__.py)

Solution

brew install protobuf
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# You need
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"

Problem

ModuleNotFoundError: No module named '_lzma'

Solution

brew install xz

CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.9.4

[MattDelac]

Thanks @woop, it worked like expected.

Just needed to reinstall protobuf as you explained

export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"

[startakovsky]

Hey @woop using my brand new macbook pro here, M1 chip running Monterey Mac OS. Thank you.

I can confirm this works also with python 3.10.1. Because my pip freeze was empty at the onset of this, I did simplify my protobuf installation to: pip install protobuf --install-option="--cpp_implementation".

[kevjumba]

Hi,
Also, people should use the Rosetta M1 Mac plugin.
I just finished the installation process for the M1 14" Macbook Pro on Mac OSX Monterey 12.1.

Here are the extra steps I had to perform.

Fixing Cryptography Installation

1. Add new dependency brew install rust
2. Add these exports before hand:

• export CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1"
• export LDFLAGS="-L/usr/local/opt/openssl/lib"
• export CPPFLAGS="-I/usr/local/opt/openssl/include"

3. I changed the installation instructions for cryptography to pip install cryptography==3.4.8 -U (This is the autogenerated version in the requirements from pip-compile.

Fixing grpc-io installation

1. Add these exports before running make.

• CFLAGS="-I/opt/homebrew/opt/openssl/include"
• LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

Fixing makefile error below

Error: Invalid value for '[SRC_FILES]...': Path 'requirements/py-ci-requirements.txt' does not exist. make: *** [install-python-ci-dependencies] Error 2

• Run export PYTHON=3.8

[kevjumba]

Another issue that I found with setup with java. The protobuf and protc-gen-grpc-java artifacts are not yet available for m1 arm mac osx. In order to fix the artifact not found error when you build make build-java, make test-java and make test-java-integration, change the two artifacts in java/datatypes/pom.xml to

<protocArtifact>
    com.google.protobuf:protoc:${protoc.version}:exe:osx-x86_64
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>
    io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:osx-x86_64
</pluginArtifact>

[expediamatt]

You might have to change backports-zoneinfo==0.2.1 to backports-zoneinfo==0.2.1;python_version<"3.9" in sdk/python/requirements/py3.8-ci-requirements.txt to make sure it doesn't grab an unworkable copy of backports-zoneinfo.

[dheerajoruganty]

In the most recent pip 24.0 release, the --install-option has been deprecated. As an alternative, you can use
--config-settings in place of --install-option.

Here's the modified the command accordingly:
pip install protobuf --force-reinstall --no-deps --config-settings="--build-option=--cpp_implementation"