DBT cli throw memory exception for ffi.callback() on Mac with Apple Sillicon

[alyiwang]

Describe the bug

dbt run or dbt --version throw error Cannot allocate write+execute memory for ffi.callback().

Steps To Reproduce

On Macbook pro with M1 chip
brew tap fishtown-analytics/dbt
brew install dbt
add default profile.yml with connection to snowflake
create new project dbt init
dbt run with the two sample models, or simply dbt --version

Expected behavior

Should complete successfully, instead of throwing exception. The process works fine on Intel-based Macbook pro.

Screenshots and log output

> dbt run
Running with dbt=0.19.0
Found 3 models, 8 tests, 0 snapshots, 0 analyses, 143 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

The operating system you're using:
macOS Big Sur 11.2.2

The output of python --version:
3.8.7

Additional context

Add any other context about the problem here.

[jtcohen6]

Hey @alyiwang, thanks for the issue! We've been hearing about plenty of installation errors with M1 macs, though this is the first I'm seeing of this specific error. Unfortunately, I don't have access to the hardware to reproduce and debug this, so I'm operating on a mix of intuition and what I can find online.

It looks like this error is coming from pyOpenSSL (pyca/pyopenssl#873), which is a dependency of snowflake-connector-python. Other CLI tools that depend on pyOpenSSL are encountering the same stumbling block on M1 chips (e.g. Azure/azure-cli#16907). If this is indeed the cause, I don't think there's anything we can do until either pyOpenSSL finds a resolution or the Snowflake connector removes it as a dependency.

In the meantime, you could try:

• pip install dbt-snowflake (N.B. this requires you to use python 3.8, since the pinned version of the Snowflake connector does not yet support py39)
• Install dbt via Homebrew using Rosetta (SO post)

[mikelanzago]

Hey @alyiwang, I was having similar issues and I was able to work through it using the following:

• First, follow this link to alter your terminal of choice (I was using iterm) to https://alexslobodnik.medium.com/apple-m1-python-pandas-and-homebrew-20f14828ccc7 open using Rosetta
• Uninstall all versions of dbt through brew uninstall dbt if you already tried installing
• I performed the above until which dbt returned nothing (at this point I had multiple installs of dbt that I had to get rid of
• Once my system was rid of dbt, I shifted over to the python virtual environment route. Using Python 3.8, I ran pip install virtualenv and created a virtual environment. Link for more info on those https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
• After activating the environment I ran pip install dbt==0.18.0 and let it run it's course. Following installation I ran dbt --version while the environment was still active and it worked. It's worth noting that if you go this route of installation, the virtual environment must first be activated before running any dbt commands and deactivated when complete

[alyiwang]

Hi @jtcohen6 and @mikelanzago , thanks for your responses! I spent quite some time on this issue, and as you mentioned, it is a dependency issue with snowflake-connector-python, more specifically, it's some of it's dependencies, like pyOpenSSL and pyarrow, that's having compatibility issue with M1 chip. I was able to install latest pyOpenssl via pip, but the installation of snowflake-connector-python failed by pyarrow error. After many try and fail, I got pip install snowflake-connector-python (v2.4.1) to pass, which involves installing latest pyarrow through conda-forge https://anaconda.org/conda-forge/pyarrow which has osx-arm64 version. Then I realized dbt is fixed to an earlier version 2.3.5 of snowflake, which in turn requires an earlier version of pyarrow that doesn't have arm version anywhere.

To make the installation work, a couple changes will need to be made from dbt to:

1. allow python3.9
2. use latest version 2.4.1 of snowflake-connector-python
   @jtcohen6 do you think this can be put into the roadmap or feature request of DBT?

Meanwhile, I got the docker image version of DBT to work locally, even though I don't prefer this way of installation, this is the only thing working for me to unblock my work. I will definitely try out what Mike detailed later today. Thanks!

[jtcohen6]

To make the installation work, a couple changes will need to be made from dbt to:

1. allow python3.9
2. use latest version 2.4.1 of snowflake-connector-python
   @jtcohen6 do you think this can be put into the roadmap or feature request of DBT?

@alyiwang This is absolutely our plan for the next minor version of dbt (v0.20.0). dbt-core already supports python3.9, it's just a matter of upgrading our pin on the Snowflake connector.

[bastienboutonnet]

Just ran into the issue as well today on my new work computer. If there is anything I can do to help out I'd be happy to chip in. The snowflake-python-connector continues to be my biggest pain in the behind with their dependency management.

[tomslade1]

We've just run headfirst into this issue with a colleague using a new macbook 😂 ; any rough timelines for a v0.20.0 pre-release @jtcohen6?

[ldbrandi]

Same issue here guys, looking forward to this new release to fix my local setup.

[ldbrandi]

Workaround installing though pip following this thread: psycopg/psycopg2#1200
Finally worked here!

[meyerjoseph]

I found this link to be helpful: https://nhancv.medium.com/macbook-pro-m1-homebrew-install-error-7170e4816894

[vklimontovich]

TL;DR: if you experience the error during dbt run, just add those lines to ~/.dbt/profiles.yml (see the full doc in profiles.yml here)

config:
    send_anonymous_usage_stats: False

Details:

Here's the stack trace (MacBook Air (M1, 2020) running Mac OS 11.3.1). The exception is raised in user behavior tracking code. Quickfix will be disabling tracking (see above). I guess the proper way to handle this would be wrapping tracker.flush() with try/except. Failure to track isn't a critical error after all

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/opt/homebrew/Cellar/python@3.8/3.8.10/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 229, in track_run
    dbt.tracking.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 428, in flush
    tracker.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/tracker.py", line 218, in flush
    emitter.sync_flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 182, in sync_flush
    Emitter.flush(self)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 149, in flush
    self.send_events(self.buffer)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 206, in send_events
    status_code = self.http_post(data).status_code
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 68, in http_post
    r = requests.post(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connection.py", line 343, in connect
    self.ssl_context = create_urllib3_context(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 290, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
    self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1028, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 331, in __init__
    self.callback = _ffi.callback(
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

[jtcohen6]

This is a weird one. The error is in pyOpenSSL (pyca/pyopenssl#873 (comment)), and it's cropping up in anonymous usage tracking (snowplow_tracker → requests → urllib3 → pyOpenSSL).

But the reason pyOpenSSL is installed in the first place is because it's explicitly required by snowflake-connector-python. If you're using a non-Snowflake adapter plugin, e.g. dbt-postgres, pyOpenSSL is never installed, urllib3 falls back to using the standard-library ssl module, and everything is fine. (The standard ssl library can be insecure for python2, but it sounds like less of a concern on python3. See: urllib3 docs, and for more explanation, section 1.4.1 of this PDF.)

Since dbt only supports python3, I'd like to figure out a way to tell urllib3 (required for anonymous usage tracking) to ignore pyOpenSSL if it's installed. It sounds like there may be a way to do it, via urllib3.contrib.pyopenssl.extract_from_urllib3(), but I'm not sure where we'd put that call, or if we're accessing it at too high a level.

@vklimontovich I'm glad you were able to get this working in the meantime by disabling anonymous usage tracking. Of course, I'd like to avoid it being an either/or choice!

cc @kwigley in case you find this interesting :)

[vklimontovich]

@jtcohen6 wouldn't wrapping tracking code into try/excerpt a good temporary solution? At least it won't fail :(

Something like this one:

diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
index d950bb84..576e949c 100644
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -425,7 +425,10 @@ def track_invalid_invocation(

 def flush():
     logger.debug("Flushing usage events")
-    tracker.flush()
+    try:
+        tracker.flush()
+    except:
+        logger.warn("Unable to flush usage events")


 def disable_tracking():

[jtcohen6]

@vklimontovich Have you been able to confirm that the addition of that try/except results in error-free runs? (i.e. by cloning dbt, editing, and installing from local)

That's fine as a temporary solution—if you can confirm it works, I'd welcome a PR for it. In addition, I'd like to find a way to work around the pyOpenSSL bug, if possible.

[brittianwarner]

@jtcohen6 Is this issue supposed to be resolved with dbt 0.20.0? After upgrading, I am able to run dbt --version however dbt run is now giving me this error. I tried the various work arounds but curious if there is an update? Thanks.

[jtcohen6]

@brittianwarner We didn't include the proposed fix above in v0.20.0. You're welcome to try turning off anonymous tracking, and see if that fixes it.

Over the past few weeks, there have been a few reports of successful installs on M1s by other means: #3239 (comment)

[michael-urrutia]

@jtcohen6 do you have any insight into when the patch to resolve this issue will be released so we do not need to find these work arounds?

[jtcohen6]

@kwigley and I just took a look at this, trying to figure out if there's a way we can undo the urllib3/pyopenssl monkey patch that snowflake-connector-python turns on here.

If we can't figure that out—or even if we can—let's add try/except around tracker.flush() here. As a general principle, a failure to send anonymous usage tracking shouldn't prevent folks from using dbt.

We'll do our best to sneak in this fix for v0.20.1.

[Nathancalon]

Hi - I’m having this issue- this is the first time trying to use dbt so a little frustrating - what‘s the latest and what should a newbie do here to get the connection working please?
Thanks

[jtcohen6]

@Nathancalon If you're using an M1 Mac, we highly recommend you install dbt via Homebrew + Rosetta, as outlined here: https://hackernoon.com/apple-m1-chip-how-to-install-homebrew-using-rosetta-su12331b

That's what I've been doing on my M1 for months now. While it's a few extra steps to set up, it's been working without a hitch.

[mr-m0nst3r]

Hey guys,

I fixed this ffi problem by reinstalling the libffi using homebrew, and install cffi wheel: https://files.pythonhosted.org/packages/3e/9b/660d6da900af1976a8b4efea713a7ce9e514bf4659eff9b17f90f00be1cf/cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl

[tpark464]

I've tried installing dbt@0.18.0 with Homebrew using Rosetta and I get an error:

Building wheels for collected packages: cryptography
Building wheel for cryptography (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for cryptography (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [142 lines of output]

any help would be much appreciated!

[leahwicz]

@tpark464 I would recommend using the latest version of dbt and the associated adapter (brew install dbt-redshift). We are on v1.2 at the moment. The older versions of dbt in Homebrew are not actively maintained and have breaking dependency changes over the long time period

[tpark464]

@tpark464 I would recommend using the latest version of dbt and the associated adapter (brew install dbt-redshift). We are on v1.2 at the moment. The older versions of dbt in Homebrew are not actively maintained and have breaking dependency changes over the long time period

for anyone still having issues, this works!!!

[sippola]

I have same issue with M1 mac, we use Snowflake with dbt.
This note from Snowflake docs fixed it for me: a fresh conda environment initialized with these commands and it worked. https://docs.snowflake.com/en/developer-guide/snowpark/python/setup#prerequisites

CONDA_SUBDIR=osx-64 conda create -n snowpark python=3.8 numpy pandas --override-channels -c https://repo.anaconda.com/pkgs/snowflake
conda activate snowpark
conda config --env --set subdir osx-64