Basic install of keras and tensorflow on M1 Mac leads to crashes

[jebyrnes]

Hello! I am trying to get keras and tensorflow to run on R 4.0.3 on a brand new installation on an M1 macbook pro.

I installed both keras and tensorflow libraries, ran install_keras() and install_tensorflow() and installed miniconda along with them as prompted.

However, whenever I try and access tensorflow, my R session crashes (both in Rstudio and R.app).

Even the following

library(tensorflow)
tf$constant("Hellow Tensorflow")

causes a hard crash. Should I be specifying different versions, etc., to my install_* functions?

(sidenote - can tensorflow be installed to make use of the GPUs on an M1? Would love to do this)

[jebyrnes]

Would it be possible to install tensorflow from https://github.com/apple/tensorflow_macos? Could this work in install_tensorflow() somehow?

[bcaessens]

Hi. Same issue here. I have tried the above, install the AppleML tensorflow version succesfully in a venv. Then with reticulate and use_python pointing to this python distribution. However, same issue. Every call to tensorflow/keras crashes Rstudio.

[bcaessens]

Hi, here is an update to this after 1.5 days of struggling:

1. Managed to get tensorflow for M1 running on the Mac, using https://github.com/apple/tensorflow_macos/issues/153 and (this)[https://github.com/Instructions to install TensorFlow in a Conda Environment apple/tensorflow_macos#153] as source.
2. Confirmed in python this is running well.
3. Started clean R session in Rstudio then ran these:

library(reticulate) use_condaenv('tf24', required=T) py_config()
And got error in py_initialize: not finding the libpython3.8.dylib image, finding other images but wrong architecture.

Stuck there!
Bernie.

[mwidjaja1]

Edit: ...I'm an idiot. I thought I was in a totally different repo, didn't realize I jumped into the Rstudio repo. My comment probably is even less helpful than it is before -- my apologies!

Rstudio is still on Intel via Rosetta -- it's not Apple Silicon Native. Integrating that with Python (especially since you used my issue which encourages you to use the ARM route) wouldn't work. My best guess would be you need to get a Rosetta compiled Python, somehow trick it to get the Intel version of Tensorflow, then trick Rstudio to accept it.

[bcaessens]

I don’t think your comment is not helpful. Quite the opposite (even if unintentional). But I think you go right down to the core of the issue, although I am not sure.

1. Using the ARM version of tensorflow 2.4, compiled by Apple works for python (and Pycharm is now on M1 silicon natively too :-) -> but that won’t fly in Rosetta 2 based Rstudio (and the reticulate package, which thinks it is on intel machinery).

2. Getting intel based python running with Rosetta 2 is not an issue, in Rstudio even. But I think Tensorflow natively has core CPU methods that won’t work on Rosetta?? <- not sure about this!!!!

3. So the best thing for now is to run all tensorflow - keras stuff in python, and provide export/import of data into R. To smooth this out, I am going to try to set up some cloud python stuff with simple POST/GET Api that we can call from R.

Bummer.

[mwidjaja1]

As an update, someone had contributed to my personal 'Data Science on ARM' notes at https://github.com/mwidjaja1/DSOnMacARM#pythonrfortran-ie-anacondaminiforge. In that section, someone implied you can use Miniforge/Conda (think of it it as a Python package manager) to awkwardly install a version of R that is ARM compatible. I am NOT an R expert in anyway, to me this sounds like witchcraft that you can use Python to install R. But that might be another option too.

[cyshuai]

Hi Folks, I am encountered with the same problem. Do you fix your problem?

[fabfaba]

Hello! Same problem .. has anyone fixed the problem?

[TullioM94]

Same problem here, tensorflow works but keras crashes. Any update?

[jebyrnes]

So, I just saw this guide to getting tensorflow up and running on your M1 mac - https://medium.com/codex/installing-tensorflow-on-m1-macs-958767a7a4b3 - it works great!

If this gets tf installed, I guess the followup would be then how to make the keras and R packages in R use this installation instead of what comes through if you try and install it as is.

Thoughts?

[jebyrnes]

Some progress, sort of... After following the instructions above, I tried some of the pointers from https://tensorflow.rstudio.com/installation/custom/ and tried

library(tensorflow)
library(keras)

#from  file $(which python)
Sys.setenv(RETICULATE_PYTHON="/Users/jebyrnes/miniforge3/bin/python")
use_python("/Users/jebyrnes/miniforge3/bin/python")
use_condaenv("tf_env")

mnist <- dataset_mnist()

And instead of a crash, got

Error in py_initialize(config$python, config$libpython, config$pythonhome,  : 
  /Users/jebyrnes/miniforge3/lib/libpython3.9.dylib - dlopen(/Users/jebyrnes/miniforge3/lib/libpython3.9.dylib, 10): no suitable image found.  Did find:
	/Users/jebyrnes/miniforge3/lib/libpython3.9.dylib: mach-o, but wrong architecture
	/Users/jebyrnes/miniforge3/lib/libpython3.9.dylib: mach-o, but wrong architecture

So - I feel like this is the right way to go.....anyone else have further thoughts?

[TullioM94]

I have different passages that allow me to import correctly tensorflow, load mnist correctly, compile a Keras API model. However, when training crashes. The different steps are (you need to have anaconda installed):

On the mac terminal type:

conda create -y --name tf2 tensorflow keras python=3.6.8

This creates a conda enviroment called tf2 then to open Rstudio:

conda activate tf2
open -na rstudio 

Once you open Rstudio execute:

library(tensorflow)
library(keras)
library(reticulate)

sys <- import("sys")
np <- import("np") 

Then execute the following lines to make sure that tensorflow works:

hello <- tf$constant("Hello")
sess <- tf$Session()
d <- sess$run(hello)

And it works with no problem, then I load MNIST dataset with no problem as:

mnist <- dataset_mnist()

And I compile a simple neural network with Keras API with no problem. However when I fit the model, after a couple of seconds the session aborts.

Sometimes Keras may not be installed, the easiest way to install it is to open anaconda navigator, select the tf2 environment and install "r-keras"

I hope these steps are useful and someone from the community can take it from here to solve the Keras issue.

Tullio

[jebyrnes]

Hrm. This does not seem to work. I created an environment, tf_env using https://raw.githubusercontent.com/ptorres001/DSOnMacARM/main/environment.yml and installed tensorflow using the instructions at https://medium.com/codex/installing-tensorflow-on-m1-macs-958767a7a4b3 (using miniconda)

From terminal, running

conda activate tf_env
open -na rstudio 

And then

library(tensorflow)
library(keras)
library(reticulate)

main <- import_main()
sys <- import("sys")
np <- import("numpy", as = "np") 

hello <- tf$constant("Hello")

causes a crash.

Out of curiosity, I uninstalled the three packages with remove.packages(), closed RStudio, and then still in the environment, reopoened, reinstalled them, and tried again - still a crash.

[jebyrnes]

OK.
OK.

I installed the nightly arm build of R from https://mac.r-project.org/
I installed the daily build of RStudio from https://dailies.rstudio.com/

I reinstalled the libraries (reticulate had to be the binary - it failed from source)

I also have

Sys.setenv(RETICULATE_PYTHON = "~/miniforge3/envs/tf_env/bin/python")

in my .Renviron

Now, import_main() above failed. and sess <- tf$Session() threw an error.

But.

minst ran.

And it ran fast.

I'll wrap this into a blog post later tonight and then close this issue? I mean, I'd like to solve the other problems, but this is functional!

[jebyrnes]

OK - blog post with instructions is up at http://imachordata.com/2021/05/07/r-tensorflow-on-an-m1-mac-without-crashing/ - hopefully the apple fork will be pulled back into the main one, and things can continue (so we can do things like use tiffs and such from other libraries) - but, good luck, all! And thanks!

[pritom65]

Hi, here is an update to this after 1.5 days of struggling:

1. Managed to get tensorflow for M1 running on the Mac, using https://github.com/apple/tensorflow_macos/issues/153 and (this)[https://github.com/Instructions to install TensorFlow in a Conda Environment apple/tensorflow_macos#153] as source.
2. Confirmed in python this is running well.
3. Started clean R session in Rstudio then ran these:

library(reticulate) use_condaenv('tf24', required=T) py_config() And got error in py_initialize: not finding the libpython3.8.dylib image, finding other images but wrong architecture.

Stuck there! Bernie.

Sir please help me to resolve it. My mail is "proy288@gmail.com" I badly need the solution for this..

[t-kalinowski]

I believe that installation on M1 Macs works on the development version of the R package. This should be all you need:

remotes::install_github("rstudio/reticulate")
remotes::install_github("rstudio/tensorflow")
remotes::install_github("rstudio/keras")
reticulate::miniconda_uninstall()
reticulate::install_miniconda()
keras::install_keras()

Please file a new issue if this doesn't work!

[pritom65]

I believe that installation on M1 Macs works on the development version of the R package. This should be all you need:

remotes::install_github("rstudio/reticulate")
remotes::install_github("rstudio/tensorflow")
remotes::install_github("rstudio/keras")
reticulate::miniconda_uninstall()
reticulate::install_miniconda()
keras::install_keras()

Please file a new issue if this doesn't work!

I believe that installation on M1 Macs works on the development version of the R package. This should be all you need:

remotes::install_github("rstudio/reticulate")
remotes::install_github("rstudio/tensorflow")
remotes::install_github("rstudio/keras")
reticulate::miniconda_uninstall()
reticulate::install_miniconda()
keras::install_keras()

Please file a new issue if this doesn't work!

This is not working for an specific part. The miniconda is not installing. The message is

tarfile.ReadError: unexpected end of data
[86472] Failed to execute script entry_point
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
File "concurrent/futures/process.py", line 368, in _queue_management_worker
File "multiprocessing/connection.py", line 251, in recv
TypeError: init() missing 1 required positional argument: 'msg'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "entry_point.py", line 69, in
File "concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
File "concurrent/futures/_base.py", line 611, in result_iterator
File "concurrent/futures/_base.py", line 439, in result
File "concurrent/futures/_base.py", line 388, in __get_result
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
[86503] Failed to execute script entry_point
Error: miniconda installation failed [exit code 1]

[ginxyang]

I have different passages that allow me to import correctly tensorflow, load mnist correctly, compile a Keras API model. However, when training crashes. The different steps are (you need to have anaconda installed):

On the mac terminal type:

conda create -y --name tf2 tensorflow keras python=3.6.8

This creates a conda enviroment called tf2 then to open Rstudio:

conda activate tf2
open -na rstudio 

Once you open Rstudio execute:

library(tensorflow)
library(keras)
library(reticulate)

sys <- import("sys")
np <- import("np") 

Then execute the following lines to make sure that tensorflow works:

hello <- tf$constant("Hello")
sess <- tf$Session()
d <- sess$run(hello)

And it works with no problem, then I load MNIST dataset with no problem as:

mnist <- dataset_mnist()

And I compile a simple neural network with Keras API with no problem. However when I fit the model, after a couple of seconds the session aborts.

Sometimes Keras may not be installed, the easiest way to install it is to open anaconda navigator, select the tf2 environment and install "r-keras"

I hope these steps are useful and someone from the community can take it from here to solve the Keras issue.

Tullio

are you able to knit your file? creating a conda env and use that conda env in rstudio will make tf working. But I am unable to knit