install_keras() and install_tensorflow() failing with ModuleNotFoundError error

[gopalpenny]

I've been attempting to install tensorflow and keras in Rstudio on my Mac (OS Mojave, 10.14.6), and keep failing. I've re-installed R, Rstudio, and conda to obtain the latest versions, but I am running into the same error.

devtools::install_github("rstudio/tensorflow") # works
devtools::install_github("rstudio/keras") # works

But when I try to install keras or tensorflow using install_tensorflow() or install_keras() I get the following error:

> keras::install_keras()
Using virtual environment '~/.virtualenvs/r-reticulate' ...
Traceback (most recent call last):
  File "/Users/gopal/.virtualenvs/r-reticulate/bin/pip", line 6, in <module>
    from pip._internal.main import main
ModuleNotFoundError: No module named 'pip._internal.main'
Error in strsplit(output, "\\s+")[[1]] : subscript out of bounds
In addition: Warning message:
In system2(pip, "--version", stdout = TRUE) :
  running command ''/Users/gopal/.virtualenvs/r-reticulate/bin/pip' --version' had status 1

I've also tried upgrading my pip command, but that did not help either. And after reading this issue, it seems there shouldn't be any problems installing with Rstudio. FWIW, I tried installing r-essentials and Rstudio within conda, but wasn't able to get the conda version of Rstudio to open. I've run the following diagnostic commands in Rstudio as requested:

> reticulate::py_discover_config()
python:         /Users/gopal/.virtualenvs/r-reticulate/bin/python
libpython:      /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/libpython3.7.dylib
pythonhome:     /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7:/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7
version:        3.7.3 (default, Jun 19 2019, 07:38:49)  [Clang 10.0.1 (clang-1001.0.46.4)]
numpy:          /usr/local/lib/python3.7/site-packages/numpy
numpy_version:  1.16.4

python versions found: 
 /Users/gopal/.virtualenvs/r-reticulate/bin/python
 /usr/bin/python
 /usr/local/bin/python
 /usr/local/bin/python3

> reticulate::use_condaenv("r-tensorflow")
Error: Unable to find conda binary. Is Anaconda installed?

> reticulate::py_config()
python:         /Users/gopal/.virtualenvs/r-reticulate/bin/python
libpython:      /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/libpython3.7.dylib
pythonhome:     /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7:/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7
version:        3.7.3 (default, Jun 19 2019, 07:38:49)  [Clang 10.0.1 (clang-1001.0.46.4)]
numpy:          /usr/local/lib/python3.7/site-packages/numpy
numpy_version:  1.16.4
tensorflow:     [NOT FOUND]

python versions found: 
 /Users/gopal/.virtualenvs/r-reticulate/bin/python
 /usr/bin/python
 /usr/local/bin/python
 /usr/local/bin/python3

I do have anaconda installed on my computer, it's located in ~/opt/anaconda2/condabin, which is in my $PATH. Overall, I'm fairly unfamiliar with conda and I'm somewhat lost as far as what to try next.

[dfalbel]

@gopalpenny Can you try:

1. Downlaod and install anconda 3 from here: https://www.anaconda.com/distribution/
2. Open RStudio as usual, not RStudio that comes with conda.
3. Run: install_tensorflow(method = "conda")

[gopalpenny]

@dfalbel Thank you, I installed anaconda 3, and I now have tensorflow and keras up and running! I also had to specifiy the conda path, so it looked like install_tensorflow(method="conda",conda=[path]), and it installed without any problems.

[faltinl]

@gopalpenny Hi,-
could you tell me, how/where you did find the [path] in install_tensorflow(method="conda",conda=[path])? It looks as if I had the same or a similar problem as you had.
Tnx in advance!

[gopalpenny]

I dug up some old lines of code and it looks like I used the following for the path:

tensorflow::install_tensorflow(method = "conda",conda="~/opt/anaconda3/bin/conda")

In other words, I found the conda installation and then looked for /bin/conda

[faltinl]

Tnx a lot for prompt reply, gopalpenny. I tried the same within my Anaconda3-directory but, strange enough, there wasn't anything looking similar. So at the moment I'm at a loss what to do next.
Thank you very much for your support anyway & all the best!

[genobobeno]

I'm on a macbook with Big Sur, and I'm brand new to integrating python into my work. For my current project, I'm trying to build a neural network to compare with gbm, rf, glm, and an ensemble... and the python configuration is confusing because the documentation is missing any reference to the workarounds for all of the errors and warnings that RStudio over-engineered into their python-R integration. These warnings and error messages in the R console are very confusing: So far, the tensorflow package installed just fine, but the call library(tensorflow) returned a message that it couldn't load the tensorflow package because the tensorflow module isn't already installed in the python environment (nothing in the documentation says anything about this)...

Error: package or namespace load failed for ‘tensorflow’: .onLoad failed in loadNamespace() for 'tensorflow', details: call: py_module_import(module, convert = convert) error: ModuleNotFoundError: No module named 'tensorflow'

and then the "install_tensorflow()" command prompts you to install a whole new miniconda installation (again, nothing in the webpage for tensorflow install says anything about this). And I just installed the python module to be able to load the tensorflow package, it now seems that running install_tensorflow() is redundant, and since there are multiple installations of python, I don't even know into which environment R is installing the dependencies and newer tensorflow module. I had very similar problems making all of this work on Windows 10 last Fall (2020), so the fact that this documentation still isn't noting these repeatable hiccups is disappointing.

So next, install_tensorflow() seemed to look up a hundred packages that have conflicts with everything in the base 'r-reticulate' environment. Literally everything says there's a conflict. I have no idea what to do about this. I'm not a python expert... I'm just trying to use R to run tensorflow. Here is just one sample of the 3000 lines of illegible output:

Package libev conflicts for: libnghttp2 -> libev[version='>=4.11|>=4.33,<4.34.0a0'] libcurl -> libnghttp2[version='>=1.41.0,<2.0a0'] -> libev[version='>=4.11|>=4.33,<4.34.0a0'] libev

And to top it all off, install_tensorflow() seems to have tried to install python 3.7 into my 'r-reticulate' environment, which makes no sense to me at all... and nowhere in the documentation am I reading that R's tensorflow installation requires python 3.7.

Error: installation of 'python=3.7' into environment '/Users/eugene/opt/anaconda3/envs/r-reticulate' failed [error code 1]

So now that this neophyte python user just installed a whole bunch of python crap that was essentially the "wrong version of everything" with conflict after conflict after conflict (not good things, not good messages to see coming from your R environment)... I even create a conda environment 'r-reticulate' to match RStudio's defaults, etc... and now there is nowhere in the documentation that teaches you how to "back out" of everything that was just installed, so that I can go back and install anaconda3 under a different python version in the hopes that tensorflow will install if I try again...

but hey, now I found this thread, so let's try install_tensorflow(method="conda"): same result. Hundreds of conflicts. Do they matter? How would I know? And the tensorflow library tried to install python 3.7 again... why would it do that? Who knows, cause the documentation doesn't say that python 3.7 is required. So since it ended with this stupid Python 3.7 error, did it install? Did anything happen? Or did the tensorflow package just burn me?

To say this is incredibly frustrating is an immense understatement.

[t-kalinowski]

Hi, I'm sorry that your experience has been so frustrating, let's see if I can help. 2 questions:

1. Are you on an M1 Mac? If yes, tensorflow::install_tensorflow()/keras::install_keras() will not work for you. Please see the documentation at ?install_tensorflow for a discussion about running tensorflow on Apple Silicon macOS and a link for how to do a workaround.

2. Please start a fresh R session (ctrl+shift+F10 in RStudio), and paste the following snippet into the console. If it does not result in a working Tensorflow installation you can use from R, then please open a new issue with the full output you see at the console:

install.packages(c("tensorflow", "keras")) # maybe upgrade to latest version

# delete previous reticulate conda installation, start from a blank slate.
unlink(reticulate::miniconda_path(), recursive = TRUE) 
reticulate::install_miniconda()

keras::install_keras() # install tensorflow + a handful of additional python modules
tensorflow::tf$abs(-1) # test if tensorflow can load and run