Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make TensorFlow compatible with PyPy #252

Open
FabHan opened this issue Nov 17, 2015 · 48 comments
Open

Make TensorFlow compatible with PyPy #252

FabHan opened this issue Nov 17, 2015 · 48 comments
Labels
stat:contribution welcome Status - Contributions welcome type:feature Feature requests

Comments

@FabHan
Copy link

FabHan commented Nov 17, 2015

I know it's not a priority and will be a long way to get there; but making TF compatible with PyPy woud be super cool.

Thoughts?

@mrry
Copy link
Contributor

mrry commented Nov 19, 2015

As a team we don't use PyPy day-to-day, but we would welcome contributions if it's easy to do this without breaking CPython compatibility.

My guess is that the two stumbling blocks would be TensorFlow's reliance on NumPy in the Python front-end, and SWIG for interfacing with the C++ backend. Are you aware of any other issues that one would face?

@girving girving added the stat:contribution welcome Status - Contributions welcome label Mar 8, 2016
@girving
Copy link
Contributor

girving commented Mar 8, 2016

I'm not sure what the state of PyPy binding layers is these days, but there's a good chance this would require rewriting the whole swig interface.

@lvella
Copy link

lvella commented Jul 13, 2016

PyPy only good interface with native code seems to be CFFI, but it is a C <-> Python interface library. Besides being slower than CPython native interface (when running on CPython, of course), it would be too painful to interface with C++ (Python <-> C <-> C++).

@girving
Copy link
Contributor

girving commented Jul 13, 2016

The medium term plan for language bindings is to go through the C API. We're in the process of moving functionality from Python to C++, at which point it will also be exposed through an extended C API. Moving stuff out of Python to C++ isn't relevant for PyPy, but the improved C API might make it easy to write a separate PyPy interface.

@classner
Copy link

Great news from the side of PyPy: since version 5.6.0 (released in November) the devs included a compatibility layer for the CPython C-API (cpyext) (see also here). With this, I was able to build numpy, scipy and sklearn out-of-the box, so finally machine learning with pypy gets interesting.

Of course I wanted to then also check tensorflow, with pretty good results! The build completes successfully out-of-the box (!!), so no need to adapt any interfaces. However, import tensorflow then fails with the well-known "ImportError: No module named _pywrap_tensorflow" (of course not from the tensorflow directory). Looking into it, ldd -r _pywrap_tensorflow.so gave me some "undefined symbol: PyPy...." errors, indicating that just an additional link command during the build could fix the issue! Since I'm not so familiar with bazel, I didn't go further to resolve it...

@girving
Copy link
Contributor

girving commented Jan 24, 2017

@classner Very cool! Presumably you have to link against a cpyext library. @martinwicke Who would be the right person to ask for assistance here?

@martinwicke
Copy link
Member

@classner can you try the cmake build? If you can make that work, then we know what needs to be added, and we can put it into the bazel build. Bazel really wants all its dependencies declared, so it would be good to have a clear picture of what exactly is missing.

@classner
Copy link

Just posted on the pypy dev mailing list to get a bit more input from that side! Will post further info here...

@classner
Copy link

One of the devs was incredibly quick to reply: apparently, all the symbols I mentioned are part of libpypy-c.so, which is in the bin folder of the pypy distribution. I'm currently super-short on time and can't look at it right now, but could do sometime during the next weeks.

@classner
Copy link

Alright, managed to build with bazel and cmake successfully (v1.0.0-rc0). The trick to make it work is to rename the generated _pywrap_tensorflow.so to _pywrap_tensorflow.pypy-XX.so, where XX is the pypy version code (in my case 41 for pypy 5.6.0). With this naming scheme, import tensorflow works nicely. So second base touched.

This part of the 'get started' code runs:

import tensorflow as tf
import numpy as np

# Create 100 phony x, y data points in NumPy, y = x * 0.1 + 0.3
x_data = np.random.rand(100).astype(np.float32)
y_data = x_data * 0.1 + 0.3

# Try to find values for W and b that compute y_data = W * x_data + b
# (We know that W should be 0.1 and b 0.3, but TensorFlow will
# figure that out for us.)
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b

# Minimize the mean squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5)
train = optimizer.minimize(loss)

# Before starting, initialize the variables.  We will 'run' this first.
init = tf.global_variables_initializer()

# Launch the graph.
sess = tf.Session()

sess.run(init) fails with a TypeError:

TypeError: in method 'TF_DeleteBuffer', argument 1 of type 'TF_Buffer *'.

@classner
Copy link

classner commented Feb 6, 2017

Any feedback on this from the interface developers? Could this TypeError be resolved easily?

@mrry
Copy link
Contributor

mrry commented Feb 6, 2017

Can you provide more information about the TypeError? From looking at session.py, it seems the most likely candidate for raising this exception is this line. What type does run_metadata_ptr have in your version of the code?

@classner
Copy link

classner commented Feb 6, 2017

The line in question is indeed causing the TypeError. This is what I get when adding an ipdb breakpoint just before:

    772       import ipdb; ipdb.set_trace()
--> 773       tf_session.TF_DeleteBuffer(run_metadata_ptr)
    774       if options:

ipdb> run_metadata_ptr
<tensorflow.python.pywrap_tensorflow.TF_Buffer;  >
ipdb> type(run_metadata_ptr)
<class 'tensorflow.python.pywrap_tensorflow.TF_Buffer'>
ipdb> n
TypeError: "in method 'TF_DeleteBuffer', argument 1 of type 'TF_Buffer *'"

(str and repr don't give any more helpful information)

@girving
Copy link
Contributor

girving commented Feb 6, 2017

I don't have anything constructive to say, but I can't resist chiming in to gripe about error messages in one of these two forms:

  1. "You gave me an X. I expected something else!"
  2. "I expected a Y. You gave me something else!"

Head...desk.

@mrry
Copy link
Contributor

mrry commented Feb 6, 2017

@girving Unfortunately that error message is in generated code. Maybe take it up with the authors of https://github.com/swig/swig? :)

Judging by that generated code, we seem to be in case 2 ("I expected a TF_Buffer*"). SWIG converts this C type to and from a pointer-wrapper object. I don't know enough about PyPy or cpyext to know if this type could be getting swizzled somehow. Here are the relevant generated code fragments for the wrappers for TF_NewBuffer() and TF_DeleteBuffer():

SWIGINTERN PyObject *_wrap_TF_NewBuffer(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  TF_Buffer *result = 0 ;
  
  if (!PyArg_ParseTuple(args,(char *)":TF_NewBuffer")) SWIG_fail;
  {
    Py_BEGIN_ALLOW_THREADS;
    result = (TF_Buffer *)TF_NewBuffer();
    Py_END_ALLOW_THREADS;
  }
  resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_TF_Buffer, 0 |  0 );
  return resultobj;
fail:
  return NULL;
}

SWIGINTERN PyObject *_wrap_TF_DeleteBuffer(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  TF_Buffer *arg1 = (TF_Buffer *) 0 ;
  void *argp1 = 0 ;
  int res1 = 0 ;
  PyObject * obj0 = 0 ;
  
  if (!PyArg_ParseTuple(args,(char *)"O:TF_DeleteBuffer",&obj0)) SWIG_fail;
  res1 = SWIG_ConvertPtr(obj0, &argp1,SWIGTYPE_p_TF_Buffer, 0 |  0 );
  if (!SWIG_IsOK(res1)) {
    SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "TF_DeleteBuffer" "', argument " "1"" of type '" "TF_Buffer *""'"); 
  }
  arg1 = reinterpret_cast< TF_Buffer * >(argp1);
  {
    Py_BEGIN_ALLOW_THREADS;
    TF_DeleteBuffer(arg1);
    Py_END_ALLOW_THREADS;
  }
  resultobj = SWIG_Py_Void();
  return resultobj;
fail:
  return NULL;
}

@aselle aselle added type:feature Feature requests and removed enhancement labels Feb 9, 2017
@standy66
Copy link
Contributor

standy66 commented Jul 20, 2017

I was able to reproduce @classner steps for tensorflow 1.2 with CUDA for pypy3 version 5.8.0-beta0 (Python 3.5.3). In order to do this, _pywrap_tensorflow_internal.so library in site-packages/tensorflow/python needs to be renamed to _pywrap_tensorflow_internal.pypy3-58-x86_64-linux-gnu.so. I am getting a slightly different error during sess.run:

$ LD_LIBRARY_PATH=/usr/local/cuda/lib64 pypy3.5

Python 3.5.3 (a37ecfe5f142bc971a86d17305cc5d1d70abec64, Jun 08 2017, 19:43:54)
[PyPy 5.8.0-beta0 with GCC 6.3.0] on linux

>>>> import tensorflow as tf
>>>> x = tf.constant(1)
>>>> y = tf.constant(2)
>>>> z = x * y
>>>> with tf.Session() as sess:
....     sess.run(z)
....     
2017-07-20 14:43:47.795964: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-07-20 14:43:47.796979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: 
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.936
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.67GiB
2017-07-20 14:43:47.796992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 
2017-07-20 14:43:47.796995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y 
2017-07-20 14:43:47.797003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/andrew/.virtualenvs/tensorflow_pypy/site-packages/tensorflow/python/client/session.py", line 780, in run
    run_metadata_ptr = tf_session.TF_NewBuffer()
TypeError: argument after * must be an iterable, not NoneType

It seems that the first line in session.run fails to execute.

@GSam
Copy link

GSam commented Mar 10, 2018

I'm quite interested in this working under pypy. I didn't have any luck with CNTK either but I've discovered that Theano actually works under pypy (with some minor hacks), so for now, it seems I'll be using that instead.

Edit: Patches are now upstream for anyone interested.

@mattip
Copy link

mattip commented Oct 13, 2019

I think the TypeError: argument after * must be an iterable, not NoneType error is due to using an old version of SWIG. Could you make sure you are using a version after 4.0.0?

@aviramha
Copy link

aviramha commented Nov 5, 2019

Any news regarding this matter?
@classner @standy66 do you mind sharing instructions how you reached that point?

@htfy96
Copy link

htfy96 commented Nov 23, 2019

I think this issue is closed by mistake. The #252 referenced in 913871c's message is actually tensorflow/mlir#252, not this issue.

@arjunchitturi
Copy link

arjunchitturi commented May 14, 2020

@mattip , was able to get it working with tensorflow 1.14.0

Here, i ran into this error:
buf = c_api.TF_NewBuffer()
TypeError: argument after * must be an iterable, not NoneType

Do you suggest any pending prs to fix this?
Thanks for the help.

@mattip
Copy link

mattip commented May 14, 2020

@arjunc77 that sounds familiar: looking higher in the thread I see a similar comment. Could you make sure you are using SWIG > 4.0.0 to generate the wrappers?

@arjunchitturi
Copy link

arjunchitturi commented May 16, 2020

Thanks @mattip , that helped.

There could be a better way to update swig, but here is what worked for me.
4b5840e

@NeighborhoodCoding
Copy link

NeighborhoodCoding commented May 22, 2020

I finally made my pypy compatible in tensorflow. It works good but tf.conv layer is still not working yet in my pypy

@mattip
Copy link

mattip commented May 22, 2020

@SungmanHong can you share the formula? What did you need to do?

@tobivcu
Copy link

tobivcu commented May 31, 2020

I finally made my pypy compatible in tensorflow. It works good but tf.conv layer is still not working yet in my pypy

Could you please help me on that? I also desperately want my tensorflow to be compatible with pypy <33 Thank you!

@redradist
Copy link

What is the status of this issue ? Is there any progress ?

@redradist
Copy link

redradist commented Jul 19, 2020

@SungmanHong

I finally made my pypy compatible in tensorflow. It works good but tf.conv layer is still not working yet in my pypy

Could you share receipt how you did this ? Or maybe article ?

@redradist
Copy link

@mrry Do you have some plan to support PyPy ?

@martinwicke
Copy link
Member

martinwicke commented Aug 11, 2020 via email

@redradist
Copy link

We have removed SWIG, so there's a chance it works now? We do not officially support it, but maybe we're lucky.

@martinwicke I have tried a few seconds ago to install tensorflow on PyPy and it still does not work ...
What do you mean that it could work ?

@mattip
Copy link

mattip commented Aug 11, 2020

@redradist: did you try with the latest nightly from http://buildbot.pypy.org/nightly/py3.6 and latest HEAD of tensorflow? Was the error similar to the one above?

@redradist
Copy link

@mattip I have tried and got the following error:

WARNING: Download from https://mirror.bazel.build/github.com/aws/aws-sdk-cpp/archive/1.7.336.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'aws':
   java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed
ERROR: /mnt/i/tensorflow/tensorflow/tools/pip_package/BUILD:171:1: //tensorflow/tools/pip_package:licenses depends on @aws//:LICENSE in repository @aws which failed to fetch. no such package '@aws//': java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: no such package '@aws//': java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed

@mattip
Copy link

mattip commented Aug 11, 2020

Are you sure this error is pypy-specific?

@redradist
Copy link

Are you sure this error is pypy-specific?

@mattip No, this error relates to TensorFlow build from master HEAD
pypy-latest works properly ...

@mattip
Copy link

mattip commented Aug 11, 2020

OK, my directions should have been "latest pypy3.6 nightly and latest tensorflow that successfully builds with cpython3.6"

@andyDoucette
Copy link

andyDoucette commented Jan 11, 2021

Does anyone have any specific instructions for making this work? I tried both version 2.3.1 (my desired version) and trying to pip install from the git repo and got this message:

[user@ba cli]$ sudo /opt/pypy3/current/bin/pypy3 -m pip install git+https://github.com/tensorflow/tensorflow.git@master
sudo: unable to resolve host ba
Collecting git+https://github.com/tensorflow/tensorflow.git@master
  Cloning https://github.com/tensorflow/tensorflow.git (to revision master) to /tmp/pip-req-build-dpufquz5
    ERROR: Command errored out with exit status 1:
     command: /opt/pypy3/current/bin/pypy3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-dpufquz5/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-dpufquz5/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ed0ud5zb
         cwd: /tmp/pip-req-build-dpufquz5/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/pypy3/current/lib-python/3/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-dpufquz5/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available.
You should consider upgrading via the '/opt/pypy3/current/bin/pypy3 -m pip install --upgrade pip' command.

I'm using pypy3.6-v7.3.3-linux64.tar.bz2

@orangelash
Copy link

Hello everyone,
I downloaded the latest pypy nightly build with python 3.6 using https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2 as suggested (pypy-c-jit-101029-118564033d0a-linux64) previously as well as trying to get the latest head of tensorflow. I have been trying to get tensorflow 2 to work in a PyPy venv using

Python 3.6.12 (118564033d0a, Nov 26 2020, 02:00:26)
[PyPy 7.3.4-alpha0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
pip 21.1.1 on pypy-venv

But I have been having issues. I have followed this thread and some other issue posts, but I remain unable to advance.

First off, running pip install tensorflow or pip install tf-nightly just returns the following error:

ERROR: Could not find a version that satisfies the requirement tf-nightly (from versions: none)
ERROR: No matching distribution found for tf-nightly

I then tried to compile tensorflow from source in the PyPy venv and, after installing the requisites and configuring the environment, the following error log occurs during bazel build //tensorflow/tools/pip_package:build_pip_package:

bazel build //tensorflow/tools/pip_package:build_pip_package
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=83
INFO: Reading rc options for 'build' from /root/prema-rpi/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /root/prema-rpi/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --noincompatible_prohibit_aapt1 --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2
INFO: Reading rc options for 'build' from /root/prema-rpi/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/root/prema-rpi/venv-pypy3.6/bin/python3 --action_env PYTHON_LIB_PATH=/root/prema-rpi/venv-pypy3.6/site-packages --python_path=/root/prema-rpi/venv-pypy3.6/bin/python3
INFO: Found applicable config definition build:short_logs in file /root/prema-rpi/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /root/prema-rpi/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /root/prema-rpi/tensorflow/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels
INFO: Found applicable config definition build:dynamic_kernels in file /root/prema-rpi/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (414 packages loaded, 33390 targets configured).
INFO: Found 1 target...
[10,060 / 15,182] 8 actions running
INFO: Found 1 target...
ERROR: /tensorflow/tensorflow/python/util/BUILD:367:11: C++ compilation of rule '//tensorflow/python/util:stack_trace' failed (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 74 argument(s) skipped)
In file included from tensorflow/python/util/stack_trace.cc:16:0:
./tensorflow/python/util/stack_trace.h: In static member function 'static tensorflow::StackTrace tensorflow::StackTrace::Capture(int)':
./tensorflow/python/util/stack_trace.h:64:55: error: 'PyThreadState {aka struct _ts}' has no member named 'frame'
     const PyFrameObject* frame = PyThreadState_GET()->frame;
                                                       ^~~~~
./tensorflow/python/util/stack_trace.h:66:58: error: 'const PyFrameObject {aka const struct _frame}' has no member named 'f_back'
     for (; i < limit && frame != nullptr; frame = frame->f_back, ++i) {
                                                          ^~~~~~
./tensorflow/python/util/stack_trace.h:71:67: error: 'const PyFrameObject {aka const struct _frame}' has no member named 'f_lasti'
       result.code_objs_.push_back(std::make_pair(code_obj, frame->f_lasti));
                                                                   ^~~~~~~
tensorflow/python/util/stack_trace.cc: In member function 'std::vector<tensorflow::StackFrame> tensorflow::StackTrace::ToStackFrames(const StackTraceMap&, const StackTraceFilter&, bool, int) const':
tensorflow/python/util/stack_trace.cc:58:77: error: 'PyCode_Addr2Line' was not declared in this scope
     const int line_number = PyCode_Addr2Line(code_obj.first, code_obj.second);
                                                                             ^
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /tensorflow/tensorflow/lite/python/BUILD:59:10 C++ compilation of rule '//tensorflow/python/util:stack_trace' failed (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 74 argument(s) skipped)
INFO: Elapsed time: 2992.692s, Critical Path: 109.58s
INFO: 9434 processes: 124 internal, 9310 local.
FAILED: Build did NOT complete successfully

As @martinwicke mentioned, since SWIG was removed I did not apply any fix to pybind like mentioned in pybind/pybind11#2146.

I have also tried to install it by using the google storage links provided in the tensorflow installation docs:

python3 -m pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.5.0-cp36-cp36m-manylinux2010_x86_64.whl

but I get the error:

ERROR: tensorflow_cpu-2.5.0-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.

I have also tried changing from manylinux2010 to manylinux1 to solve this, but it also did not work.

Looking for most of these issues online leads me to 64 and 32 bit conflicts, but I have made sure that I am running the proper 64bit version.

I have also tried doing the pip install with the latest tensorflow git head as source. I dont get any output after cloning and tensorflow remains not installed.

pypy -m pip install git+https://github.com/tensorflow/tensorflow.git@master

Collecting git+https://github.com/tensorflow/tensorflow.git@master
  Cloning https://github.com/tensorflow/tensorflow.git (to revision master) to /tmp/pip-req-build-y109k5h2
  Running command git clone -q https://github.com/tensorflow/tensorflow.git /tmp/pip-req-build-y109k5h2

I have come across several installation issues with other libraries in pypy, but I have usually been able to resolve them either by compiling from source or falling back to a specific version. Tensorflow is being rather stuborn.

Something of note is that I am able to import tensorflow but unable to actually use any module like from tensorflow import keras or tensorflow.zeroes[int].

If I freeze pip, tensorflow does not appear in the list.
I have also tried with the latest releases of PyPy using python 3.7 and python 3.8, with varying degrees of success. The 3.6 version I am trying now seems to be having better results, considering compatibility with other libraries I am using. @mattip are there any other directions you could give?

@mattip
Copy link

mattip commented May 19, 2021

You will not find a binary package available for PyPy, so the attempts to do pip install will not succeed. As for python3.6/python3.7: python3.6 is in security-fix only support. Many package maintainers like numpy and scipy have dropped support for it, as has PyPy. Please use 3.7.

Since there are no binary packages, you will have to build from source. The relevant information in your analysis is:

  • The code in ./tensorflow/python/util/stack_trace.h 'static tensorflow::StackTrace tensorflow::StackTrace::Capture(int)' will not compile on PyPy. The best strategy here is for tensorflow to adopt the workaround from pybind11
  • PyCode_Addr2Line is not implemented in PyPy. That API call is undocumented and not implemented in PyPy. I opened an issue for this in the PyPy issue tracker.

@sisrfeng
Copy link

I finally made my pypy compatible in tensorflow. It works good but tf.conv layer is still not working yet in my pypy

Hi, could you share your method?

@mohantym mohantym self-assigned this Oct 13, 2021
@mohantym mohantym added stat:contribution welcome Status - Contributions welcome and removed stat:contribution welcome Status - Contributions welcome labels Oct 13, 2021
@mohantym mohantym removed their assignment Oct 27, 2021
@lcy-mich

This comment was marked as spam.

@github-actions
Copy link

github-actions bot commented Sep 4, 2023

This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Sep 4, 2023
mahmoud-abuzaina added a commit to Intel-tensorflow/tensorflow that referenced this issue Oct 4, 2023
@leso-kn
Copy link

leso-kn commented Jan 21, 2024

@mattip @orangelash Quick ping on the status of PyPy: the call PyCode_Addr2Line has been implemented by the time!

@google-ml-butler google-ml-butler bot removed the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stat:contribution welcome Status - Contributions welcome type:feature Feature requests
Projects
None yet
Development

No branches or pull requests