New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make TensorFlow compatible with PyPy #252
Comments
As a team we don't use PyPy day-to-day, but we would welcome contributions if it's easy to do this without breaking CPython compatibility. My guess is that the two stumbling blocks would be TensorFlow's reliance on NumPy in the Python front-end, and SWIG for interfacing with the C++ backend. Are you aware of any other issues that one would face? |
I'm not sure what the state of PyPy binding layers is these days, but there's a good chance this would require rewriting the whole swig interface. |
PyPy only good interface with native code seems to be CFFI, but it is a C <-> Python interface library. Besides being slower than CPython native interface (when running on CPython, of course), it would be too painful to interface with C++ (Python <-> C <-> C++). |
The medium term plan for language bindings is to go through the C API. We're in the process of moving functionality from Python to C++, at which point it will also be exposed through an extended C API. Moving stuff out of Python to C++ isn't relevant for PyPy, but the improved C API might make it easy to write a separate PyPy interface. |
Great news from the side of PyPy: since version 5.6.0 (released in November) the devs included a compatibility layer for the CPython C-API (cpyext) (see also here). With this, I was able to build numpy, scipy and sklearn out-of-the box, so finally machine learning with pypy gets interesting. Of course I wanted to then also check tensorflow, with pretty good results! The build completes successfully out-of-the box (!!), so no need to adapt any interfaces. However, |
@classner Very cool! Presumably you have to link against a |
@classner can you try the cmake build? If you can make that work, then we know what needs to be added, and we can put it into the bazel build. Bazel really wants all its dependencies declared, so it would be good to have a clear picture of what exactly is missing. |
Just posted on the pypy dev mailing list to get a bit more input from that side! Will post further info here... |
One of the devs was incredibly quick to reply: apparently, all the symbols I mentioned are part of |
Alright, managed to build with bazel and cmake successfully (v1.0.0-rc0). The trick to make it work is to rename the generated This part of the 'get started' code runs:
|
Any feedback on this from the interface developers? Could this TypeError be resolved easily? |
Can you provide more information about the |
The line in question is indeed causing the TypeError. This is what I get when adding an ipdb breakpoint just before:
( |
I don't have anything constructive to say, but I can't resist chiming in to gripe about error messages in one of these two forms:
Head...desk. |
@girving Unfortunately that error message is in generated code. Maybe take it up with the authors of https://github.com/swig/swig? :) Judging by that generated code, we seem to be in case 2 ("I expected a SWIGINTERN PyObject *_wrap_TF_NewBuffer(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
TF_Buffer *result = 0 ;
if (!PyArg_ParseTuple(args,(char *)":TF_NewBuffer")) SWIG_fail;
{
Py_BEGIN_ALLOW_THREADS;
result = (TF_Buffer *)TF_NewBuffer();
Py_END_ALLOW_THREADS;
}
resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_TF_Buffer, 0 | 0 );
return resultobj;
fail:
return NULL;
}
SWIGINTERN PyObject *_wrap_TF_DeleteBuffer(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
TF_Buffer *arg1 = (TF_Buffer *) 0 ;
void *argp1 = 0 ;
int res1 = 0 ;
PyObject * obj0 = 0 ;
if (!PyArg_ParseTuple(args,(char *)"O:TF_DeleteBuffer",&obj0)) SWIG_fail;
res1 = SWIG_ConvertPtr(obj0, &argp1,SWIGTYPE_p_TF_Buffer, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "TF_DeleteBuffer" "', argument " "1"" of type '" "TF_Buffer *""'");
}
arg1 = reinterpret_cast< TF_Buffer * >(argp1);
{
Py_BEGIN_ALLOW_THREADS;
TF_DeleteBuffer(arg1);
Py_END_ALLOW_THREADS;
}
resultobj = SWIG_Py_Void();
return resultobj;
fail:
return NULL;
} |
I was able to reproduce @classner steps for tensorflow 1.2 with CUDA for pypy3 version 5.8.0-beta0 (Python 3.5.3). In order to do this, $ LD_LIBRARY_PATH=/usr/local/cuda/lib64 pypy3.5
Python 3.5.3 (a37ecfe5f142bc971a86d17305cc5d1d70abec64, Jun 08 2017, 19:43:54)
[PyPy 5.8.0-beta0 with GCC 6.3.0] on linux
>>>> import tensorflow as tf
>>>> x = tf.constant(1)
>>>> y = tf.constant(2)
>>>> z = x * y
>>>> with tf.Session() as sess:
.... sess.run(z)
....
2017-07-20 14:43:47.795964: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-07-20 14:43:47.796979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.936
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.67GiB
2017-07-20 14:43:47.796992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-07-20 14:43:47.796995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: Y
2017-07-20 14:43:47.797003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/home/andrew/.virtualenvs/tensorflow_pypy/site-packages/tensorflow/python/client/session.py", line 780, in run
run_metadata_ptr = tf_session.TF_NewBuffer()
TypeError: argument after * must be an iterable, not NoneType It seems that the first line in |
I'm quite interested in this working under pypy. I didn't have any luck with CNTK either but I've discovered that Theano actually works under pypy (with some minor hacks), so for now, it seems I'll be using that instead. Edit: Patches are now upstream for anyone interested. |
I think the |
I think this issue is closed by mistake. The #252 referenced in 913871c's message is actually tensorflow/mlir#252, not this issue. |
@mattip , was able to get it working with tensorflow 1.14.0 Here, i ran into this error: Do you suggest any pending prs to fix this? |
@arjunc77 that sounds familiar: looking higher in the thread I see a similar comment. Could you make sure you are using SWIG > 4.0.0 to generate the wrappers? |
I finally made my pypy compatible in tensorflow. It works good but tf.conv layer is still not working yet in my pypy |
@SungmanHong can you share the formula? What did you need to do? |
Could you please help me on that? I also desperately want my tensorflow to be compatible with pypy <33 Thank you! |
What is the status of this issue ? Is there any progress ? |
@SungmanHong
Could you share receipt how you did this ? Or maybe article ? |
@mrry Do you have some plan to support PyPy ? |
We have removed SWIG, so there's a chance it works now? We do not
officially support it, but maybe we're lucky.
|
@martinwicke I have tried a few seconds ago to install tensorflow on PyPy and it still does not work ... |
@redradist: did you try with the latest nightly from http://buildbot.pypy.org/nightly/py3.6 and latest HEAD of tensorflow? Was the error similar to the one above? |
@mattip I have tried and got the following error: WARNING: Download from https://mirror.bazel.build/github.com/aws/aws-sdk-cpp/archive/1.7.336.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'aws':
java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed
ERROR: /mnt/i/tensorflow/tensorflow/tools/pip_package/BUILD:171:1: //tensorflow/tools/pip_package:licenses depends on @aws//:LICENSE in repository @aws which failed to fetch. no such package '@aws//': java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: no such package '@aws//': java.io.IOException: Error extracting /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws/1.7.336.tar.gz to /home/redra/.cache/bazel/_bazel_redra/41d86d998710872df1eae1e2464f95b2/external/aws: writing file failed |
Are you sure this error is pypy-specific? |
@mattip No, this error relates to TensorFlow build from master HEAD |
OK, my directions should have been "latest pypy3.6 nightly and latest tensorflow that successfully builds with cpython3.6" |
Does anyone have any specific instructions for making this work? I tried both version 2.3.1 (my desired version) and trying to pip install from the git repo and got this message:
I'm using pypy3.6-v7.3.3-linux64.tar.bz2 |
Hello everyone,
But I have been having issues. I have followed this thread and some other issue posts, but I remain unable to advance. First off, running
I then tried to compile tensorflow from source in the PyPy venv and, after installing the requisites and configuring the environment, the following error log occurs during
As @martinwicke mentioned, since SWIG was removed I did not apply any fix to pybind like mentioned in pybind/pybind11#2146. I have also tried to install it by using the google storage links provided in the tensorflow installation docs:
but I get the error:
I have also tried changing from manylinux2010 to manylinux1 to solve this, but it also did not work. Looking for most of these issues online leads me to 64 and 32 bit conflicts, but I have made sure that I am running the proper 64bit version. I have also tried doing the pip install with the latest tensorflow git head as source. I dont get any output after cloning and tensorflow remains not installed.
I have come across several installation issues with other libraries in pypy, but I have usually been able to resolve them either by compiling from source or falling back to a specific version. Tensorflow is being rather stuborn. Something of note is that I am able to If I freeze pip, tensorflow does not appear in the list. |
You will not find a binary package available for PyPy, so the attempts to do Since there are no binary packages, you will have to build from source. The relevant information in your analysis is:
|
Hi, could you share your method? |
This comment was marked as spam.
This comment was marked as spam.
This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you. |
@mattip @orangelash Quick ping on the status of PyPy: the call |
I know it's not a priority and will be a long way to get there; but making TF compatible with PyPy woud be super cool.
Thoughts?
The text was updated successfully, but these errors were encountered: