Make it easier to install the python driver.

[AtnNn]

As seen in rethinkdb/rethinkdb-www#82, rethinkdb/rethinkdb-www#83 and #1284.

It might make many lives easier if there was an easy way to install the python driver and the python protobuf library with the C++ backend.

Doing so involves a lot of steps that are easy to get wrong. And currently the instructions we have only work with python 2 and protobuf 2.4

pip and setuptools are not intelligent enough to figure out the correct dependencies and execute the complicated build steps that are needed.

I'm going to write a script that would automate the installation, allowing users to possibly do

curl http://rethinkdb.github.io/install-python.sh | bash

[coffeemug]

@AtnNn -- could you explain what the actual issue is?

[AtnNn]

@coffemug sorry I pressed the wrong key.

[neumino]

On Ubuntu 12.04 LTS

That works (on a fresh VM)

sudo apt-get python-dev protobuf libprotobuf-dev python-pip python-protobuf
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
sudo pip install rethinkdb

[neumino]

Note: It was Ubuntu server -- even if it probably doesn't change anything

And I still can't get it working on Arch :'(

[underrun]

oh! okay ... the ubuntu apt package for python-protobuf has the cpp extension built. when i'm trying to install into a virtual environement, the pypi version of protobuf fails to build the cpp extension. and i get this:

*** WARNING: The installed protobuf library does not seem to include the C++ extension
*** WARNING: The RethinkDB driver will fallback to using the pure python implementation

I've also tried building building and installing the protobuf python extension from the protobuf source but to no avail.

on the bright side i can use the cpp protobuf extension on my ubuntu desktop but, alas, still not from my gentoo server.

[neumino]

@underrun on Ubuntu, these instructions should give you the python driver with the fast protobuf library:

sudo apt-get python-dev protobuf libprotobuf-dev python-pip python-protobuf
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
sudo pip install rethinkdb

[AtnNn]

Here are some of the steps that we used to get it to work on OS X:

brew install protobuf
sudo pip uninstall protobuf rethinkdb
tar xf protobuf-2.5.0.tgz
git clone git://github.com/rethinkdb/rethinkdb
cd protobuf-2.5.0/python
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
python setup.py build
sudo python setup.py install
cd ../../rethinkdb/drivers/python
make sdist
sudo pip install ../../build.packages/python/dist/rethinkdb-1.8.0-0.tgz

We retried some of the steps 6 or 7 times in a different order or variation before getting the cpp backend to work.

[underrun]

here's what i had to do to get it to work on gentoo:

# first install protobuf librarires
sudo emerge protobuf
# then uninstall the python package - repeat as necessary
yes | sudo pip uninstall protobuf
# get the version of protobuf gentoo uses
wget https://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz
# unpack and build it but don't install
tar zxf protobuf-2.4.1.tar.gz
cd protobuf-2.4.1
./configure; make
# go into the python dir
cd python
# export the cpp implemenation env var
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
# build the python package with -I and -L flags (necessary for cpp impl)
python setup.py build build_ext -I ../src -L ../src/.libs
# install newly built protobuf python package with cpp extensiton
sudo python setup.py install
# go get rethinkdb and install python module
cd ../../
git clone git://github.com/rethinkdb/rethinkdb
git checkout v1.10.0
cd rethinkdb/drivers/python
make sdist
sudo pip install ../../build/packages/python/dist/rethinkdb-1.10.0-0.tgz

[neumino]

Hum the instructions on Ubuntu works on a fresh VM, but I can't get them work on my server.
I have some strange conflicts...

[AtnNn]

Moving to 1.9.x

I'll start writing this script. Thanks @neumino and @underrun for your logs.

[AtnNn]

Talked to @coffeemug -- here's what we decided to do.

• pip install rethinkdb will not try to install the optimized protobuf backend. This way new users will never encounter this error.
• For people that want the optimized backend, we'll have separate instructions

[robert-zaremba]

That looks like an ugly workaround.
What about this: while people will try to install rethinkdb using pip they will get a warning that this driver doesn't use optimized protobuf backend.

[underrun]

@AtnNn that is unfortunate as I am almost certain that those who will want the "optimized backend" is 100% of your python users.

performance and resource utilization on the python client side lag other nosql options like mongo even with the cpp version of protobuf. without it i just can't see how anyone who wants to use python as a primary interface is going to be interested in rethinkdb. with the cpp protobuf implementation, i feel like the tradeoffs are worth what i gain with rethinkdb, but it is a pain in the butt to build and install.

you guys should take a look at zeromq for an example of how to bend pip to your will ( https://github.com/zeromq/pyzmq ). when installing the python bindings for zeromq, if the libraries are not detected on the system, they are actually built and installed as python extensions. this is by far the most powerful approach i've ever seen as it allows me to test different library version inside virtual environments without needing root access for anything.

please put the effort into making this work with native python package installation even though packaging for python sucks.

[AtnNn]

@underrun I will take a look at pyzmq

[coffeemug]

@underrun -- we'll take a look at zeromq and make this work if we can. However, in our tests the cpp backend doesn't necessarily make that much difference in performance. We've put a lot of effort into improving performance by automatically using json instead of protobufs on insert, and will do the same for data coming back from the server. We'll look at other optimizations too -- unfortunately the cpp backend isn't a panacea for python driver performance problems. (We'll still make the install easy if we can, but not at the expense of confusing new users)

[AtnNn]

The zeromq setup.py is 2/3 the size of the rethinkdb python driver.

[coffeemug]

Ok, let's go with the original plan for now.

[AtnNn]

In RethinkDB 1.11, the python driver will not build the C++ extension by default. It will only build the extension if the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable is set to cpp, which is what the upstream protobuf library does.

Detailed documentation for building the C++ implementation of the python protobuf library is being written: rethinkdb/docs#74

The current python driver can only build the C++ extension for protobuf 2.4. The new driver will also work with protobuf 2.5

The r.protobuf_implementation variable will keep the same meaning.