Build out some integration with Cloud Bigtable

[jgeewax]

We want to make users of Happybase "just work", so...some ideas:

Option 1: A wrapped import module

from gcloud.bigtable import happybase

Option 2: Some sort of monkey patching?

from gcloud import bigtable
import happybase
happybase = bigtable.monkey_patch(happybase)

[jgeewax]

/cc @maxluebbe

[jgeewax]

/cc @dhermes @tseaver : Max might have some spare cycles to work on this, figured you guys should chat.

[dhermes]

Relevant docs:
https://happybase.readthedocs.org/en/latest/
https://cloud.google.com/bigtable/docs/

[tseaver]

ISTM that monkey-patching the happybase third-party module would be a poor choice. I think we need to think of this as "write a happybase API emulation layer" instead (which may or may not itself even use the real happybase code).

[dhermes]

Can you elaborate? Why is it a bad idea? (I have yet to look at the happybase code/docs.)

[tseaver]

I think our goal is that they should be able to port scripts easily (ideally, just changing import happytable to from gcloud.bigtable import happytable and adjusting the connection parameters). I don't see interfering with the actual happytable module's internals as a useful way to approach that goal. For instance, users of gcloud.bigtable might need to interact with a real happybase/Thrift backend (e.g., to move or compare data between it and the bigtable dataset).

[jgeewax]

I agree that there may be times where someone would want to talk to Cloud Bigtable and HBase at the same time, so monkey-patching happybase sounds like we'd screw over those multi-tenant situations.

Maybe there's a way to from gcloud import bigtable and then happybase_compatible_client = bigtable.get_happybase_client() which uses the real happybase code, but swaps out a connection of sorts at the lower level...

[tseaver]

The gcloud.bigtable docs say:

Cloud Bigtable works with Hadoop 2.4 and higher and is interface compatible with Apache HBase 1.0.

If that means that it exports the same REST API as HBase / Thrift, then maybe we don't have to do much to enable happybase, beyond ensuring that we set the connection up with gcloud-relevant credentials.

[jgeewax]

This is the interesting piece. It is not wire-protocol compatible. That is -- it doesn't speak Thrift (AFAIK). Instead, they made the Java code compatible, and swapped out the thing that does serialization and sending of requests underneath.

I think @maxluebbe can comment more to confirm.

[dhermes]

Yeah that one is important. I was under the impression from @jgeewax previous comments that the web APIs were the same / close cousins.

[brianolson]

The Java "HBase" client to Bigtable appears to be an emulation layer. A Python client should be built on the same protobuf rpc interface as that:
https://github.com/GoogleCloudPlatform/cloud-bigtable-client/tree/master/bigtable-protos/src/main/proto/google

I'd imagine that Python client having a native mode that just handles the proto-rpc; and then there could be a happybase emulation layer on top of that.

[maxluebbe]

Sorry have been OOO. Some clarifications.

1. Additional work item is modernizing Happybase to 1.0 HBase API.
2. Cloud Bigtable does not support Thrift/Rest interfaces. Part of the work that needs to be done here is decoupling the thrift parts of happybase from the interface provided to the user, so that we can drop in gRPC in place of Thrift.

[jgeewax]

gRPC or Protobuf over HTTP 1.0?

gRPC is not possible right now given the installation process (we need to keep our install to pip install gcloud and can't ask people to do a bunch of Makefile madness to use our library).

[brianolson]

Is it acceptable for gRPC to be an optional install? (and python bigtable might not work without it, but the rest of gcloud-python would)
Maybe spin off the python bigtable client into something not part of 'gcloud-python'? (Seems fair, the Java bigtable client is its own project/repo.)

[jgeewax]

You're telling me there's no way to talk to Cloud Bigtable's API without using a gRPC client? We can't use HTTP 1.0 ? That seems.... really really weird. @maxluebbe @carterpage can you confirm?

[carterpage]

We're designed to be very low latency and high-throughput so all of our focus is on gRPC. There is a REST interface, which comes for free with our envelope, but we don't recommend using for much more than slow admin API calls or ad hoc test queries against the data API. Almost everything in the API is a binary string, including keys and values, so there would be a lot of base64 encoding going on -- a big CPU hit on both sides. We also have not tested streaming scans would work in REST.

[dhermes]

Can we help make the gRPC install process easy?

[jgeewax]

Some references.... (Short answer... not really. The best we have for Linux is sudo apt-get install linuxbrew && sudo brew install grpc)

• Distribute Python gRPC via PyPI grpc/grpc#936
• Consider a "python-grpc" Ubuntu package? grpc/grpc#1165

I believe the issue here is that gRPC requires some crazy stuff in SSL libs and others, which won't make it into mainstream world for quite some time, and users can't have two versions of those libraries the way apt handles packaging (for Linux).

As for @carterpage 's comment...

We're designed to be very low latency and high-throughput so all of our focus is on gRPC. There is a REST interface, which comes for free with our envelope, but we don't recommend using for much more than slow admin API calls or ad hoc test queries against the data API.

HBase today doesn't speak HTTP 2.0 right? So wouldn't the REST-ful HTTP API calls be comparable to existing HBase clusters in latency and HTTP overhead?

[carterpage]

HBase has their own bespoke RPC system that's actually fairly fast. The issue for us is, using Google's plumbing, gRPC is the only option to provide the sort of latencies we need.

RESTful HTTP API calls would be better than running a Thrift server in front of HBase, since that adds an extra hop, but that's a fairly low bar. I think the performance comparison we want to aim for is Happybase against raw HBase. I believe that uses HBase's binary RPC format, so I don't see how we'd match it without gRPC.

[jgeewax]

Got it. We won't be able to "depend" on gRPC via PyPI, so that means in our docs we'll basically have to say "If you want to use Cloud Bigtable via gcloud-python, you'll need to follow these steps to install gRPC on your machine".

This will clearly have more drop-off, but I suspect we already have a somewhat small pool of people using Cloud Bigtable (given the lowest cost product we offer people is $1500 / month), so adding an extra "here install this" hurdle wouldn't be the end of the world.

[dhermes]

Shouldn't the dev exp be even better for the people paying more? (Sorry if that is an unpopular opinion.)

[jgeewax]

Haha - yes, it should. The developer experience in this case is blazing fast managed HBase -- which is pretty incredible (and what you're paying for).

As far as "installing the client", it seems that our problem with packaging gRPC is not really in our control. The library itself (I'm told) depends on some pretty serious changes to libraries which are in the pipeline but not aimed for main-stream for quite some time. The best we can do (again, I'm told) is using linux brew (brew install grpc).

[carterpage]

JJ hit the nail on the head. This is a pretty bleeding edge service, so there are still some rough edges that will get polished as more network stacks are comfortable with HTTP2, etc.. Our typical customer is technically sophisticated and willing to tolerate a little extra effort for the added power. Hopefully we'll have the best of both worlds before long.

[dhermes]

So you're extending existing Python libraries to support HTTP2 or making your own? Why not just ship a custom fork?

Is the issue compiling C extensions for Python or is the issue making new versions of system socket libraries (e.g. things in /usr/lib)? It should still be possible to just point to local paths for things that would typically be found in /usr/lib.

[dhermes]

Some context for Python 3 support in happybase (and deps):

python-happybase/happybase#40
python-happybase/happybase#78
python-happybase/happybase#95

[maxluebbe]

Let's not think about Python 3 at this time. Getting a solid Python 2.x experience is our top priority.

[tseaver]

@maxluebbe the rest of gcloud-python is already Python3-compatible. At a minimum, we need all "hard" dependencies to be compatible as well, so that our CI story stays sane. I guess we could keep our code source-compatible, make happylib a "soft" dependency, and define the wrappers / tests only if it is importable.

[dhermes]

@jgeewax Suggested we use happybase. I'm trying to get a sense of its popularity after @maxluebbe made me step back and ask "why happybase?".

It only has 17 tagged StackOverflow questions (as of June 10 at noon Pacific).

Checking installs is not so small:

$ pip install -U vanity
$ vanity happybase
...
happybase has been downloaded 487933 times!

This is in comparsion to 37,602,602 for boto and 10,308,673 for httplib2, so I'm not sure how to gauge it.

[dhermes]

@jgeewax Small distinction: service accounts work (i.e. they are "good") for the Table Admin API and Data API, just not the Cluster Admin API (here is where they are "no good").

[coryoconnor]

@dmmcerlean @sduskis any ideas?

[lesv]

Try with one of the following scopes:
https://www.googleapis.com/auth/cloud-bigtable.admin
https://www.googleapis.com/auth/cloud-bigtable.admin.cluster

[dhermes]

Trying only one at a time as wells as both at once, it still fails with the message

Project has not enabled the API. Please use Google Developers Console to activate the API for your project.

---

If you like, I've made an all-inclusive gist that makes this easy to run and tweak:
https://gist.github.com/dhermes/d27070c90a9862213a3b

The script main_with_cluster_admin.go uses bigtable.ClusterAdminScope. This value (defined in doc.go) is slightly different than the one you suggested:

https://www.googleapis.com/auth/bigtable.admin.cluster        # In GOLANG
https://www.googleapis.com/auth/cloud-bigtable.admin.cluster  # Your suggestion

You can run it with

make list_clusters

for a service account or

make list_clusters USE_APP_DEFAULT=True

for using the user creds from gcloud login.

[dhermes]

Here is a well-polished jumping off point for Python:
https://gist.github.com/dhermes/2edb97d9581b5ec471eb

Will merge this upstream into https://github.com/dhermes/gcloud-python-bigtable

[lesv]

@AngusDavis -- Please take a look.

[lesv]

I've talked with engineering -- it is probably best not to provide the cluster APIs for now.

[jgeewax]

That doesn't quite make sense. Why not?

[lesv]

gcloud, when acquiring OAuth tokens, use the client id and client secret (from the installed application OAuth flow). This results in gcloud being able to use the API, but not tokens associated with service accounts.

When the Cluster API is fully enabled, it will be usable programmatically. Until then, the recommendation is to use gcloud or cloud console for the infrequent case of creating a cluster and the data and table APIs for the more frequent cases.

So basically, the API is visible to some internal projects and not to everyone else. We're trying to get this exposed via a service acct, it's currently not... hang tight

[dhermes]

Hanging tight; from the outside perspective, service accounts should be prioritized over user accounts since service accounts are used in a deployed application.

[dhermes]

OK here is a new one, I can't even create a cluster through the API cloud console anymore:

Note the "Table Admin API" service is enabled on the project

and as mentioned before there is no Cluster Admin API as a choice to enable:

[lesv]

I've reported it to engineering.

[dhermes]

Thanks. I think it's just my personal (@gmail.com) account. @jgeewax was able to create one through the UI with a @google.com account.

[lesv]

Yep -- it fails on my personal gmail account when I'm not on the corporate network as well.

[lesv]

Fix is rolling out now, should be done in the next few hours.

[dhermes]

Fix for UI or fix for supporting service accounts with Cluster Admin API?

[lesv]

At this point, it should be just like things were last week. We rolled back some changes.

[dhermes]

Gotcha. Thanks.

[dhermes]

@lesv ISTM that it became possible to use service accounts for the Cluster Admin API yesterday. Can you confirm? (Though the API still doesn't show up in the list of APIs to enable on cloud console.)

[lesv]

That might have happened as part of the fix for the problem you found recently -- the engineer I need to talk w/ is out this week -- so, don't take it as a given that it will stay this way. Eventually, it will be correct and you can use them -- but I'm not sure it has happened officially yet.

[lesv]

@dhermes @jgeewax The Cluster Admin API is definitely not supposed to be enabled yet. The cluster proto was open sourced for some testing reasons, but wasn't supposed to be made available externally yet. One reason is that there are some naming issues that have come up, and the proto may have to be changed because of this. That conversation won't take place until next week and we probably won't see any public results for at least a week after that.

[dhermes]

@lesv Thanks. I have the entire Cluster Admin API implemented, including

• an automated generator for the *_pb2.py modules
• a system test suite that covers every API method (except for UndeleteCluster) with real API calls
• a unit test suite with 100% code coverage

---

Though it's not supposed to be enabled yet, any changes will be quickly picked up by the 3 suites mentioned above.

Thanks for the heads up though.

---

PS ISTM that supporting cloud alpha bigtable clusters * would be impossible without enabling the API.

[dhermes]

I have everything implemented except for the stream requests (ReadRows and SampleRowKeys), which I'm not sure how to handle with gRPC Python.

Golang handles it by using a cancel-able stream.

[nathanielmanistaatgoogle]

@dhermes: what's unclear? gRPC Python supports cancellation. (Ping me; I have some time open this afternoon if you'd like to videochat.)

[dhermes]

Nothing unclear (that I know of), I just don't know how to do it. Let's chat.

[jgeewax]

@dhermes : Looks like we're "done enough" with the Bigtable stuff. I'm going to close this out and open a new issue focused on "what exactly do we need to do to make gcloud_bigtable become gcloud.bigtable"

Sound reasonable? (Re-open if not)