Add support for cloud tracing

[sduskis]

A user worked out how to use cloud tracing with cloud bigtable on stack overflow.
Incorporate this into the cloud bigtable client.

[ankurcha]

Having some kind of a callback(s) on each backend rpc would be very helpful. Currently the entire thing looks like a single big rpc but knowing the individual rpc time would help diagnose hotspots.

[sduskis]

I don't know much about tracing, which is why this wasn't built yet. Any pointers would be appreciated.

FYI, we do have quite a bit of performance metrics: https://cloud.google.com/bigtable/docs/hbase-metrics

[ankurcha]

From what i understand about the way bigtable grpc client is being used (please correct me if I am wrong), the getScanner() returns an iterator which fetches rows in batches (by doing an rpc) to the bigtable instance. The best/easiest/most flexible way would be to allow the user to specify a callback during the ReadRowsRequest generation (by the HBASE_ADAPTER) and then another callback which is fired when the ReadRowsResponse is returned. This would probably be a good first step.

Following is a presentation for htrace-hbase which does the tracing for hbase: http://www.slideshare.net/hadoopxnttdata/htrace-hbase-meetup-nyc-20141015

[sduskis]

Scans have a streaming RPC, meaning 1 request for a range of rows, and the server responds in multiple batches.

We can definitely work in a callback mechanism for this. The easiest way for us is via the metrics framework mentioned above. We currently track time to first response in the scanner, but we could also add another Timers for time between ReadRowsResponses.

I'll also read up on htrace-hbase. Thanks for the pointer.

[sduskis]

FYI, we are working on an integration with opencensus-java, which will implement tracing.

[sduskis]

This is implemented