Add Datastore Tests

[aschmahmann]

Uses the Datastore TestSuite to the test the Measure datastore. There are issues that it already identifies including, that this datastore does not properly implement the Batch interface.

I'm happy to fix the Batching in this PR as long as we come up with a reasonable solution.

Problem:

• Currently creating a Batch separately tracks Puts + Deletes. Additionally, when Committing a Batch we separately commit the Deletes and then the Puts. This means that Put(x) followed by Delete(x) will result in a Put(x) to the underlying Datastore

Possible Solution:

1. Track the result of Puts + Deletes
   a) Use a single Batch, from the passed in datastore, for Puts + Deletes and keep an accounting of the keys
   b) Use an internal map (or map datastore) for the batch
2. Measurements
   a) Only log the net results (e.g. do not log a Put(x) for a Put(x) + Delete(x), after Put(x, val1) + Put(x,val2) only log the size of val2)
   b) Log the total number of puts + deletes (including duplicates), but only add latency observations for the net results
   c) Log all puts + deletes and latency observations for each of them

I'm not sure what metrics we actually want to get out of a Batch, since according to the documentation "Batches do NOT have transactional semantics". For now doing 1a + 2a seems reasonable, but I'm up for alternative approaches.

EDIT: From @Stebalien on IRC it sounds like we don't really care about logging the Puts/Gets that go into Batches, so for now we're just going to setup a separate metric for Batching.

[Stebalien]

gofmt?

[Stebalien]

I'd say 1a and 2b. We're going to get pretty wonky per-key latencies otherwise given that many datastores will rate-limit batch commits.

Ideally, we'd track tuples of "(time, deletes, puts)" but I'm not sure how to do that with our current metrics library. I'm fine punting that for now.

We still ocationally do individual puts/deletes (I think?). Those should give us an accurate measurement of expected latency (which is all we really care about).

[Stebalien]

We'll need to update CI to go 1.12.

[aschmahmann]

@Stebalien how does this version look? I separately recorded all puts, deletes and commits from batches so we can figure out what to do with them later.

Getting other information can be a bit tricky. For example, Put(x, largeData) followed by Put(x, smallData) and Commit() will end up with smallData on disk, but bytes written might be largeData+smallData or just smallData.

[aschmahmann]

@Stebalien Also, how much of the gx stuff should I remove? Just the CI build, or also the .gx folder, json.package and Makefile?

[Stebalien]

@Stebalien Also, how much of the gx stuff should I remove? Just the CI build, or also the .gx folder, json.package and Makefile?

All of it except .gx.

[Stebalien]

I'm not sure if batch delete/put latencies are worth tracking but it doesn't hurt.

Once GX stuff is removed and CI passes, LGTM!