sql: asynchronous [CREATE|DROP] INDEX

[petermattis]

See #2035. Adding an index will be a multi-step process:

• Adding the index to the TableDescriptor but marking it as write-only.
• Waiting for the TableDescriptor change to propagate to all of the nodes.
• Backfilling the index with some sort of mapreduce over the table.
• Marking the index as read-write in the TableDescriptor.

Note that the backfilling of a unique index might detect a duplicate key violation requiring the whole process to be rolled back.

Dropping an index will involve:

• Marking the index as deleting in the TableDescriptor.
• Waiting for the TableDescriptor change to propagate to all of the nodes.
• Deleting the index keys using a DelRange operation.
• Removing the index from the TableDescriptor.

[petermattis]

I think we're going to need parts of this sooner than I expected. We currently can't use the gossiped table descriptors safely in almost any circumstance and not using the gossiped descriptors is a significant performance hit (at least 1 and perhaps 2 extra reads). We can't use the gossiped descriptors in {INSERT,UPDATE,DELETE} operations because those operations might be using an old descriptor and thus not adding index entries or default values. We can't use the gossiped descriptors in SELECT operations because an old descriptor might refer to an index/column that has been dropped.

The challenge is to know when a change to a descriptor has been propagated to all of the nodes using that descriptor. F1 solves this by using leases on the order of minutes and making sure a write operation never commits after the version of the table descriptor it is using is more than 1 version old. It would be unfortunate for tests if a CREATE INDEX took minutes even when there was no index data to backfill.

[tbg]

I think we'll be able to find a solution to keep tests running smoothly no matter how we go about it (at least that shouldn't concern us as much). Having leases on the schema seems like it was going to happen eventually anyways (or are there any viable alternatives?) - in that case, we could put something together that skips the lease timeout (i.e. when a schema change is issued and commits, we advance the clock or manually invalidate the cache, the latter likely being preferable).

[petermattis]

Yeah, we can always fix the test issue by advancing time, but I think we'll need to fix this for a real system too. It would be highly unexpected for someone to start a test cluster, create a table and then try to create an index and have that CREATE INDEX statement take minutes to complete. I have a half-baked idea for how we can address this. I'll try and find some time today or this weekend to write it up as an RFC.

[tbg]

I have some half-baked ideas too. Will wait for your RFC.

[tamird]

The remaining work here is tracked by #2845.