Binding Behavior Updates

[williamhbaker]

Discussed in #1219

These changes enable bindings for captures, derivations, and materializations to work in a consistent way with respect to "resetting" the state of a binding and re-starting its backfill when a binding is removed from a spec, as well as adding capabilities for a binding to be "reset" without removing it from the spec via a new backfillVersion property. Additionally, they will allow materializations to drop & re-create the respective materialized table (or equivalent) when a materialization binding is reset.

• New backfillVersion and bindingKey concepts:
  • Add backfillVersion to capture/derivation/materialization spec and bindingKey to protocol
  • Implement automatic pruning of driver & runtime checkpoints for bindingKey's that are removed from task specs
• Captures: Use bindingKey for keying binding state (depends on spec updates, above)
• Graceful shutdown of connectors:
  • Make connector-init propagate shutdown signals to connectors and give them a chance to exit gracefully
  • Update materializations to clean up after themselves when signaled to exit (related: SQL warehouse materializations don't do a very good job of cleaning up after themselves connectors#984) - for this scope of work, most relevant for those that track column-like type information and produce constraints for existing tables
• Materializations drop & re-create tables (depends on graceful shutdown of connectors, above):
  • Update "activate" logic to disable materialization shards before applying changes (when needed), and re-enable them after changes have been applied
  • Make materializations drop tables that were previously being materialized when the target tables have a different bindingKey
• Add temporary migration shim which populates state_key on behalf of connectors even for captures / materializations built without it.
• Enforce that backfill counters are always monotonically increasing in the control plane
• Use "asynchronous", runtime-driven strategy for Apply operations

[jgraettinger]

#1292 is up with implementation for backfill counters and threading through a persistent state_key for connector use.

Implement automatic pruning of driver & runtime checkpoints for bindingKey's that are removed from task specs

I did not implement this part yet (though it doesn't seem strictly required to move this feature forward).

Since our initial design conversations, an improvement made as part of the runtime refactors is that handling of connector states is now fully incremental: we don't re-write the complete connector state with each transaction, we only write the delta change. I'd like to do the same thing with runtime checkpoints.

An implication is that there isn't a lot of downside that I can see in simply not pruning these states, which also means we can avoid explicit coordination between the runtime and the connector on precisely where state keys live within the overall connector state.