storage: stream SSTs instead of KV_BATCH when sending snapshots

[jeffrey-xiao]

#38932 introduces the logic for incrementally building SSTs on the receiver side. This approach does not get the compression benefit from streaming SSTs when sending snapshots. I did an informal experiment to see the savings we'd get from streaming SSTs instead of KV_BATCH and the results are promising.

For an antagonist workload like kv0 where all the keys and values are random data and we see no benefits in compression, the amount of overhead that streaming SSTs instead of KV_BATCHES is fairly small. For kv0 I saw a 4% increase in the amount of data we were streaming for various range sizes.

For a more realistic workload like tpcc, I saw as much as an 80% reduction in the amount of data we were sending.

It might be worthwhile to send the user range as an SST and the replicated range-id local keys and range-local keys as KV_BATCHES.

Unlike #38932, this change would have to be gated behind a version flag since the logic on the sender side would have to change.

Jira issue: CRDB-5567

[ajwerner]

One question I have here is about the effectiveness of transport level compression. By default outside of roachprod we use compression on our gRPC connections. My hunch is this isn't worth the complexity.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[erikgrinaker]

Reopening this, since it'd be nice to standardize on SSTs for bulk data transport -- we have pretty mature infrastructure to handle SSTs, while the binary Pebble batch tooling in CRDB is mostly used for Raft snapshots and is much more primitive. If we could ingest the SSTs directly it'd likely net some nice performance gains too.

[blathers-crl]

cc @cockroachdb/replication

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!