roachpb: investigate disabling roachpb.Value checksum computation

[petermattis]

#21395 disabled verification of the roachpb.Value checksums, providing a 15-20% speed boost to large MVCC scans. Checksums are still computed when values are written, they are just not verified. The justification for removing the verification is that TLS and RocksDB checksums already provide adequate coverage. RocksDB checksums have caught disk corruption in the wild. The roachpb.Value checksums never have.

Computation of the checksums was left in because disabling them may be problematic. We'd need to verify that checksums are never computed below Raft or perform a migration. But before doing any of that work, we should verify that disabling the checksum provides a speed boost. Workloads to investigate are simple KV operations are bulk insert operations.

@nvanbenschoten for triage.

Jira issue: CRDB-5886

[nvb]

I did some experimentation with this, both by simply disabling checksum computation and keeping the first 4 bytes in each value empty in nvanbenschoten/someChecksum and by removing the entire 4-byte prefix of each value entirely in nvanbenschoten/noChecksum. I then ran kv -concurrency=32 -read-percent=0 against a local single-node cluster. This produced the following results after three trials in each configuration:

	computing checksum	leaving checksum empty	removing checksum prefix
avg. latency (ms)	7.7	7.666666667	7.4
latency delta	-	-0.4329004329%	-3.896103896%
avg. throughput (ops/sec)	4162.533333	4186.533333	4327.866667
throughput delta	-	+0.5765719594%	+3.971940165%

While leaving the checksum field in roachpb.Values empty is a simple change, it doesn't provide much of a speedup at all. Because of the minimal performance impact and because it may put us in a difficult situation if we decide to turn on checksum verification in the future, it's probably not worth doing.

On the other hand, removing the entire 4-byte checksum field from roachpb.Values does provide a relatively significant speedup. Unfortunately, doing this in a backward compatible manner is much more challenging. roachpb.Values are not versioned, so in order to change their header size, we'd need to perform some kind of migration. The migration could look like:

1. wait for a new cluster version
2. add a version prefix to all values
3. begin writing new values with a new version and without a checksum
4. migrate all existing values to the new version

This seems like a pretty big change and undoubtedly has some nasty subtleties, so I don't think there's much to do here for 2.0.

[petermattis]

Heh, I was about to take a look at this. I think it makes sense to disabling checksum computation completely, but leave the 4-byte checksum field blank. Note that the 0-value for checksum indicates "checksum uninitialized", so I don't think it will be problematic to turn on later.

Also, we're still verifying the checksums when a ScanResponse is returned over the wire via ScanResponse.Verify -> Value.Verify.

[nvb]

If this doesn't complicate turning checksum verification back on then I'm fine with disabling the checksum computation. As you mentioned, the only complication here would be if we created roachpb.Values beneath Raft. I didn't find any instance of us doing this when I first looked, but I'll check again.

[petermattis]

The below Raft thing might be a deal breaker. I think you should try to hook into the below-raft protos test to really be sure we're never calling Value.InitChecksum below Raft.

[nvb]

TestBelowRaftProtos doesn't allow roachpb.Values to be marshaled beneath Raft, so that alone should prove that we don't call Value.InitChecksum for anything that could cause inconsistencies, right? Is there more we'd want to assert here?

A grep shows that the only call to Value.InitChecksum near Raft is in Replica.append, but this is called on Raft log entries, which are unreplicated, so this shouldn't be a problem.

Note that the 0-value for checksum indicates "checksum uninitialized", so I don't think it will be problematic to turn on later.

This also means that the change won't even require a new cluster version, since any uninitialized checksums are simply ignored.

[petermattis]

TestBelowRaftProtos doesn't allow roachpb.Values to be marshaled beneath Raft, so that alone should prove that we don't call Value.InitChecksum for anything that could cause inconsistencies, right? Is there more we'd want to assert here?

We don't marshal roachpb.Value into the values of key/values. Instead we store roachpb.Value.RawBytes directly in the value. So I'm not sure if it shows anything that roachpb.Value is not marshaled below Raft. Rather, I think we need to make any call to Value.InitChecksum act the same as marshaling of roachpb.Value and use the below raft protos infrastructure to verify there are no such calls below Raft.

This also means that the change won't even require a new cluster version, since any uninitialized checksums are simply ignored.

Yep.

[nvb]

We don't marshal roachpb.Value into the values of key/values. Instead we store roachpb.Value.RawBytes directly in the value.

Good point. Hooking into Value.InitChecksum directly shows that it is called twice beneath Raft, both through MVCCPutProto.

The first caller of MVCCPutProto is StateLoader.SetHardState. This one is fine because the RaftHardStateKey is unreplicated.

The second caller of MVCCPutProto is StateLoader.SetMVCCStats. This is more problematic because the RangeStatsKey is replicated, so changing things here naively would cause issues with the consistency checker. That said, we already have other changes that are going to require us to massage the MVCCStats while computing checksums. We should be able to wrap the changes up together, migrating the v2 stats key if present and stripping any checksum that may exist from either version before performing the checksum. This would allow us to make the rest of the proposed change here without any risk of inconsistent state.