bug: tend to delete not-exist sstable after comaction

[MrCroxx]

A not-exist sst is tend to be deleted after comapction.

2022-05-16T15:13:15.277677Z ERROR runkv_rudder::worker::compaction_detector: trigger compaction l0 error: other: compaction error: [Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697679 not exists, diff: VersionDiff { id: 696, sstable_diffs: [SstableDiff { id: 68719476789, level: 0, op: Delete, data_size: 68762 }, SstableDiff { id: 68719476790, level: 0, op: Delete, data_size: 67887 }, SstableDiff { id: 68719476791, level: 0, op: Delete, data_size: 69175 }, SstableDiff { id: 68719476792, level: 0, op: Delete, data_size: 68799 }, SstableDiff { id: 433791697679, level: 1, op: Delete, data_size: 33857 }, SstableDiff { id: 433791697684, level: 1, op: Delete, data_size: 33266 }, SstableDiff { id: 433791697689, level: 1, op: Delete, data_size: 34152 }, SstableDiff { id: 433791697695, level: 1, op: Delete, data_size: 33991 }, SstableDiff { id: 433791697699, level: 1, op: Delete, data_size: 33625 }, SstableDiff { id: 433791697704, level: 1, op: Delete, data_size: 34298 }, SstableDiff { id: 433791697708, level: 1, op: Delete, data_size: 3211975 }, SstableDiff { id: 433791697750, level: 1, op: Insert, data_size: 36418 }, SstableDiff { id: 433791697757, level: 1, op: Insert, data_size: 35829 }, SstableDiff { id: 433791697764, level: 1, op: Insert, data_size: 36713 }, SstableDiff { id: 433791697768, level: 1, op: Insert, data_size: 36553 }, SstableDiff { id: 433791697772, level: 1, op: Insert, data_size: 36059 }, SstableDiff { id: 433791697776, level: 1, op: Insert, data_size: 36856 }, SstableDiff { id: 433791697779, level: 1, op: Insert, data_size: 3458003 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697685 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 107374182453, level: 0, op: Delete, data_size: 68025 }, SstableDiff { id: 107374182454, level: 0, op: Delete, data_size: 68593 }, SstableDiff { id: 107374182455, level: 0, op: Delete, data_size: 68662 }, SstableDiff { id: 107374182456, level: 0, op: Delete, data_size: 68806 }, SstableDiff { id: 433791697680, level: 1, op: Delete, data_size: 34008 }, SstableDiff { id: 433791697685, level: 1, op: Delete, data_size: 34511 }, SstableDiff { id: 433791697690, level: 1, op: Delete, data_size: 34004 }, SstableDiff { id: 433791697694, level: 1, op: Delete, data_size: 34295 }, SstableDiff { id: 433791697701, level: 1, op: Delete, data_size: 33997 }, SstableDiff { id: 433791697705, level: 1, op: Delete, data_size: 34002 }, SstableDiff { id: 433791697709, level: 1, op: Delete, data_size: 34281 }, SstableDiff { id: 433791697713, level: 1, op: Delete, data_size: 33783 }, SstableDiff { id: 433791697715, level: 1, op: Delete, data_size: 34362 }, SstableDiff { id: 433791697717, level: 1, op: Delete, data_size: 3109571 }, SstableDiff { id: 433791697752, level: 1, op: Insert, data_size: 36841 }, SstableDiff { id: 433791697759, level: 1, op: Insert, data_size: 37365 }, SstableDiff { id: 433791697762, level: 1, op: Insert, data_size: 36705 }, SstableDiff { id: 433791697767, level: 1, op: Insert, data_size: 36928 }, SstableDiff { id: 433791697770, level: 1, op: Insert, data_size: 36710 }, SstableDiff { id: 433791697775, level: 1, op: Insert, data_size: 36565 }, SstableDiff { id: 433791697778, level: 1, op: Insert, data_size: 36848 }, SstableDiff { id: 433791697781, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697782, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697784, level: 1, op: Insert, data_size: 3346709 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697692 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 120259084341, level: 0, op: Delete, data_size: 68268 }, SstableDiff { id: 120259084342, level: 0, op: Delete, data_size: 68832 }, SstableDiff { id: 120259084343, level: 0, op: Delete, data_size: 68050 }, SstableDiff { id: 120259084344, level: 0, op: Delete, data_size: 69146 }, SstableDiff { id: 433791697681, level: 1, op: Delete, data_size: 33993 }, SstableDiff { id: 433791697688, level: 1, op: Delete, data_size: 33634 }, SstableDiff { id: 433791697692, level: 1, op: Delete, data_size: 34287 }, SstableDiff { id: 433791697696, level: 1, op: Delete, data_size: 33994 }, SstableDiff { id: 433791697700, level: 1, op: Delete, data_size: 34297 }, SstableDiff { id: 433791697703, level: 1, op: Delete, data_size: 33773 }, SstableDiff { id: 433791697707, level: 1, op: Delete, data_size: 33855 }, SstableDiff { id: 433791697710, level: 1, op: Delete, data_size: 34293 }, SstableDiff { id: 433791697712, level: 1, op: Delete, data_size: 33852 }, SstableDiff { id: 433791697714, level: 1, op: Delete, data_size: 34500 }, SstableDiff { id: 433791697716, level: 1, op: Delete, data_size: 3075627 }, SstableDiff { id: 433791697754, level: 1, op: Insert, data_size: 36557 }, SstableDiff { id: 433791697758, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697763, level: 1, op: Insert, data_size: 36849 }, SstableDiff { id: 433791697766, level: 1, op: Insert, data_size: 36421 }, SstableDiff { id: 433791697771, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697774, level: 1, op: Insert, data_size: 36415 }, SstableDiff { id: 433791697777, level: 1, op: Insert, data_size: 36416 }, SstableDiff { id: 433791697780, level: 1, op: Insert, data_size: 36926 }, SstableDiff { id: 433791697783, level: 1, op: Insert, data_size: 36562 }, SstableDiff { id: 433791697785, level: 1, op: Insert, data_size: 37073 }, SstableDiff { id: 433791697786, level: 1, op: Insert, data_size: 3311271 }] }"))))]

#149 may be related to this bug.