Add integrity checks in Pravega (Tier 1) #6804
Labels
area/debuggability
A fix or feature that helps with debuggability Pravega
area/lts
area/segmentstore
area/tier1
kind/feature
New feature that should be added
priority/P1
version/0.13.0
Is your feature request related to a problem? Please describe.
We need a mechanism to check the integrity of data within Pravega ingestion pipeline. Clearly, checking data integrity has a performance impact. For this reason, we may want to use this feature mainly during the development and testing cycles of Pravega. It could be used also in production, but considering the potential impact on performance.
Describe the solution you'd like
Make sure that the data Appended by clients is exactly the same as the one moved to LTS.
SegmentAggregator
reads from the cache before it is written to LTS. If data integrity check fails, we should fail the write and provide information that helps us to debug the issue.Note that the approach taken here may be more biased towards ensuring correctness during development and testing cycles, to proactively detect and correct any potential data corruption issues that could be related to Pravega.
Describe alternatives you've considered
n/a
Additional context
This feature may be very useful to discover data corruption issues proactively during development and testing to avoid them to happen in production.
The text was updated successfully, but these errors were encountered: