Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integrity checks in Pravega (Tier 1) #6804

Closed
RaulGracia opened this issue Jul 11, 2022 · 0 comments · Fixed by #6805
Closed

Add integrity checks in Pravega (Tier 1) #6804

RaulGracia opened this issue Jul 11, 2022 · 0 comments · Fixed by #6805
Assignees
Labels
area/debuggability A fix or feature that helps with debuggability Pravega area/lts area/segmentstore area/tier1 kind/feature New feature that should be added priority/P1 version/0.13.0

Comments

@RaulGracia
Copy link
Contributor

RaulGracia commented Jul 11, 2022

Is your feature request related to a problem? Please describe.
We need a mechanism to check the integrity of data within Pravega ingestion pipeline. Clearly, checking data integrity has a performance impact. For this reason, we may want to use this feature mainly during the development and testing cycles of Pravega. It could be used also in production, but considering the potential impact on performance.

Describe the solution you'd like
Make sure that the data Appended by clients is exactly the same as the one moved to LTS.

  • This entails checking the data that SegmentAggregator reads from the cache before it is written to LTS. If data integrity check fails, we should fail the write and provide information that helps us to debug the issue.

Note that the approach taken here may be more biased towards ensuring correctness during development and testing cycles, to proactively detect and correct any potential data corruption issues that could be related to Pravega.

Describe alternatives you've considered
n/a

Additional context
This feature may be very useful to discover data corruption issues proactively during development and testing to avoid them to happen in production.

@RaulGracia RaulGracia added area/lts area/tier1 area/segmentstore kind/feature New feature that should be added priority/P1 area/debuggability A fix or feature that helps with debuggability Pravega version/0.13.0 labels Jul 11, 2022
@RaulGracia RaulGracia self-assigned this Jul 11, 2022
@RaulGracia RaulGracia changed the title Add integrity checks in Pravega Add integrity checks in Pravega (Tier 1) Jul 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/debuggability A fix or feature that helps with debuggability Pravega area/lts area/segmentstore area/tier1 kind/feature New feature that should be added priority/P1 version/0.13.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant