Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate SegmentStore MaxDirectMemory and Cache settings before startup #6268

Closed
abhinb opened this issue Sep 3, 2021 · 0 comments · Fixed by #6313
Closed

Validate SegmentStore MaxDirectMemory and Cache settings before startup #6268

abhinb opened this issue Sep 3, 2021 · 0 comments · Fixed by #6313

Comments

@abhinb
Copy link
Contributor

abhinb commented Sep 3, 2021

Description
We need to make sure that we validate the MaxDirectMemory and Cache values that we configure for the SegmentStore.
As noted in this link, and additionaly, the below invariant for a segment store container should hold before its startup:-

  • POD_MEM_LIMIT >= JVM Heap + Direct Memory
  • Direct Memory > pravegaservice.cache.size.max ( direct addresable memory by SegemenStore should be greater than the configured cache size preferably 1-2GB more)

If we do not abide by this we can see errors of the form below when SegmentStore starts:-
00:39:19,662 10936089 [core-23] ERROR i.p.s.server.logs.DurableLog - DurableLog[40] Recovery FAILED. io.pravega.segmentstore.storage.cache.CacheFullException: DirectMemoryCache full: Stored = 11788056931, Used = 11811160064, Reserved = 23068672, Allocated = 11811160064, Max = 11811160064

Importance
should-have

Location
SegmentStore container entrypoint script

Suggestions for an improvement
Validate the cache , direct addresable memory and pod memory limit settings before starting up the segment store container and fail fast to avoid issues later when the container starts up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant