Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core File Client - Roll files #7517

Closed
ettec opened this issue Feb 7, 2023 · 0 comments · Fixed by #7561 or #7662
Closed

Core File Client - Roll files #7517

ettec opened this issue Feb 7, 2023 · 0 comments · Fixed by #7561 or #7662
Assignees
Labels
datanode enhancement New feature or request

Comments

@ettec
Copy link
Contributor

ettec commented Feb 7, 2023

The cores file client currently persists all events to a single file. This means there is no way to compress and cleanup storage as the file cannot be cleanly split (without a process to read it in and do this). If we make the file client split the file after X events (such that we get reasonable event file size say ~ 100MB uncompressed which should be about 10MB compressed), then it will easy to add a job to cleanup and compress these as we go meaning that, if we cannot always have all event data as it may be too large to store from block 0, we can at least have the most recent few days data which will make it easier to diagnose issues. Related to this, the data-node file reader would need to change to sequentially read in a time sorted collection of event files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datanode enhancement New feature or request
Projects
Archived in project
2 participants