You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The cores file client currently persists all events to a single file. This means there is no way to compress and cleanup storage as the file cannot be cleanly split (without a process to read it in and do this). If we make the file client split the file after X events (such that we get reasonable event file size say ~ 100MB uncompressed which should be about 10MB compressed), then it will easy to add a job to cleanup and compress these as we go meaning that, if we cannot always have all event data as it may be too large to store from block 0, we can at least have the most recent few days data which will make it easier to diagnose issues. Related to this, the data-node file reader would need to change to sequentially read in a time sorted collection of event files.
The text was updated successfully, but these errors were encountered:
The cores file client currently persists all events to a single file. This means there is no way to compress and cleanup storage as the file cannot be cleanly split (without a process to read it in and do this). If we make the file client split the file after X events (such that we get reasonable event file size say ~ 100MB uncompressed which should be about 10MB compressed), then it will easy to add a job to cleanup and compress these as we go meaning that, if we cannot always have all event data as it may be too large to store from block 0, we can at least have the most recent few days data which will make it easier to diagnose issues. Related to this, the data-node file reader would need to change to sequentially read in a time sorted collection of event files.
The text was updated successfully, but these errors were encountered: