Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

During tablet recovery filter logs out that do not define tablet. #881

Merged
merged 1 commit into from Jan 10, 2019

Conversation

@keith-turner
Copy link
Contributor

commented Jan 8, 2019

In a situation where a tablet had lots of write ahead logs, log
recovery consumed a lot of memory. The reason for the memory
consumption was that all recovery logs are opened at once. This
means the index for each file and a few key/values from each file
are read into memory.

This patch modifies the code to first open logs one at time to see if
they define the tablet. Then only the logs that define the tablet are
opened at the same time. This has the potential to use less memory
when a subset of the logs define a tablet.

During tablet recovery filter logs out that do not define tablet.
In a situation where a tablet had lots of write ahead logs, log
recovery consumed a lot of memory.  The reason for the memory
consumption was that all recovery logs are opened at once.  This
means the index for each file and a few key/values from each file
are read into memory.

This patch modifies the code to first open logs one at time to see if
they define the tablet. Then only the logs that define the tablet are
opened at the same time.  This has the potential to use less memory
when a subset of the logs define a tablet.
@keith-turner

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2019

Ran full build and all ITs passed.

@keith-turner keith-turner merged commit 577dd9a into apache:1.9 Jan 10, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@keith-turner keith-turner deleted the keith-turner:filter-wals branch Jan 10, 2019

@ctubbsii ctubbsii added this to Done in 1.9.3 Jun 14, 2019

@ctubbsii ctubbsii added this to Done in 2.0.0 Jun 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
4 participants
You can’t perform that action at this time.