Shmem6: support incremental loading in memmgr #51
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this is a first step of incremental loading.
also, extract its initialization code into a separate method.
also renamed ZoneDataLoaderHelper to ZoneDataUpdaterHelper, as that's more appropriate (it's a wrapper for ZoneDataUpdater), and diff-based loader is actually updating a zone, rather than fully loading it.
…asses this will make the code more readable by avoiding indirection of functors.
this will make it more readable by reducing the indirection of functor. also, moved construction of MasterLoader from updateRRsets to initUpdate (which has been actually part of initialization, but had been deferred until the actual load time due to internal restriction. thanks to the set of refactoring we don't have such restriction anymore).
…ses. this will help simplify the ZoneDataLoader constructor and make it more readable.
…with it. as a result, the return type of load() can just be ZoneData* (again). The separation will be useful when we introduce incremental loading.
also added a test case, and another, unrelated test related to incremental loading.
also, it now always raises exceptions on error even if catch_error is True, as it's reportedly more pythonic. existing tests were adjusted for these.
this will help perform shutdown or reconfigure more rapidly, even in the middle of loading a very large zone.
|
now merging, will create a separate issue for the review. |
jinmei
added a commit
that referenced
this pull request
Jun 27, 2014
Shmem6: support incremental loading in memmgr
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This set of changes allows memmgr to load zone data incrementally. The primary benefit of this feature is that it can now drop the attempt of loading a very large zone if it's canceled due to datasrc reconfig or shutdown.
It also makes memmgr more responsible while loading. Due to the global locking of python threads, the builder and main threads of memmgr do not actually run in parallel, so unless the load is incremental the main thread could block while the builder thread is loading a large zone.
The first few commits (up to 8ef008d) are just internal refactoring and cleanups of the ZoneDataLoader class. On top of that, commits up to 2a604ea implement the necessary feature to enable the incremental loading in the C++ datasrc library. The rest of the changes are its python wrapper and the use of it from memmgr.