[SPARK-20647][core] Port StorageTab to the new UI backend. #19679
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This required adding information about StreamBlockId to the store, which is not available yet via the API. So an internal type was added until there's a need to expose that information in the API. The UI only lists RDDs that have cached partitions, and that information wasn't being correctly captured in the listener, so that's also fixed, along with some minor (internal) API adjustments so that the UI can get the correct data. Because of the way partitions are cached, some optimizations w.r.t. how often the data is flushed to the store could not be applied to this code; because of that, some different ways to make the code more performant were added to the data structures tracking RDD blocks, with the goal of avoiding expensive copies when lots of blocks are being updated. Tested with existing and updated unit tests.
|
For context:
|
|
Note that unit tests will fail; they depend on changes being added in #19678. I'm posting this to speed up the review process. |
|
Test build #83509 has finished for PR 19679 at commit
|
squito
approved these changes
Nov 7, 2017
| @@ -685,12 +700,55 @@ class AppStatusListenerSuite extends SparkFunSuite with BeforeAndAfter { | |||
| assert(exec.info.diskUsed === 0L) | |||
| } | |||
|
|
|||
| // Add a block from a different RDD. Verify the executor is updated correctly and also that | |||
| // the distribution data for rdd1 is up to date. | |||
There was a problem hiding this comment.
you're only checking the distribution data for rdd2, right?
|
I figured out what is going on with StreamBlocks now, both before and after this change, so I'm OK with it |
|
Test build #83560 has finished for PR 19679 at commit
|
|
Test build #83604 has finished for PR 19679 at commit
|
|
Test build #83619 has finished for PR 19679 at commit
|
|
merged to master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This required adding information about StreamBlockId to the store,
which is not available yet via the API. So an internal type was added
until there's a need to expose that information in the API.
The UI only lists RDDs that have cached partitions, and that information
wasn't being correctly captured in the listener, so that's also fixed,
along with some minor (internal) API adjustments so that the UI can
get the correct data.
Because of the way partitions are cached, some optimizations w.r.t. how
often the data is flushed to the store could not be applied to this code;
because of that, some different ways to make the code more performant
were added to the data structures tracking RDD blocks, with the goal of
avoiding expensive copies when lots of blocks are being updated.
Tested with existing and updated unit tests.