YARN-6969. Remove method getMinShareMemoryFraction and getPendingCont… #260
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ainers in class FairSchedulerQueueInfo
|
💔 -1 overall
This message was automatically generated. |
|
https://issues.apache.org/jira/browse/YARN-6969 has been fixed. Closing. |
|
💔 -1 overall
This message was automatically generated. |
shanthoosh
added a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
…StreamProcessorSession Fix flaky and slow integration tests in TestZkStreamProcessor and TestZkStreamProcessorSession Reason for failures: There’re three configurable wait times in rebalancing phase in samza standalone before consensus is acheived and processing resumes with updated jobModel. * debounceTime (Specified by `job.debounce.time.ms`. Upon processor change, leader waits for this interval before generating jobModel expecting stabilization in processors group(new arrival, deletion etc)). * taskShutdownMs (Specified by `task.shutdown.ms`. Wait time for SamzaContainer shutdown in StreamProcessor). * barrierWaitTimeOutMs (Specified by `job.coordinator.zk.consensus.timeout.ms`. Wait time for all processors in the group to join the barrier after creation). Above wait times affects rebalancing phase duration. All these wait time have defaults in order of 40-60 seconds and not set to low values. Flaky tests expects processors to come back up after rebalancing phase and drain message sources(Accomplished by checking a latch.count. RemoteApplicationRunner integration tests does exact same thing by checking if kafka input queue is drained directly with similar logic). In worst case rebalancing phases can last upto 3-4 minutes(Making these tests sometime take 10 minutes at worst case). Change: Set all the above timeouts to 2 seconds(Sufficient for tests and verified by local build). Benefits: * Faster build time(Average runtime of these individual tests were reduced from 1m56s to 14s) * More predicability in assertions(Didn’t fail even once in 30-40 attempts locally). NOTE: If this doesn’t fix TestZkStreamProcessor and TestZkStreamProcessorSession, longer term fix should be to use message markers in input source and shutdown taskCoordinator upon receiving them from TaskImpl(Or use bounded collection based pluggable InMemorySystemConsumer/InMemorySystemProducer). Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com> Reviewers: Bharath Kumarasubramanian <codin.martial@gmail.com>, Navina Ramesh <navina@apache.org> Closes apache#260 from shanthoosh/FIX_ZK_PROCESSOR_FLAKY_TESTS
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please add me as a contributor in jira so that I can assign this task to me.