Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-9968][STREAMING] Reduced time spent within synchronized block to prevent lock starvation #8204

Closed
wants to merge 2 commits into from

Conversation

@tdas
Copy link
Contributor

tdas commented Aug 14, 2015

When the rate limiter is actually limiting the rate at which data is inserted into the buffer, the synchronized block of BlockGenerator.addData stays blocked for long time. This causes the thread switching the buffer and generating blocks (synchronized with addData) to starve and not generate blocks for seconds. The correct solution is to not block on the rate limiter within the synchronized block for adding data to the buffer.

@tdas

This comment has been minimized.

Copy link
Contributor Author

tdas commented Aug 14, 2015

@zsxwing Could you take a look?

@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Aug 14, 2015

Test build #40882 has finished for PR 8204 at commit a73b645.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@@ -106,6 +106,7 @@ private[streaming] class BlockGenerator(
private val blockQueueSize = conf.getInt("spark.streaming.blockQueueSize", 10)
private val blocksForPushing = new ArrayBlockingQueue[Block](blockQueueSize)
private val blockPushingThread = new Thread() { override def run() { keepPushingBlocks() } }
private val dataAddingLock = new Object

This comment has been minimized.

Copy link
@zsxwing

zsxwing Aug 14, 2015

Member

dataAddingLock is not used.

This comment has been minimized.

Copy link
@tdas

tdas Aug 14, 2015

Author Contributor

Right, sorry, I thought of using it but I realized that it changes the locking logic significantly, which I dont want to do.

@zsxwing

This comment has been minimized.

Copy link
Member

zsxwing commented Aug 14, 2015

LGTM except one minor comment

@tdas

This comment has been minimized.

Copy link
Contributor Author

tdas commented Aug 14, 2015

Thanks @zsxwing. I removed the unused variable. Since the tests already passed, I will merge this as soon as this compiles.

@tdas

This comment has been minimized.

Copy link
Contributor Author

tdas commented Aug 14, 2015

Merging this master and branch 1.5.

asfgit pushed a commit that referenced this pull request Aug 14, 2015
… to prevent lock starvation

When the rate limiter is actually limiting the rate at which data is inserted into the buffer, the synchronized block of BlockGenerator.addData stays blocked for long time. This causes the thread switching the buffer and generating blocks (synchronized with addData) to starve and not generate blocks for seconds. The correct solution is to not block on the rate limiter within the synchronized block for adding data to the buffer.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #8204 from tdas/SPARK-9968 and squashes the following commits:

8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block

(cherry picked from commit 18a761e)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 18a761e Aug 14, 2015
@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Aug 14, 2015

Test build #40929 has finished for PR 8204 at commit 8cbcc1b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
CodingCat added a commit to CodingCat/spark that referenced this pull request Aug 17, 2015
… to prevent lock starvation

When the rate limiter is actually limiting the rate at which data is inserted into the buffer, the synchronized block of BlockGenerator.addData stays blocked for long time. This causes the thread switching the buffer and generating blocks (synchronized with addData) to starve and not generate blocks for seconds. The correct solution is to not block on the rate limiter within the synchronized block for adding data to the buffer.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#8204 from tdas/SPARK-9968 and squashes the following commits:

8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.