Skip to content
This repository has been archived by the owner. It is now read-only.

Switch to CMS GC #133

Merged
merged 1 commit into from Aug 24, 2017
Merged

Switch to CMS GC #133

merged 1 commit into from Aug 24, 2017

Conversation

@c-w
Copy link
Contributor

@c-w c-w commented Aug 24, 2017

By default, Java 8 uses Parallel GC. This is not desirable for streaming applications as it can lead to stop-the-world GC pauses which leads to unpredictable batch times.

For a streaming application, we want predictable times so that our events don't get backlogged. As such, using Concurrent Mark and Sweep GC is more suited since it is overall more expensive but has fewer large pauses.

You can read more about this topic here.

By default, Java 8 uses Parallel GC. This is not desirable for streaming
applications as it can lead to stop-the-world GC pauses which leads to
unpredictable batch times. For a streaming application, we want
predictable times so that our events don't get backlogged. As such,
using Concurrent Mark and Sweep GC is more suited since it is overall
more expensive but has fewer large pauses.

You can read more about this topic here:
https://github.com/haoyuan/spark-tachyon/blob/master/docs/streaming-programming-guide.md#memory-tuning
@c-w c-w requested a review from kevinhartman Aug 24, 2017
@c-w c-w requested a review from erikschlegel Aug 24, 2017
@erikschlegel
Copy link
Contributor

@erikschlegel erikschlegel commented Aug 24, 2017

LOVE THIS!

Copy link
Contributor

@erikschlegel erikschlegel left a comment

LGTM

@kevinhartman
Copy link
Contributor

@kevinhartman kevinhartman commented Aug 24, 2017

I've found a few stages that this may improve:

screen shot 2017-08-24 at 4 36 15 pm

The 0.2s for these tasks is coming from GC.

@c-w c-w merged commit 49e09eb into master Aug 24, 2017
2 checks passed
@c-w c-w deleted the spark-gc branch Aug 24, 2017
@c-w c-w removed the in progress label Aug 24, 2017
rachelnicole pushed a commit that referenced this issue Jan 24, 2018
Only keep most granular location for a given name
rachelnicole pushed a commit that referenced this issue Jan 24, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants