Skip to content
This repository has been archived by the owner. It is now read-only.

Switch to CMS GC #133

Merged
merged 1 commit into from Aug 24, 2017

Conversation

Projects
None yet
3 participants
@c-w
Copy link
Member

commented Aug 24, 2017

By default, Java 8 uses Parallel GC. This is not desirable for streaming applications as it can lead to stop-the-world GC pauses which leads to unpredictable batch times.

For a streaming application, we want predictable times so that our events don't get backlogged. As such, using Concurrent Mark and Sweep GC is more suited since it is overall more expensive but has fewer large pauses.

You can read more about this topic here.

Switch to CMS GC
By default, Java 8 uses Parallel GC. This is not desirable for streaming
applications as it can lead to stop-the-world GC pauses which leads to
unpredictable batch times. For a streaming application, we want
predictable times so that our events don't get backlogged. As such,
using Concurrent Mark and Sweep GC is more suited since it is overall
more expensive but has fewer large pauses.

You can read more about this topic here:
https://github.com/haoyuan/spark-tachyon/blob/master/docs/streaming-programming-guide.md#memory-tuning

@c-w c-w requested a review from kevinhartman Aug 24, 2017

@c-w c-w added the in progress label Aug 24, 2017

@c-w c-w requested a review from erikschlegel Aug 24, 2017

@erikschlegel

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2017

LOVE THIS!

@erikschlegel
Copy link
Contributor

left a comment

LGTM

@kevinhartman

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2017

I've found a few stages that this may improve:

screen shot 2017-08-24 at 4 36 15 pm

The 0.2s for these tasks is coming from GC.

@c-w c-w merged commit 49e09eb into master Aug 24, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@c-w c-w deleted the spark-gc branch Aug 24, 2017

@c-w c-w removed the in progress label Aug 24, 2017

rachelnicole pushed a commit that referenced this pull request Jan 24, 2018

Merge pull request #133 from CatalystCode/keep-only-granular-locations
Only keep most granular location for a given name

rachelnicole pushed a commit that referenced this pull request Jan 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.