Avoid thundering herd when SoftwareProcess rebinds its sensors #1318
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Applies a random delay of up to 10s before an entity calls connectSensors(). Without the delay all entities attempt to reconnect at once, which regularly causes SSH connection refused errors if several of the entities are running on the same machine. This doesn't guarantee to avoid the problem but makes it much less likely.
|
Brooklyn Central » brooklyn #2058 FAILURE |
|
Think it's a buildhive issue: |
| @@ -54,7 +56,9 @@ | |||
|
|
|||
| private static final SoftwareProcessDriverLifecycleEffectorTasks LIFECYCLE_TASKS = | |||
| new SoftwareProcessDriverLifecycleEffectorTasks(); | |||
|
|
|||
|
|
|||
| private static final long MAX_REBIND_SENSOR_CONNECT_DELAY = Duration.TEN_SECONDS.toMilliseconds(); | |||
There was a problem hiding this comment.
Can we make this a ConfigKey instead. Either a Long or Duration and also a Boolean to enable and disable the feature.
Set config key to null or to == Duration.ZERO to connect sensors immediately.
|
@grkvlt I've altered it so that there's one ConfigKey that defaults to Duration.TEN_SECONDS. If the key is set to null or to a value == Duration.ZERO then no delay occurs. Acceptable? |
|
@sjcorbett Yep, that's what I was thinking of. Love the description of the issue! |
|
So, I didn't realise thundering herd was the actual name of the problem. The wiki page also references the sleeping barber problem, and of course we also have the byzantine generals - distributed computing is fun! |
|
Brooklyn Central » brooklyn #2063 SUCCESS |
grkvlt
pushed a commit
that referenced
this pull request
Apr 11, 2014
Avoid thundering herd when SoftwareProcess rebinds its sensors
|
Merged 🐄 🐄 🐄 🐄 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Applies a random delay of up to 10s before a rebinding SoftwareProcess entity calls connectSensors(). Without the delay all entities attempt to reconnect at once, which reliably causes SSH connection refused errors if several of the entities are running on the same machine. This doesn't guarantee to prevent refused connections but makes them much less likely.