Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurably omit offers from hosts that are overloaded #1776

Merged
merged 5 commits into from Apr 11, 2018
Merged

Conversation

@ssalinas
Copy link
Member

@ssalinas ssalinas commented Apr 4, 2018

If, according to the metrics collected from the slave metrics endpoint, a slave is overloaded, omit these offers from scoring and do not schedule tasks there. This feature is configurable via omitOverloadedHosts under the mesos config, and defaults to false/off

Overloaded is currently defined as a load5 > 1.0 to catch ongoing load, or a load1 > 1.5 to catch shorter spikes large enough to be disruptive to tasks on the host.

@JsonIgnore
public boolean isOverloaded() {
// Any host where load5 is > 1 is overloaded. Also consider a higher threshold for load1 to take into account spikes large enough to be disruptive
return systemLoad5Min > 1.0 || systemLoad1Min > 1.5;

This comment has been minimized.

@baconmania

baconmania Apr 4, 2018
Contributor

Do we want to make these values tunable knobs as well?

This comment has been minimized.

@ssalinas

ssalinas Apr 5, 2018
Author Member

yeah was going back and forth on that. Might as well for flexibility. I'll update 👍

@ssalinas ssalinas added the hs_staging label Apr 5, 2018
@baconmania
Copy link
Contributor

@baconmania baconmania commented Apr 5, 2018

🚢

@ssalinas ssalinas added hs_qa labels Apr 5, 2018
@baconmania
Copy link
Contributor

@baconmania baconmania commented Apr 5, 2018

🚢

@@ -353,6 +355,11 @@ private double score(SingularityOfferHolder offerHolder, Map<String, Integer> ta
return 0;
}

if (mesosConfiguration.isOmitOverloadedHosts() && maybeSlaveUsage.isPresent() && maybeSlaveUsage.get().isOverloaded()) {
LOG.debug("Slave {} is overloaded (), ignoring offer");

This comment has been minimized.

@baconmania

baconmania Apr 5, 2018
Contributor

Looks like we missed the interp'd parameters here.

This comment has been minimized.

@ssalinas

ssalinas Apr 5, 2018
Author Member

Fixed in future PR already, since this is already 🚢 ed

@ssalinas ssalinas added this to the 0.20.0 milestone Apr 11, 2018
@ssalinas ssalinas merged commit 05f5f59 into master Apr 11, 2018
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@ssalinas ssalinas deleted the omit_overloaded branch Apr 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.