introduce offer cache to allow better resource allocation #1402

Merged
merged 19 commits into from Mar 10, 2017

Conversation

Projects
None yet
5 participants
@wsorenson
Member

wsorenson commented Jan 19, 2017

Currently, does not use cached offers.

@ssalinas ssalinas modified the milestone: 0.14.0 Jan 20, 2017

@wsorenson wsorenson added the hs_qa label Jan 26, 2017

@ssalinas

This comment has been minimized.

Show comment
Hide comment
@ssalinas

ssalinas Feb 1, 2017

Member

just curious, is there a performance improvement from this?

just curious, is there a performance improvement from this?

This comment has been minimized.

Show comment
Hide comment
@jhaber

jhaber Feb 1, 2017

Member

Internally the logger will do this same check so I think it should be about the same

Member

jhaber replied Feb 1, 2017

Internally the logger will do this same check so I think it should be about the same

This comment has been minimized.

Show comment
Hide comment
@wsorenson

wsorenson Feb 4, 2017

Member

Not in this case - only when you can skip method evaluations. I suppose in this case toString() is probably called after the logger decides to log, so its irrelevant

Member

wsorenson replied Feb 4, 2017

Not in this case - only when you can skip method evaluations. I suppose in this case toString() is probably called after the logger decides to log, so its irrelevant

wsorenson added some commits Feb 1, 2017

+
+ private void checkOut() {
+ LOG.trace("Checking out offer {}", offerId);
+ Preconditions.checkState(offerState == OfferState.AVAILABLE, "Offer %s was in state %s", offerId, offerState);

This comment has been minimized.

@ssalinas

ssalinas Feb 3, 2017

Member

Saw a sentry come through for this the other day. This exception will bubble all the way back up to the LeaderOnlyPoller and cause us to abort. Should we maybe catch this and just skip the offer in the case that something else has it checked out?

@ssalinas

ssalinas Feb 3, 2017

Member

Saw a sentry come through for this the other day. This exception will bubble all the way back up to the LeaderOnlyPoller and cause us to abort. Should we maybe catch this and just skip the offer in the case that something else has it checked out?

wsorenson added some commits Feb 3, 2017

@wsorenson

This comment has been minimized.

Show comment
Hide comment
@wsorenson

wsorenson Feb 4, 2017

Member

In my test case of 1000 tasks and 250 offers, saw a total improvement of about 3m21s to 14s

Member

wsorenson commented on e4c42f1 Feb 4, 2017

In my test case of 1000 tasks and 250 offers, saw a total improvement of about 3m21s to 14s

@ssalinas ssalinas added the hs_stable label Feb 24, 2017

+
+ public void returnOffer(CachedOffer cachedOffer);
+
+ public List<Offer> peakOffers();

This comment has been minimized.

@tpetr

tpetr Mar 2, 2017

Member

peekOffers()?

@tpetr

tpetr Mar 2, 2017

Member

peekOffers()?

ssalinas added some commits Mar 10, 2017

@ssalinas

This comment has been minimized.

Show comment
Hide comment
@ssalinas

ssalinas Mar 10, 2017

Member

Fixed the merge conflicts and re-tested in each environment as well, going to merge this one 👍

Member

ssalinas commented Mar 10, 2017

Fixed the merge conflicts and re-tested in each environment as well, going to merge this one 👍

@ssalinas ssalinas merged commit 118c4e9 into master Mar 10, 2017

0 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details
@miry

This comment has been minimized.

Show comment
Hide comment
@miry

miry May 8, 2017

@wsorenson how to disable this this feature?

I could not find related info in http://getsingularity.com/Docs/reference/configuration.html.

I think this feature eats all Mesos resources and then free some of them in 1 minute, and then again takes. So other frameworks could not run job fast.

miry commented May 8, 2017

@wsorenson how to disable this this feature?

I could not find related info in http://getsingularity.com/Docs/reference/configuration.html.

I think this feature eats all Mesos resources and then free some of them in 1 minute, and then again takes. So other frameworks could not run job fast.

@tpetr

This comment has been minimized.

Show comment
Hide comment
@tpetr

tpetr May 8, 2017

Member

@miry It looks like the documentation needs to be updated, thanks for the heads up. You can disable offer caching by setting cacheOffers: false in your Singularity config YAML.

@ssalinas @wsorenson thoughts on having offer caching disabled by default?

Member

tpetr commented May 8, 2017

@miry It looks like the documentation needs to be updated, thanks for the heads up. You can disable offer caching by setting cacheOffers: false in your Singularity config YAML.

@ssalinas @wsorenson thoughts on having offer caching disabled by default?

@ssalinas

This comment has been minimized.

Show comment
Hide comment
@ssalinas

ssalinas May 8, 2017

Member

For the sake of those running in multi-framework clusters it might make sense to have it off by default. I can submit a PR with that and the documentation update

Member

ssalinas commented May 8, 2017

For the sake of those running in multi-framework clusters it might make sense to have it off by default. I can submit a PR with that and the documentation update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment