New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forbid health checks longer than kill time #1498

Merged
merged 3 commits into from Apr 26, 2017

Conversation

Projects
None yet
3 participants
@PtrTeixeira
Contributor

PtrTeixeira commented Apr 10, 2017

Singularity is configured to automatically kill tasks which are
nonresponsive for a certain amount of time after startup
(killAfterTasksDoNotRunDefaultSeconds, set in the config file). If a
task is configured such that it won't start accepting healthchecks for
longer than that interval, Singularity will just kill it off, without
ever sending health checks. This makes it so that that task
configuration will be rejected ahead of time by Singularity.

Forbid health checks longer than kill time
Singularity is configured to automatically kill tasks which are
nonresponsive for a certain amount of time after startup
(`killAfterTasksDoNotRunDefaultSeconds`, set in the config file).  If a
task is configured such that it won't start accepting healthchecks for
longer than that interval, Singularity will just kill it off, without
ever sending health checks. This makes it so that that task
configuration will be rejected ahead of time by Singularity.
Show outdated Hide outdated ...ice/src/main/java/com/hubspot/singularity/data/SingularityValidator.java
int startUpDelay = deploy.getHealthcheck().get().getStartupDelaySeconds().get();
checkBadRequest(startUpDelay < defaultKillAfterNotHealthySeconds,
String.format("Health check startup delay time must be less than kill after wait time %s (was %s)", defaultKillAfterNotHealthySeconds, startUpDelay));

This comment has been minimized.

@ssalinas

ssalinas Apr 10, 2017

Member

probably not as important for the user to know that we call it the 'kill after wait time'. Maybe just Health check startup delay time must be less than {time} seconds (was {time} seconds)

Other than that LGTM

@ssalinas

ssalinas Apr 10, 2017

Member

probably not as important for the user to know that we call it the 'kill after wait time'. Maybe just Health check startup delay time must be less than {time} seconds (was {time} seconds)

Other than that LGTM

PtrTeixeira added some commits Apr 11, 2017

Change label on new deploy form
There were two fields on the new deploy form that were labeled "HC
startup delay." This renames one of them to "HC startup timeout." They
bot functioned correctly and pointed to the the correct fields in the
deploy JSON; the label on one of them was just mixed up.
Modify error message
Was previously giving perhaps too much information to the user (ie, that
the upper limit was coming from how long we would wait to kill a task
that didn't appear to be starting). Not it just reflects the maximum
amount of time that you are allowed to put down.
WebApplicationException exn = (WebApplicationException) catchThrowable(() -> validator.checkDeploy(request, deploy));
assertThat((String) exn.getResponse().getEntity())
.contains("Health check startup delay");

This comment has been minimized.

@darcatron

darcatron Apr 12, 2017

Contributor

whoa, this catch is very cool

@darcatron

darcatron Apr 12, 2017

Contributor

whoa, this catch is very cool

@ssalinas ssalinas modified the milestone: 0.15.0 Apr 19, 2017

@ssalinas ssalinas merged commit d9515d8 into master Apr 26, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ssalinas ssalinas deleted the forbid-impossible-healthcheck-delay branch Apr 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment