Smarter Healthchecks #1306

ssalinas · 2016-09-27T20:43:58Z

This PR attempts to do a few things regarding our existing healthcheck setup

Split out the configuration into its own object so we stop cluttering the SingularityDeploy
Add the concept of a startup period (identified primarily by 'connection refused' responses vs actuall http responses)
- Add a separate startupDelay (time before running any checks) and startupInterval (how often to check during this time)
- Add a startupTimeout - time limit for app to actually respond to a check
Make the healthcheckTotalTimeoutSeconds a calculated field based on the others present rather than an override
Ability to specify a specific port number for a healthcheck (/fixes Singularity should support healthchecks on statically defined service ports #1287)

Still TODO:

Add validation on maxRetries * intervalSeconds (as a replacement for having a separate static max total timeout)
Finish wiring up startupTimeout in deploy and new task checkers
Smarter reactions based on status code
Tests specific to startup period and status codes
Clean up excess startup healthchecks since they provide no extra information
Ensure new task checks run at an appropritate interval
Update deploy form
Update healthcheck notifications and messages in UI
Fix healthcheck/deploy checker timeout race condition

/cc @tpetr @darcatron

matush-v · 2016-09-29T16:17:58Z

SingularityBase/src/main/java/com/hubspot/deploy/HealthcheckOptions.java

+    return uri;
+  }
+
+  @ApiModelProperty(required = false, value="Perform healthcheck on this dynamically allocated port (e.g. 0 for first port), defaults to first port")


nit: required = false => required=false

matush-v · 2016-09-29T16:18:11Z

SingularityBase/src/main/java/com/hubspot/deploy/HealthcheckOptions.java

+    return portIndex;
+  }
+
+  @ApiModelProperty(required = false, value="Perform healthcheck on this port (portIndex cannot also be used when using this setting)")


nit: required = false => required=false

matush-v · 2016-09-29T16:22:41Z

SingularityBase/src/main/java/com/hubspot/deploy/HealthcheckOptions.java

+      Objects.equal(startupDelaySeconds, that.startupDelaySeconds) &&
+      Objects.equal(intervalSeconds, that.intervalSeconds) &&
+      Objects.equal(responseTimeoutSeconds, that.responseTimeoutSeconds) &&
+      Objects.equal(maxRetries, that.maxRetries);


should this have startupInvervalSeconds as well?

matush-v · 2016-09-29T16:26:35Z

SingularityBase/src/main/java/com/hubspot/deploy/HealthcheckOptions.java

+  private final Optional<HealthcheckProtocol> protocol;
+  private final Optional<Integer> startupTimeoutSeconds;
+  private final Optional<Integer> startupDelaySeconds;
+  private final Optional<Integer> startupIntervalSeconds;


maybe call it startupFrequencySeconds?

matush-v · 2016-09-29T16:27:05Z

SingularityBase/src/main/java/com/hubspot/deploy/HealthcheckOptions.java

+      .add("intervalSeconds", intervalSeconds)
+      .add("responseTimeoutSeconds", responseTimeoutSeconds)
+      .add("maxRetries", maxRetries)
+      .toString();


should this have startupInvervalSeconds as well?

matush-v · 2016-09-29T16:32:36Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularityDeploy.java

@@ -446,6 +472,11 @@ public String getId() {
    return healthcheckMaxTotalTimeoutSeconds;
  }

+  @ApiModelProperty(required = false, value="HTTP Healthcheck settings")


nit: required = false => required=false

matush-v · 2016-09-29T16:46:30Z

SingularityService/src/main/java/com/hubspot/singularity/data/SingularityValidator.java

+          checkBadRequest(deploy.getResources().get().getNumPorts() > deploy.getHealthcheck().get().getPortIndex().get(), String
+            .format("Must request %s ports for healthcheckPortIndex %s, only requested %s", deploy.getHealthcheck().get().getPortIndex().get() + 1, deploy.getHealthcheck().get().getPortIndex().get(),
+              deploy.getResources().get().getNumPorts()));
+        }


maybe assign HealthcheckOptions healthcheck = deploy.getHealthcheck().get() after verifying isPresent() so the later uses are a little cleaner?

matush-v · 2016-09-29T17:05:49Z

...vice/src/main/java/com/hubspot/singularity/scheduler/SingularityHealthcheckAsyncHandler.java

+    if (throwable.isPresent() && throwable.get() instanceof ConnectException) {
+      inStartup = true;
+    }
+


boolean inStartup = throwable.isPresent() && throwable.get() instanceof ConnectException

matush-v · 2016-09-29T17:10:01Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularityDeployBuilder.java

  private Optional<Integer> healthcheckMaxRetries;
+  @Deprecated
  private Optional<Long> healthcheckMaxTotalTimeoutSeconds;


@Deprecated use {@link healthcheck}?

matush-v · 2016-09-29T17:10:11Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularityDeploy.java

  private final Optional<Integer> healthcheckMaxRetries;
+  @Deprecated
  private final Optional<Long> healthcheckMaxTotalTimeoutSeconds;


@Deprecated use {@link healthcheck}?

ssalinas · 2016-10-20T16:29:27Z

First frontend+backend draft of this is done now (save for the race condition fix). The setup of the new healthchecks looks like this:

UI messages are also updated to reflect this. Some examples:

Failed because the app never responded (never left the startup phase). This was with a 10s startupDelay and 15s startupTimeout set. Note that only a single healthcheck appreas in the table. This is because startup phase healthchecks are cleaned up after task checks run (except if it is the last healthcheck). This way we can check more often during the startup phase without cluttering healthchecks.
Failed due to a status code in the failureStatusCodes list
Failed due to too many retries

Open to any feedback on the ui messages, overall setup, or additional features before we merge this into qa or stable

@tpetr @darcatron

ssalinas · 2016-12-14T17:50:57Z

Since we recently did a release, going to merge this and I will address any bugs we find in future PRs. Been looking good in hs_stable though.

nyarly · 2017-05-09T18:09:45Z

@ssalinas I don't understand this new behavior at all. Is the reasoning documented somewhere?

Specifically, from what I can tell, once we leave startup, it looks like we poll the healthcheck URL looking for failure until the HC timeout elapses. What's the reasoning behind that? i.e. after one successful healthcheck, why not transition to "started"?

stevenschlansker · 2017-05-09T18:17:32Z

Hm, my understanding is that's exactly what happens -- any single successful check turns it to running state and no further checks will be performed.

nyarly · 2017-05-09T18:19:44Z

I skimmed the code and it seemed to be what I was concerned by. In that case, why are there multiple polls then? Because I was pretty sure that a single failure status (as defined by a configuration) marks the service UNHEALTHY, which I assume means Singularity kills the task.

ssalinas · 2017-05-09T18:21:18Z

There's a few things to divide up here:

In terms of the task state, we only start health checks once the task enters TASK_RUNNING. This just means the process is running not that it is healthy yet
It is correct that we stop after a single successful check

In the old method of healthchecks there was just a total amount of time/number of checks as the constraint. Now, there is a constraint around startup (i.e. time until we stop getting connection refused responses) and constraint around the actual response (i.e. time until a 'successfull' status code is received)

ssalinas · 2017-05-09T18:22:57Z

A single failure status does not mark the task as unhealthy unless you set it to do so. Once it has received some sort of valid http response, you then have maxRetries attempts left for the app to return an successful check

ssalinas added 2 commits September 27, 2016 16:32

First pass at smarter healthchecks

2ed3bba

add healthcheck options class

61ef2c0

ssalinas modified the milestone: 0.12.0 Sep 27, 2016

add missing dep in SingularityBase

9d185e2

matush-v reviewed Sep 29, 2016

View reviewed changes

ssalinas added 11 commits October 6, 2016 15:45

updates for PR comments

7a9bde5

Merge branch 'master' into smarter_healthchecks

7e6c69f

fix test after merge with master

64acf77

validation on max healthcheck time

1dc793c

wire up startup, add tests

660a43f

fix es to startupDelay

82924a3

default to an empty list

6c65fe9

fix retry count, clear excess starup checks

0e80962

finish making startupTimeout work

12011a0

fixes for cleaning up excess startup checks

23370c6

fix new task checker delay

57def25

ssalinas modified the milestone: 0.12.0 Oct 17, 2016

ssalinas added 4 commits October 18, 2016 15:18

Merge branch 'master' into smarter_healthchecks

64cc6fb

progress on smarter healthchecks frontend

cc79e9c

more finishing touches on healthcheck messages

36b9b29

update new deploy form for new healthchecks

56e7432

ssalinas changed the title ~~(WIP) Smarter Healthchecks~~ Smarter Healthchecks Oct 20, 2016

ssalinas added the hs_staging label Oct 20, 2016

ssalinas added 2 commits October 20, 2016 13:08

fix ui bug when healthcheckOptions not present

a8106c4

prevent race conditions with final healthcheck

78b6c4b

ssalinas added 11 commits November 16, 2016 15:04

check uri present before showing healthcheck section

c4dc365

validate a valid port is present to hit along with uri

41c21ae

missing }

3d3601d

corrected canDisableHealthchecks method for new helthcheck format

8df42d0

Remove last references to old style of healthcheck config

06227f7

remove isRequired on prop type

dca8fb1

fix prop types

162ddda

fix merge conflicts with master

29edb72

small tweak in utils

0b57bc4

use isLongRunning method here

4972454

remove extra import from fixing merges

6ee777d

ssalinas added the hs_stable label Dec 8, 2016

ssalinas modified the milestones: 0.13.0, 0.14.0 Dec 8, 2016

ssalinas mentioned this pull request Dec 13, 2016

Singularity should support healthchecks on statically defined service ports #1287

Closed

ssalinas merged commit ea60453 into master Dec 14, 2016

ssalinas deleted the smarter_healthchecks branch December 14, 2016 17:51

ssalinas removed hs_qa labels Dec 14, 2016

ssalinas added Deprecated Field(s) and removed Deprecated Field(s) labels Jan 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Smarter Healthchecks #1306

Smarter Healthchecks #1306

ssalinas commented Sep 27, 2016 •

edited

Loading

matush-v Sep 29, 2016

matush-v Sep 29, 2016 •

edited

Loading

matush-v Sep 29, 2016

matush-v Sep 29, 2016 •

edited

Loading

matush-v Sep 29, 2016

matush-v Sep 29, 2016

matush-v Sep 29, 2016

matush-v Sep 29, 2016

matush-v Sep 29, 2016

matush-v Sep 29, 2016

ssalinas commented Oct 20, 2016 •

edited

Loading

ssalinas commented Dec 14, 2016

nyarly commented May 9, 2017

stevenschlansker commented May 9, 2017

nyarly commented May 9, 2017 •

edited

Loading

ssalinas commented May 9, 2017

ssalinas commented May 9, 2017

Smarter Healthchecks #1306

Smarter Healthchecks #1306

Conversation

ssalinas commented Sep 27, 2016 • edited Loading

Choose a reason for hiding this comment

matush-v Sep 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matush-v Sep 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssalinas commented Oct 20, 2016 • edited Loading

ssalinas commented Dec 14, 2016

nyarly commented May 9, 2017

stevenschlansker commented May 9, 2017

nyarly commented May 9, 2017 • edited Loading

ssalinas commented May 9, 2017

ssalinas commented May 9, 2017

ssalinas commented Sep 27, 2016 •

edited

Loading

matush-v Sep 29, 2016 •

edited

Loading

matush-v Sep 29, 2016 •

edited

Loading

ssalinas commented Oct 20, 2016 •

edited

Loading

nyarly commented May 9, 2017 •

edited

Loading