Add KSQL Server Docker healthcheck

[rmoff]

Testing done : added healthcheck to a local build of the image, tested it successfully:

Docker shows the container as healthy:

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                   CREATED             STATUS                   PORTS                              NAMES
8d6064b719f5        confluentinc/cp-enterprise-control-center:5.3.0       "bash -c 'echo \"Wait…"   5 minutes ago       Up 5 minutes             0.0.0.0:9021->9021/tcp             control-center
f433a6ef6f96        confluentinc/cp-ksql-cli:5.3.0                        "/bin/sh"                 5 minutes ago       Up 5 minutes                                                ksql-cli
2934a6fd17d5        confluentinc/cp-kafka-connect:5.3.0                   "bash -c 'echo \"Inst…"   5 minutes ago       Up 5 minutes             0.0.0.0:8083->8083/tcp, 9092/tcp   kafka-connect-01
a5c50f2590ea        confluentinc/cp-schema-registry:5.3.0                 "/etc/confluent/dock…"    5 minutes ago       Up 5 minutes             0.0.0.0:8081->8081/tcp             schema-registry
a6e6efadde0d        ksql-server-healthcheck                               "/etc/confluent/dock…"    5 minutes ago       Up 4 minutes (healthy)   0.0.0.0:8088->8088/tcp             ksql-server
[…]

$ docker inspect ksql-server|jq '.[].State.Health'
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2019-08-06T21:24:45.2994123Z",
      "End": "2019-08-06T21:24:45.5405994Z",
      "ExitCode": 1,
      "Output": "Tue Aug 6 21:24:45 UTC 2019 \tKSQL HTTP state:  000  (waiting for 200)\n"
    },
    {
      "Start": "2019-08-06T21:24:50.5501618Z",
      "End": "2019-08-06T21:24:51.2078062Z",
      "ExitCode": 1,
      "Output": "Tue Aug 6 21:24:51 UTC 2019 \tKSQL HTTP state:  503  (waiting for 200)\n"
    },
    {
      "Start": "2019-08-06T21:24:56.2132311Z",
      "End": "2019-08-06T21:24:56.4744598Z",
      "ExitCode": 0,
      "Output": "Woohoo! KSQL is up!\n"
    },
    {
      "Start": "2019-08-06T21:25:01.4814845Z",
      "End": "2019-08-06T21:25:01.6563264Z",
      "ExitCode": 0,
      "Output": "Woohoo! KSQL is up!\n"
    },
    {
      "Start": "2019-08-06T21:25:06.6621997Z",
      "End": "2019-08-06T21:25:06.8860803Z",
      "ExitCode": 0,
      "Output": "Woohoo! KSQL is up!\n"
    }
  ]
}

[rmoff]

I've taken @rodesai's feedback and used a variable to store the response. I've also amended the script to use the REST endpoint for status probe. This is because the KSQL server /info will still return HTTP 200 even if the Kafka cluster cannot be reached.

New testing when Kafka is unavailable shows the healthcheck then picks this up :

$ docker inspect ksql-server|jq '.[].State.Health'
{
  "Status": "unhealthy",
  "FailingStreak": 6,
  "Log": [
    {
      "Start": "2019-10-07T10:16:06.6133346Z",
      "End": "2019-10-07T10:16:16.6150504Z",
      "ExitCode": -1,
      "Output": "Health check exceeded timeout (10s)"
    },
    {
      "Start": "2019-10-07T10:16:21.5917883Z",
      "End": "2019-10-07T10:16:21.8185479Z",
      "ExitCode": 1,
      "Output": "Mon Oct 7 10:16:21 UTC 2019  KSQL server query response code:  500  (waiting for 200)\n"
    },
    {
      "Start": "2019-10-07T10:16:26.8263431Z",
      "End": "2019-10-07T10:16:27.0491532Z",
      "ExitCode": 1,
      "Output": "Mon Oct 7 10:16:27 UTC 2019  KSQL server query response code:  500  (waiting for 200)\n"
    },
    {
      "Start": "2019-10-07T10:16:32.0598393Z",
      "End": "2019-10-07T10:16:32.2837176Z",
      "ExitCode": 1,
      "Output": "Mon Oct 7 10:16:32 UTC 2019  KSQL server query response code:  500  (waiting for 200)\n"
    },
    {
      "Start": "2019-10-07T10:16:37.2917967Z",
      "End": "2019-10-07T10:16:37.4894039Z",
      "ExitCode": 1,
      "Output": "Mon Oct 7 10:16:37 UTC 2019  KSQL server query response code:  500  (waiting for 200)\n"
    }
  ]
}

$ docker-compose ps
      Name                    Command                   State                                 Ports
-----------------------------------------------------------------------------------------------------------------------------
[…]
kafka              /etc/confluent/docker/run        Exit 143
kafka-connect-01   bash -c echo "Installing C ...   Up               0.0.0.0:8083->8083/tcp, 9092/tcp
kibana             bash -c /usr/local/bin/kib ...   Up               0.0.0.0:5601->5601/tcp
ksql-cli           /bin/sh                          Up
ksql-server        /etc/confluent/docker/run        Up (unhealthy)   0.0.0.0:8088->8088/tcp

[rmoff]

Since @rodesai is on PTO now, could someone else on @confluentinc/ksql take a look at this please? ta.

[vcrfxia]

LGTM