JBPM-6740 - Expose readiness and liveness checks in KIE Server #1332

mswiderski · 2018-01-12T09:57:02Z

here is the implementation that aims at providing basic readiness and liveness checks for KIE Server and all extensions activated.
In general:

readiness will either respond with 200 (OK) when it's actually ready or with 503 (Service Unavailable) when it's still booting/deploying containers/waiting for controller.
liveness (aka health check) will perform following:
- will check readiness
- will check for failed kie containers
- will ask each active extension to health check itself
  response codes for health check are same as for readiness. Meaning that any error found will result in response 503, regardless if that is failed container, failed extension or not ready yet

Health check can be invoked in two modes:

basic - that will return status only 200 or 503
report - will respond with both status and report in response body that will provide info like below (body can be XML or JSON)

Now I did implement some checks in extensions but not sure if the make sense or more stuff should be added or not. So please take a moment and let me know what you would like to health check in extension you deal with.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<list-type>
    <items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>KIE Server 'managed-server' is ready to serve requests true</content>
            <content>Server is up for 0 days, 0 hours, 2 minutes, 31 seconds</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.375+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>Drools is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.375+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>jBPM is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>Case-Mgmt is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>jBPM-UI is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>OptaPlanner is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>DMN is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>Swagger is alive</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
        <items xsi:type="kie-message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <content>Health check done in 3 ms</content>
            <severity>INFO</severity>
            <timestamp>2018-01-12T10:40:34.378+01:00</timestamp>
        </items>
    </items>
</list-type>

sutaakar · 2018-01-12T11:35:15Z

...ces/kie-server-services-common/src/main/java/org/kie/server/services/impl/KieServerImpl.java

+
+    }
+
+    private String calculateUptime() {


Did you consider using java.time.Duration for these calculations?

I actually did and the problem is that it does not take into account already used parts of it, see example:

0 days, 0 hours, 30 minutes, 1800 seconds 0 days, 0 hours, 30 minutes, 0 seconds

first one is done with duration.toMinutes and duration.toMillis()/1000 as there seems to be missing toSeconds method.

Or did you have something else in mind?

I see. Was thinking about using some formatter to convert Duration to some string representation, but it seems that there isn't any out of the box.
Current approach seems ok.

You can use plain toString method on a duration, it will generate for example PT41M29.064S. It is ISO 8601 format. In fact, Duration's toString method does it manually with div and mod too.

@MarianMacik correct, but it's way less readable in my opinion, especially for those that are not familiar with ISO date format.

There are also easy to use things DurationFormatUtils in commons-lang3

@jhrcek you rock! replaced with DurationFormatUtils, thanks!

sutaakar

Looks great, thanks. One idea included in comment.

tsurdilo · 2018-01-16T19:51:05Z

...-rest-common/src/main/java/org/kie/server/remote/rest/common/resource/KieServerRestImpl.java

+            response=Void.class, code=200)
+    @ApiResponses(value = { @ApiResponse(code = 503, message = "Service not yet available") })
+    @GET
+    @Path("readiness")


Would rename this to "readycheck" (same for method name)

tsurdilo · 2018-01-16T19:51:20Z

...-rest-common/src/main/java/org/kie/server/remote/rest/common/resource/KieServerRestImpl.java

+            response=Void.class, code=200)
+    @ApiResponses(value = { @ApiResponse(code = 503, message = "If any of the checks failed") })
+    @GET
+    @Path("health")


Would rename this to "healthcheck" (same for method name)

tsurdilo · 2018-01-16T19:54:00Z

...-rest-common/src/main/java/org/kie/server/remote/rest/common/resource/KieServerRestImpl.java

+        if (server.isKieServerReady()) {
+            return Response.status(Response.Status.OK).build();
+        }
+        return serviceUnavailable();


I think all "check/monitoring" methods should have always OK response status and show issues as messages to user in case of error. Might be better if users build monitoring apps against these checks

these endpoints are meant for machine reading rather than human - so the most important is to communicate clearly over status codes with minimal payload that's why by default they always return just status and no payload. When user wants to see what is not healthy then there is ?report=true query param that will return payload with messages regardless of the status code (both 200 and 503)

sutaakar reviewed Jan 12, 2018

View reviewed changes

sutaakar approved these changes Jan 12, 2018

View reviewed changes

mswiderski force-pushed the JBPM-6740 branch 2 times, most recently from 2666487 to acdc678 Compare January 12, 2018 19:02

tsurdilo reviewed Jan 16, 2018

View reviewed changes

JBPM-6740 - Expose readiness and liveness checks in KIE Server

6c9a951

mswiderski force-pushed the JBPM-6740 branch from acdc678 to 6c9a951 Compare January 17, 2018 08:09

tsurdilo approved these changes Jan 17, 2018

View reviewed changes

mswiderski merged commit dccaa60 into kiegroup:master Jan 17, 2018

mswiderski deleted the JBPM-6740 branch January 17, 2018 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JBPM-6740 - Expose readiness and liveness checks in KIE Server #1332

JBPM-6740 - Expose readiness and liveness checks in KIE Server #1332

mswiderski commented Jan 12, 2018

sutaakar Jan 12, 2018

mswiderski Jan 12, 2018

sutaakar Jan 12, 2018

MarianMacik Jan 12, 2018

mswiderski Jan 12, 2018

jhrcek Jan 12, 2018

mswiderski Jan 12, 2018

sutaakar left a comment

tsurdilo Jan 16, 2018

mswiderski Jan 17, 2018

tsurdilo Jan 16, 2018

mswiderski Jan 17, 2018

tsurdilo Jan 16, 2018

mswiderski Jan 17, 2018

JBPM-6740 - Expose readiness and liveness checks in KIE Server #1332

JBPM-6740 - Expose readiness and liveness checks in KIE Server #1332

Conversation

mswiderski commented Jan 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sutaakar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment