[WFLY-12342] Add server probes for readiness health check

* Separate handling of the server readiness checks from the deployment
  checks to ensure that the server should not report UP until readiness
  checks from the deployment have been installed and called.
* Change the semantic of empty-readiness-check-status and
  empty-readiness-checks-status so that they only deal with empty
  *deployment* checks.
* Support the `mp.health.disable-default-procedures` config property to
  disable any vendor procedures. When the TCK is run as it assumes that
  there will be only checks installed by the TCK itself.
* Updated subsystem documentation

 JIRA: https://issues.jboss.org/browse/WFLY-12342

Signed-off-by: Jeff Mesnil <jmesnil@redhat.com>

docs/src/main/asciidoc/_admin-guide/subsystem-configuration/MicroProfile_Health.adoc

@@ -19,18 +19,45 @@ element to the xml or by using the following CLI operation:
 [standalone@localhost:9990 /]/extension=org.wildfly.extension.microprofile.health-smallrye:add
 ----
 
-== Management Operation
+== Management Operations
+
+The healthiness of the application server can be queried by calling 3 different operations:
+
+* `check` to check both the liveness and readiness of the runtime
+* `check-live` to check only the liveness of the runtime
+* `check-ready` to check only the readiness of the runtime
 
-The healthiness of the application server can be queried by calling the `check`:
 
 [source,options="nowrap"]
 ----
 [standalone@localhost:9990 /] /subsystem=microprofile-health-smallrye:check
 {
-    "outcome" => "success", <1>
+    "outcome" => "success",
     "result" => {
-        "outcome" => "UP", <2>
-        "checks" => []
+        "status" => "UP",
+        "checks" => [
+            {
+                "name" => "empty-liveness-checks",
+                "status" => "UP"
+            },
+            {
+                "name" => "server-state",
+                "status" => "UP",
+                "data" => {"value" => "running"}
+            },
+            {
+                "name" => "boot-errors",
+                "status" => "UP"
+            },
+            {
+                "name" => "deployments-status",
+                "status" => "UP"
+            },
+            {
+                "name" => "empty-readiness-checks",
+                "status" => "UP"
+            }
+        ]
     }
 }
 ----
@@ -40,6 +67,7 @@ The healthiness of the application server can be queried by calling the `check`:
 == HTTP Endpoints
 
 The MicroProfile Health Check specifications defines three HTTP endpoints:
+
 * `/health` to test both the liveness and readiness of the runtime.
 * `/health/live` to test the liveness of the runtime.
 * `/health/ready` to test the readiness of the runtime.
@@ -58,7 +86,7 @@ If the application server is healthy, it will return a `200 OK` response:
 $ curl -v http://localhost:9990/health
 < HTTP/1.1 200 OK
 ...
-{"outcome":"UP","checks":[]}
+{"status":"UP","checks":[{"name":"empty-liveness-checks","status":"UP"},{"name":"server-state","status":"UP","data":{"value":"running"}},{"name":"boot-errors","status":"UP"},{"name":"deployments-status","status":"UP"},{"name":"empty-readiness-checks","status":"UP"}]}
 ----
 
 If the application server (and its deployment) is not healthy, it returns `503 Service Unavailable`
@@ -79,28 +107,28 @@ created by the `add-user` script. For example:
 $ curl -v --digest -u myadminuser:myadminpassword http://localhost:9990/health
 < HTTP/1.1 200 OK
 ...
-{"outcome":"UP","checks":[]}
+{"status":"UP","checks":[{"name":"empty-liveness-checks","status":"UP"},{"name":"server-state","status":"UP","data":{"value":"running"}},{"name":"boot-errors","status":"UP"},{"name":"deployments-status","status":"UP"},{"name":"empty-readiness-checks","status":"UP"}]}
 ----
 
 If the authentication fails, the  server will reply with a `401 NOT AUTHORIZED` response.
 
-=== Global Status when probes are not defined
+=== Default Server Procedures
+
+WildFly provides some readiness procedures that are checked to determine if the application server is ready to serve requests:
 
-By default, the HTTP endpoints will return `UP` if there are no health check probes defined by the application.
+* `boot-errors` checks that there were no errors during the server boot sequence
+* `deployments-status` checks that all deployments were deployed without errors
+* `server-state` checks that the server state is `running`
+* `empty-readiness-checks` determines the status when there are no deployments on the server. The outcome of this procedure is determined by the `empty-readiness-checks-status` attribute. If the attribute is
+   `UP` (by default), the server can be ready when there are no readiness checks in the deployments. Setting the `empty-readiness-checks-status` attribute to `DOWN` will make this procedure fail when there are no readiness checks in the deployments.
 
-However, it is possible to change this behaviour.
-The `/health/live` HTTP endpoint (as well as the `check-live` management operation) can return `DOWN` when there are no liveness check probes by setting the `empty-liveness-checks-status` attribute
-on the `subsystem` resource to `DOWN`.
-Similarly, the `/health/ready` HTTP endpoint (as well as the `check-ready` management operation) can return `DOWN` when there are no readiness check probes by setting the `empty-readiness-checks-status` attribute
-on the `subsystem` resource to `DOWN`.
+WildFly also provide a liveness procedure that is checked to determine if the application server is live:
 
-This allows application to ensure that the HTTP liveness and readiness endpoints will return `DOWN` until their probes are deployed and used to determine the actual
-liveness and readiness state of the application.
+* `empty-liveness-checks` determines the status when there are no deployments on the server. The outcome of this procedure is determined by the `empty-liveness-checks-status` attribute. If the attribute is
+`UP` (by default), the server can be live when there are no liveness checks in the deployments.  Setting the `empty-liveness-checks-status` attribute to `DOWN` will make this procedure fail when there are no liveness checks in the deployments.
 
-[NOTE]
-There is a specific behaviour for deployments that provides no _readiness_ probes. In that the case, WildFly automatically adds a readiness probe for the deployment that always return
-`UP`. This allows existing deployments that do not provide readiness probes to configure WildFly so it will not be ready until this deployment is actually deployed.
 
+It is possible to disable all these server procedures by using the MicroProfile Config property `mp.health.disable-default-procedures`.
 
 == Component Reference
 

microprofile/health-smallrye/pom.xml

@@ -64,6 +64,10 @@
                 </exclusion>
             </exclusions>
         </dependency>
+        <dependency>
+            <groupId>org.wildfly.core</groupId>
+            <artifactId>wildfly-controller-client</artifactId>
+        </dependency>
         <dependency>
             <groupId>org.wildfly.core</groupId>
             <artifactId>wildfly-server</artifactId>
@@ -129,11 +133,6 @@
             <artifactId>vdx-wildfly</artifactId>
             <scope>test</scope>
         </dependency>
-        <dependency>
-            <groupId>org.wildfly.core</groupId>
-            <artifactId>wildfly-controller-client</artifactId>
-            <scope>test</scope>
-        </dependency>
         <dependency>
             <groupId>org.wildfly.core</groupId>
             <artifactId>wildfly-subsystem-test</artifactId>

microprofile/health-smallrye/src/main/java/org/wildfly/extension/microprofile/health/HealthReporter.java

@@ -22,6 +22,7 @@
 package org.wildfly.extension.microprofile.health;
 
 
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
 
@@ -42,48 +43,87 @@ public class HealthReporter {
 
     public static final String DOWN = "DOWN";
     public static final String UP = "UP";
+    private final boolean defaultServerProceduresDisabled;
     private Map<HealthCheck, ClassLoader> healthChecks = new HashMap<>();
     private Map<HealthCheck, ClassLoader> livenessChecks = new HashMap<>();
     private Map<HealthCheck, ClassLoader> readinessChecks = new HashMap<>();
+    private Map<HealthCheck, ClassLoader> serverReadinessChecks = new HashMap<>();
 
-    private final String emptyLivenessChecksStatus;
-    private final String emptyReadinessChecksStatus;
+    private final HealthCheck emptyDeploymentLivenessCheck;
+    private final HealthCheck emptyDeploymentReadinessCheck;
 
-    public HealthReporter(String emptyLivenessChecksStatus, String emptyReadinessChecksStatus) {
-        this.emptyLivenessChecksStatus = emptyLivenessChecksStatus;
-        this.emptyReadinessChecksStatus = emptyReadinessChecksStatus;
+    private static class EmptyDeploymentCheckStatus implements HealthCheck {
+        private final String name;
+        private final String status;
+
+        EmptyDeploymentCheckStatus(String name, String status) {
+            this.name = name;
+            this.status = status;
+        }
+
+        @Override
+        public HealthCheckResponse call() {
+            return HealthCheckResponse.named(name)
+                    .state(status.equals("UP"))
+                    .build();
+        }
+    }
+
+
+    public HealthReporter(String emptyLivenessChecksStatus, String emptyReadinessChecksStatus, boolean defaultServerProceduresDisabled) {
+        this.emptyDeploymentLivenessCheck  = new EmptyDeploymentCheckStatus("empty-liveness-checks", emptyLivenessChecksStatus);
+        this.emptyDeploymentReadinessCheck  = new EmptyDeploymentCheckStatus("empty-readiness-checks", emptyReadinessChecksStatus);
+        this.defaultServerProceduresDisabled = defaultServerProceduresDisabled;
     }
 
     public SmallRyeHealth getHealth() {
-        String emptyChecksStatus = DOWN.equals(emptyLivenessChecksStatus) || DOWN.equals(emptyReadinessChecksStatus) ? DOWN : UP;
-        return getHealth(emptyChecksStatus, healthChecks, livenessChecks, readinessChecks);
+        HashMap<HealthCheck, ClassLoader> deploymentChecks = new HashMap<>();
+        deploymentChecks.putAll(healthChecks);
+        deploymentChecks.putAll(livenessChecks);
+        deploymentChecks.putAll(readinessChecks);
+
+        HashMap<HealthCheck, ClassLoader> serverChecks= new HashMap<>();
+        serverChecks.putAll(serverReadinessChecks);
+        if (deploymentChecks.size() == 0 && !defaultServerProceduresDisabled) {
+            serverChecks.put(emptyDeploymentLivenessCheck, Thread.currentThread().getContextClassLoader());
+            serverChecks.put(emptyDeploymentReadinessCheck, Thread.currentThread().getContextClassLoader());
+        }
+
+        return getHealth(serverChecks, deploymentChecks);
     }
 
     public SmallRyeHealth getLiveness() {
-        return getHealth(emptyLivenessChecksStatus, livenessChecks);
+        final Map<HealthCheck, ClassLoader> serverChecks;
+        if (livenessChecks.size() == 0 && !defaultServerProceduresDisabled) {
+            serverChecks = Collections.singletonMap(emptyDeploymentLivenessCheck, Thread.currentThread().getContextClassLoader());
+        } else {
+            serverChecks = Collections.emptyMap();
+        }
+        return getHealth(serverChecks, livenessChecks);
     }
 
     public SmallRyeHealth getReadiness() {
-        return getHealth(emptyReadinessChecksStatus, readinessChecks);
+        final Map<HealthCheck, ClassLoader> serverChecks = new HashMap<>();
+        serverChecks.putAll(serverReadinessChecks);
+        if (readinessChecks.size() == 0 && !defaultServerProceduresDisabled) {
+            serverChecks.put(emptyDeploymentReadinessCheck, Thread.currentThread().getContextClassLoader());
+        }
+        return getHealth(serverChecks, readinessChecks);
     }
 
-
-    @SafeVarargs
-    private final SmallRyeHealth getHealth(String emptyChecksStatus, Map<HealthCheck, ClassLoader>... checks) {
+    private final SmallRyeHealth getHealth(Map<HealthCheck, ClassLoader> serverChecks, Map<HealthCheck, ClassLoader> deploymentChecks) {
         JsonArrayBuilder results = Json.createArrayBuilder();
         HealthCheckResponse.State status = HealthCheckResponse.State.UP;
 
-        if (checks != null) {
-            for (Map<HealthCheck, ClassLoader> entry : checks) {
-                status = processChecks(entry, results, status);
-            }
-        }
+        status = processChecks(serverChecks, results, status);
+
+        status = processChecks(deploymentChecks, results, status);
 
         JsonObjectBuilder builder = Json.createObjectBuilder();
 
         JsonArray checkResults = results.build();
 
-        builder.add("status", checkResults.isEmpty() ? emptyChecksStatus : status.toString());
+        builder.add("status", status.toString());
         builder.add("checks", checkResults);
 
         return new SmallRyeHealth(builder.build());
@@ -171,6 +211,12 @@ public void addReadinessCheck(HealthCheck check, ClassLoader moduleClassLoader)
         }
     }
 
+    public void addServerReadinessCheck(HealthCheck check, ClassLoader moduleClassLoader) {
+        if (check != null) {
+            serverReadinessChecks.put(check, moduleClassLoader);
+        }
+    }
+
     public void removeReadinessCheck(HealthCheck check) {
         readinessChecks.remove(check);
     }

microprofile/health-smallrye/src/main/java/org/wildfly/extension/microprofile/health/HealthReporterService.java

@@ -31,6 +31,7 @@
 
 import io.smallrye.health.ResponseProvider;
 import io.smallrye.health.SmallRyeHealthReporter;
+import org.eclipse.microprofile.config.ConfigProvider;
 import org.eclipse.microprofile.health.HealthCheckResponse;
 import org.jboss.as.controller.CapabilityServiceBuilder;
 import org.jboss.as.controller.LocalModelControllerClient;
@@ -80,21 +81,25 @@ private HealthReporterService(Supplier<ModelControllerClientFactory> modelContro
 
     @Override
     public void start(StartContext context) {
-        healthReporter = new HealthReporter(emptyLivenessChecksStatus, emptyReadinessChecksStatus);
+        // MicroProfile Health supports the mp.health.disable-default-procedures to let users disable any vendor procedures
+        final boolean defaultServerProceduresDisabled = ConfigProvider.getConfig().getOptionalValue("mp.health.disable-default-procedures", Boolean.class).orElse(false);
+        healthReporter = new HealthReporter(emptyLivenessChecksStatus, emptyReadinessChecksStatus, defaultServerProceduresDisabled);
 
         modelControllerClient = modelControllerClientFactory.get().createClient(managementExecutor.get());
 
-        healthReporter.addReadinessCheck(new ServerStateCheck(modelControllerClient));
-        healthReporter.addReadinessCheck(new NoBootErrorsCheck(modelControllerClient));
-        healthReporter.addReadinessCheck(new DeploymentsStatusCheck(modelControllerClient));
+        if (!defaultServerProceduresDisabled) {
+            healthReporter.addServerReadinessCheck(new ServerStateCheck(modelControllerClient), Thread.currentThread().getContextClassLoader());
+            healthReporter.addServerReadinessCheck(new NoBootErrorsCheck(modelControllerClient), Thread.currentThread().getContextClassLoader());
+            healthReporter.addServerReadinessCheck(new DeploymentsStatusCheck(modelControllerClient), Thread.currentThread().getContextClassLoader());
+        }
 
         HealthCheckResponse.setResponseProvider(new ResponseProvider());
     }
 
     @Override
     public void stop(StopContext context) {
-        this.modelControllerClient.close();
-        this.healthReporter = null;
+        modelControllerClient.close();
+        healthReporter = null;
         HealthCheckResponse.setResponseProvider(null);
     }
 

microprofile/health-smallrye/src/main/resources/org/wildfly/extension/microprofile/health/LocalDescriptions.properties

@@ -5,5 +5,5 @@ microprofile-health-smallrye.check=Check both the liveness and readiness of the
 microprofile-health-smallrye.check-live=Check the liveness of the application server and its deployments
 microprofile-health-smallrye.check-ready=Check the readiness of the application server and its deployments
 microprofile-health-smallrye.security-enabled=True if authentication is required to access the HTTP endpoints on the HTTP management interface.
-microprofile-health-smallrye.empty-liveness-checks-status=Defines the global status returned by the Health checks endpoints if no liveness probes have been defined.
-microprofile-health-smallrye.empty-readiness-checks-status=Defines the global status returned by the Health checks endpoints if no readiness probes have been defined.
+microprofile-health-smallrye.empty-liveness-checks-status=Defines the global status returned by the Health checks endpoints if no liveness probes have been defined in deployments.
+microprofile-health-smallrye.empty-readiness-checks-status=Defines the global status returned by the Health checks endpoints if no readiness probes have been defined in deployments.

microprofile/health-smallrye/src/test/java/org/wildfly/extension/microprofile/health/Subsystem_2_0_ParsingTestCase.java

@@ -26,6 +26,7 @@
 import static org.junit.Assert.assertTrue;
 import static org.wildfly.extension.microprofile.health.MicroProfileHealthExtension.VERSION_1_0_0;
 import static org.wildfly.extension.microprofile.health.MicroProfileHealthSubsystemDefinition.HTTP_EXTENSIBILITY_CAPABILITY;
+import static org.wildfly.extension.microprofile.health.MicroProfileHealthSubsystemDefinition.MANAGEMENT_EXECUTOR;
 
 import java.io.IOException;
 import java.util.List;
@@ -110,7 +111,8 @@ private static String getMicroProfileSmallryeHeatlhGAV(ModelTestControllerVersio
     protected AdditionalInitialization createAdditionalInitialization() {
         return AdditionalInitialization.withCapabilities(
                 HTTP_EXTENSIBILITY_CAPABILITY,
-                WELD_CAPABILITY_NAME);
+                WELD_CAPABILITY_NAME,
+                MANAGEMENT_EXECUTOR);
     }
 
     private void testRejectingTransformers(ModelTestControllerVersion controllerVersion, ModelVersion healthVersion) throws Exception {

testsuite/integration/basic/src/test/java/org/wildfly/test/integration/microprofile/health/MicroProfileHealthApplicationReadyHTTPEndpointTestCase.java

@@ -55,11 +55,12 @@ void checkGlobalOutcome(ManagementClient managementClient, String operation, boo
         try (CloseableHttpClient client = HttpClients.createDefault()) {
 
             CloseableHttpResponse resp = client.execute(new HttpGet(healthURL));
-            assertEquals(mustBeUP ? 200 : 503, resp.getStatusLine().getStatusCode());
-
             String content = EntityUtils.toString(resp.getEntity());
             resp.close();
 
+            assertEquals(content, mustBeUP ? 200 : 503, resp.getStatusLine().getStatusCode());
+
+
             try (
                     JsonReader jsonReader = Json.createReader(new StringReader(content))
             ) {

testsuite/integration/basic/src/test/java/org/wildfly/test/integration/microprofile/health/MicroProfileHealthApplicationReadySetupTask.java

@@ -34,7 +34,7 @@ public class MicroProfileHealthApplicationReadySetupTask implements ServerSetupT
     public void setup(ManagementClient managementClient, String containerId) throws Exception {
         final ModelNode address = Operations.createAddress("subsystem", "microprofile-health-smallrye");
         final ModelNode op = Operations.createWriteAttributeOperation(address, "empty-readiness-checks-status", "DOWN" );
-        ModelNode result = managementClient.getControllerClient().execute(op);
+        managementClient.getControllerClient().execute(op);
 
         ServerReload.reloadIfRequired(managementClient);
     }

testsuite/integration/basic/src/test/java/org/wildfly/test/integration/microprofile/health/MicroProfileHealthApplicationWitouhtReadinessHTTPEndpointTestCase.java → testsuite/integration/basic/src/test/java/org/wildfly/test/integration/microprofile/health/MicroProfileHealthApplicationWithoutReadinessHTTPEndpointTestCase.java

@@ -43,7 +43,7 @@
 /**
  * @author <a href="http://jmesnil.net/">Jeff Mesnil</a> (c) 2019 Red Hat inc.
  */
-public class MicroProfileHealthApplicationWitouhtReadinessHTTPEndpointTestCase extends MicroProfileHealthApplicationWithoutReadinessTestBase {
+public class MicroProfileHealthApplicationWithoutReadinessHTTPEndpointTestCase extends MicroProfileHealthApplicationWithoutReadinessTestBase {
 
     void checkGlobalOutcome(ManagementClient managementClient, String operation, boolean mustBeUP, String probeName) throws IOException {
 

testsuite/integration/microprofile-tck/health/src/test/resources/arquillian.xml

@@ -5,7 +5,7 @@
     <container qualifier="jboss" default="true" >
         <configuration>
             <property name="jbossHome">${jboss.home}</property>
-            <property name="javaVmArguments">${microprofile.jvm.args}</property>
+            <property name="javaVmArguments">${microprofile.jvm.args} -Dmp.health.disable-default-procedures=true</property>
             <property name="serverConfig">${jboss.server.config.file.name:standalone-microprofile.xml}</property>
             <property name="managementAddress">127.0.0.1</property>
             <property name="managementPort">9990</property>