Add PENDING type to healthchecks #360

andythsu · 2024-05-24T21:12:01Z

Description

Resolves #222 part 1

Additional context and related issues

Release notes

(X) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

gateway-ha/src/main/java/io/trino/gateway/ha/resource/EntityEditorResource.java

rdsarvar

some small nitpicks, mostly around use of a 'healthy' variable that has > 2 states

rdsarvar · 2024-05-29T19:15:53Z

gateway-ha/src/main/java/io/trino/gateway/ha/router/QueryCountBasedRouter.java

@@ -46,7 +47,7 @@ static class LocalStats
    {
        private int runningQueryCount;
        private int queuedQueryCount;
-        private boolean healthy;
+        private TrinoHealthStateType healthy;


question: what are your thoughts on having this be heathState instead of healthy?

as well as respective places where healthy is used

I agree. healthy kinda has a binary implication

rdsarvar · 2024-05-29T19:17:50Z

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/HealthChecker.java

@@ -34,7 +34,7 @@ public HealthChecker(Notifier notifier)
    public void observe(List<ClusterStats> clustersStats)
    {
        for (ClusterStats clusterStats : clustersStats) {
-            if (!clusterStats.healthy()) {
+            if (clusterStats.healthy() == TrinoHealthStateType.UNHEALTHY) {


nitpick: feels weird to read does healthy() = UNHEALTHY ?, would healthState() == UNHEALTHY read better?

or maybe just clusterStats.Health() ?

Sure, open to whichever naming - seeing the ‘y’ suffix (to me) on healthy implies boolean

I like healthState() == UNHEALTHY

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoHealthStateType.java

Chaho12

How about TrinoHealthStateType to TrinoHealthType ? i find it quite a redundant to say healthState as health is state of some status.

Chaho12 · 2024-05-30T00:38:16Z

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/HealthChecker.java

@@ -34,7 +34,7 @@ public HealthChecker(Notifier notifier)
    public void observe(List<ClusterStats> clustersStats)
    {
        for (ClusterStats clusterStats : clustersStats) {
-            if (!clusterStats.healthy()) {
+            if (clusterStats.healthy() == TrinoHealthStateType.UNHEALTHY) {


or maybe just clusterStats.Health() ?

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsInfoApiMonitor.java

Chaho12

Please resolve conversation after fixes has been made.
It makes it easier to PR (know that fix has been made for the commend)

LGTM 👍

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoHealthStateType.java

ebyhr · 2024-05-31T03:46:30Z

gateway-ha/src/main/java/io/trino/gateway/baseapp/BaseApp.java

@@ -68,6 +68,9 @@ public abstract class BaseApp
 {
    private static final Logger logger = Logger.get(BaseApp.class);
    private final ImmutableList.Builder<Module> appModules = ImmutableList.builder();
+    // this injector reference is needed to use reflection in
+    // TestGatewayHaSingleBackend and TestGatewayMultipleBackend


Please avoid using reflection. We should find different ways.

if we don't use reflection another way is to add a timeout to actually run the healthcheck TestGatewayHaSingleBackend and TestGatewayMultipleBackend are integration tests

TestGatewayMultipleBackend uses the TrinoContainer from TestContainers for trino1 and trino2, which does not finish its startup() until SELECT 1 returns. So the health check should succeed. customBackend is a MockWebServer so you will need to add an endpoint to satisfy the healthcheck. I do not believe you should need to set health status manually through injection

gateway-ha/src/main/java/io/trino/gateway/ha/resource/EntityEditorResource.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingManager.java

ebyhr · 2024-05-31T03:50:53Z

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingManager.java

@@ -181,7 +182,7 @@ protected String findBackendForUnknownQueryId(String queryId)
    }

    // Predicate helper function to remove the backends from the list
-    // We are returning the unhealthy (not healthy)
+    // We are returning the unhealthy (not healthState)


gateway-ha/src/test/java/io/trino/gateway/ha/router/TestStochasticRoutingManager.java

willmostly · 2024-06-06T14:16:09Z

gateway-ha/src/main/java/io/trino/gateway/baseapp/BaseApp.java

@@ -68,6 +68,9 @@ public abstract class BaseApp
 {
    private static final Logger logger = Logger.get(BaseApp.class);
    private final ImmutableList.Builder<Module> appModules = ImmutableList.builder();
+    // this injector reference is needed to use reflection in
+    // TestGatewayHaSingleBackend and TestGatewayMultipleBackend


TestGatewayMultipleBackend uses the TrinoContainer from TestContainers for trino1 and trino2, which does not finish its startup() until SELECT 1 returns. So the health check should succeed. customBackend is a MockWebServer so you will need to add an endpoint to satisfy the healthcheck. I do not believe you should need to set health status manually through injection

willmostly · 2024-06-06T15:09:35Z

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoHealthStateType.java

+/**
+ * PENDING is for ui/observability purpose and functionally it's unhealthy
+ * We should use PENDING when Trino clusters are still spinning up
+ * HEALTHY is when health checks report clusters as up
+ * UNHEALTHY is when health checks report clusters as down
+ */


this should be added to the docs. As to placement I think a section on health checks should be added, and linked to from the routing logic and operation sections. Wdyt @mosabua ?

andythsu · 2024-06-06T15:20:07Z

@willmostly

TestGatewayMultipleBackend uses the TrinoContainer from TestContainers for trino1 and trino2, which does not finish its startup() until SELECT 1 returns. So the health check should succeed. customBackend is a MockWebServer so you will need to add an endpoint to satisfy the healthcheck. I do not believe you should need to set health status manually through injection

Sorry for the confusion.

In TestGatewayMultipleBackend and TestGatewaySingleBackend, the clusters are added by calling the post api

The default healthstate when clusters are first added to the gateway is PENDING (and should be). Because PENDING is functionally treated as unhealthy, the test cases will fail (for example,

trino-gateway/gateway-ha/src/test/java/io/trino/gateway/ha/TestGatewayHaSingleBackend.java

Lines 65 to 70 in d45c64f

    
           Request request = 
        
                   new Request.Builder() 
        
                           .url("http://localhost:" + routerPort + "/v1/statement") 
        
                           .addHeader("X-Trino-User", "test") 
        
                           .post(requestBody) 
        
                           .build();

) since all clusters' states are still in PENDING when the test cases run. Unless we wait until the first round of healthcheck kicks in and changes the states from PENDING to HEALTHY, no clusters are available.

cla-bot bot added the cla-signed label May 24, 2024

andythsu commented May 24, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/resource/EntityEditorResource.java Show resolved Hide resolved

rdsarvar suggested changes May 29, 2024

View reviewed changes

Chaho12 reviewed May 30, 2024

View reviewed changes

Fix log when cluster is turned on/off

53af36e

andythsu force-pushed the healthstate branch from 45ee624 to 14a2d0f Compare May 30, 2024 16:04

Chaho12 approved these changes May 31, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoHealthStateType.java Outdated Show resolved Hide resolved

ebyhr requested changes May 31, 2024

View reviewed changes

andythsu force-pushed the healthstate branch from 14a2d0f to f1a2261 Compare May 31, 2024 17:26

Add PENDING type to healthchecks

bca99f9

andythsu force-pushed the healthstate branch from f1a2261 to bca99f9 Compare May 31, 2024 17:27

willmostly requested changes Jun 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PENDING type to healthchecks #360

Add PENDING type to healthchecks #360

andythsu commented May 24, 2024 •

edited

Loading

rdsarvar left a comment

rdsarvar May 29, 2024

andythsu May 30, 2024

rdsarvar May 29, 2024

Chaho12 May 30, 2024

rdsarvar May 30, 2024

andythsu May 30, 2024

Chaho12 left a comment

Chaho12 May 30, 2024

Chaho12 left a comment

ebyhr May 31, 2024

andythsu May 31, 2024 •

edited

Loading

willmostly Jun 6, 2024

ebyhr May 31, 2024

willmostly Jun 6, 2024

willmostly Jun 6, 2024

andythsu commented Jun 6, 2024 •

edited

Loading

Add PENDING type to healthchecks #360

Are you sure you want to change the base?

Add PENDING type to healthchecks #360

Conversation

andythsu commented May 24, 2024 • edited Loading

Description

Additional context and related issues

Release notes

rdsarvar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Chaho12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Chaho12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andythsu May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andythsu commented Jun 6, 2024 • edited Loading

andythsu commented May 24, 2024 •

edited

Loading

andythsu May 31, 2024 •

edited

Loading

andythsu commented Jun 6, 2024 •

edited

Loading