Having healthz endpoint with appHealthCheck false

[shivamkm07]

In what area(s)?

/area runtime

/area operator

/area placement

/area docs

/area test-and-release

What version of Dapr?

edge: output of git describe --dirty

Expected Behavior

If app health check is disabled, it shouldn't be required for applications to implement healthz endpoint.

Actual Behavior

Even with app health disabled, If app doesn't implement healthz, actors(and so Dapr workflows) don't work. This happens because placement detects app as unhealthy and disconnects with log Disconnecting from placement service by the unhealthy app.

Steps to Reproduce the Problem

1. Run tests/integration/suite/actors/healthz/deactivateOnPlacementFail test with healthz endpoint commented out:

   dapr/tests/integration/suite/actors/healthz/deactivate-on-placement-fail.go

   Line 58 in 328b4b3

   r.Get("/healthz", func(w http.ResponseWriter, r *http.Request) {

2. It would fail even when app health check is false

Release Note

RELEASE NOTE:

[ItalyPaleAle]

Just some additional context. Actors health checks were implemented long time before app health checks. There are also some oddities, such as the fact that if there's no app channel, or the app channel is gRPC, then actors work without health checks too (this can happen for apps that use workflow).

Also worth pointing out that all Actors SDKs today implement /healthz automatically.

[mukundansundar]

Also worth pointing out that all Actors SDKs today implement /healthz automatically.

Currently this situation is observed, when workflows are used, and no actors are used. So it seems that workflow SDKs potentially do not implement the /healthz endpoint ....

[ItalyPaleAle]

That's correct, Workflow SDKs do not implement /healthz.

If the app doesn't have an app-channel, or if the app-channel's protocol is gRPC, then Workflow will work without /healthz

However if the app has a HTTP app channel and uses Workflow, but no Actors, then in that case /healthz needs to be implemented

It's important to note that this is something that's part of Dapr Actors and it's completely unrelated from app health checks. They just happen to shrae the same endpoint.

[shivamkm07]

That's correct, Workflow SDKs do not implement /healthz.

If the app doesn't have an app-channel, or if the app-channel's protocol is gRPC, then Workflow will work without /healthz

However if the app has a HTTP app channel and uses Workflow, but no Actors, then in that case /healthz needs to be implemented

It's important to note that this is something that's part of Dapr Actors and it's completely unrelated from app health checks. They just happen to shrae the same endpoint.

So is it a breaking change then? There can be users who are using workflow without using actors and http app channel right? For actors though actors SDK implement it, should we be checking app Health if healthCheck is disabled?

[JoshVanL]

I can confirm this is a breaking change as this is causing the version skew integration tests to fail.
https://github.com/dapr/dapr/pull/7368/files

[ItalyPaleAle]

One possibility to fix this without breaking changes is to add a CLI flag like --disable-actor-healthchecks (false by default) which allows actors to work without a /healthz endpoint when the app channel is configured as HTTP.

[yaron2]

One possibility to fix this without breaking changes is to add a CLI flag like --disable-actor-healthchecks (false by default) which allows actors to work without a /healthz endpoint when the app channel is configured as HTTP.

Will this mean that users who just want to use workflows and pubsub/service invocation for example and nothing else need to configure this flag? If so, that's a less than ideal user experience because it would require them to take a special action that is easy to miss

[ItalyPaleAle]

Will this mean that users who just want to use workflows and pubsub/service invocation for example and nothing else need to configure this flag?

Yes. The flag will be needed when (and only when) all these 3 are true:

• App uses workflow
• There's an app channel configured
• App channel uses HTTP

So that does include your case of "workflow + pubsub / service invocation" (as long as it's HTTP for app channel).

---

Another option could be to disable actor healthchecks if the app doesn't explicitly list actors in /dapr/config. This means the app could still be an actor host, but only for internal actors (which as of today includes workflow only)?

[yaron2]

Another option could be to disable actor healthchecks if the app doesn't explicitly list actors in /dapr/config. This means the app could still be an actor host, but only for internal actors (which as of today includes workflow only)?

That's a much better option

[ItalyPaleAle]

Another option could be to disable actor healthchecks if the app doesn't explicitly list actors in /dapr/config. This means the app could still be an actor host, but only for internal actors (which as of today includes workflow only)?

That's a much better option

Ok. It should be doable.

If we're going to merge #7022 however let's get that in first please!

[ItalyPaleAle]

Another option could be to disable actor healthchecks if the app doesn't explicitly list actors in /dapr/config. This means the app could still be an actor host, but only for internal actors (which as of today includes workflow only)?

Looks like this behavior was already, incidentally, implemented in master with some recent commit. So we should be all good already.

@shivamkm07 could you verify with the master builds if everything works?

Josh is adding ITs but some user validation would help too

[shivamkm07]

Another option could be to disable actor healthchecks if the app doesn't explicitly list actors in /dapr/config. This means the app could still be an actor host, but only for internal actors (which as of today includes workflow only)?

Looks like this behavior was already, incidentally, implemented in master with some recent commit. So we should be all good already.

@shivamkm07 could you verify with the master builds if everything works?

Josh is adding ITs but some user validation would help too

I tested this locally and verified now workflow runs fine without requiring app to implement healthz endpoint.
Since it's not a breaking change now, created this PR #7405 to remove the patch that was added to fix for this scenario. version skew e2e is passing in the PR for workflows (for dapr-sidecar-master mode, for which the patch was added).