[release-1.12] Fix race condition in app health and actors/workflow initialization

[ItalyPaleAle]

Fixes the issue discovered in #6968 (comment)

Dapr reports in /healthz (and /healthz/outbound) a ready status when the API servers are ready, but the actor runtime (and thus workflow) is initialized asynchronously.

This means that apps that try to invoke actors or use workflow fail. We've experienced this error in E2E tests in #6968, where a refactoring on unrelated things made the race condition appear.

This PR fixes the issue by making all actor and workflow APIs (including those managed by the WF engine) block while the actor runtime is initialized (successfully or not).

I validated this fix by adding a time.Sleep(10 * time.Second) at the start of runtime.appHealthReadyInit

[ItalyPaleAle]

/test-sdk-all

[ItalyPaleAle]

/ok-to-test

[dapr-bot]

Dapr SDK Java test

🔗 Link to Action run

Commit ref: 7f14294

❌ Java SDK tests failed

Please check the logs for details on the error.

[dapr-bot]

Dapr SDK Go test

🔗 Link to Action run

Commit ref: 7f14294

✅ Go SDK tests passed

[dapr-bot]

Dapr E2E test

🔗 Link to Action run

Commit ref: 7f14294

✅ Build succeeded for linux/amd64

• Image tag: dapre2e888e959df3l
• Test image tag: dapre2e888e959df3l

✅ Infrastructure deployed

Cluster	Resource group name	Azure region
Linux	Dapr-E2E-dapre2e888e959df3l	westus3
Windows	Dapr-E2E-dapre2e888e959df3w	westus3
Linux/arm64	Dapr-E2E-dapre2e888e959df3la	eastus

✅ Build succeeded for windows/amd64

• Image tag: dapre2e888e959df3w
• Test image tag: dapre2e888e959df3w

❌ Tests failed on windows/amd64

Please check the logs for details on the error.

✅ Tests succeeded on linux/amd64

• Image tag: dapre2e888e959df3l
• Test image tag: dapre2e888e959df3l

[dapr-bot]

Dapr SDK Python test

🔗 Link to Action run

Commit ref: 7f14294

✅ Python SDK tests passed

[dapr-bot]

Dapr SDK JS test

🔗 Link to Action run

Commit ref: 7f14294

✅ JS SDK tests passed

[JoshVanL]

@ItalyPaleAle Can you add an integration test to prove this works? We can test by running daprd (with a placement address configured), testing that healthz fails, then starting placement and wait for the healthz endpoint to return healthy.

[ItalyPaleAle]

@ItalyPaleAle Can you add an integration test to prove this works? We can test by running daprd (with a placement address configured), testing that healthz fails, then starting placement and wait for the healthz endpoint to return healthy.

No, this is not what this PR does.

1. Healthz hasn't changed
2. Also this isn't impacted by whether placement is configured or not. If the actor runtime can't start (because there's no placement configured, or because there's no actor state store, or because the app doens't respond to /dapr/configuration), healthz still returns true and the actors runtime is considered "initialized" (even though is inactive)

The issue this fixes is different:

1. Apps that use actors (and workflow) are requested to wait for /healthz to report success. See

   dapr/tests/apps/workflowsapp/Controller.cs

   Line 40 in e4f8901

   await daprClient.WaitForSidecarAsync();

2. Actors are initialized async in the meanwhile
3. The app tries to invoke actors or workflow (example:

   dapr/tests/apps/workflowsapp/Controller.cs

   Line 43 in e4f8901

   var startResponse = await daprClient.StartWorkflowAsync(

   )
   • If the initialization of actors is done, this succeeds
   • If not, it fails

The fix makes it so calls like the one above will block until actors is initialized, whether successfully or not.

I am not sure how to even write a test for this, since testing for absence of race conditions is always a hell of a problem.

Not relevant

[JoshVanL]

@ItalyPaleAle Please cherry pick #6974 which will pass on this branch. It doesn't test workflows as we don't have any infra for them in integration tests yet but at least covers actors.

[ItalyPaleAle]

@ItalyPaleAle Please cherry pick #6974 which will pass on this branch. It doesn't test workflows as we don't have any infra for them in integration tests yet but at least covers actors.

Thank you, merged this

[codecov]

Codecov Report

Attention: 19 lines in your changes are missing coverage. Please review.

Comparison is base (9514d96) 64.84% compared to head (82f9ad9) 64.65%.

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.12    #6972      +/-   ##
================================================
- Coverage         64.84%   64.65%   -0.19%     
================================================
  Files               228      230       +2     
  Lines             20848    20847       -1     
================================================
- Hits              13518    13478      -40     
- Misses             6203     6236      +33     
- Partials           1127     1133       +6     

Files	Coverage Δ
pkg/runtime/config.go	30.00% <100.00%> (+0.35%)	⬆️
pkg/grpc/api.go	72.91% <93.33%> (+<0.01%)	⬆️
pkg/grpc/universalapi/api_workflow.go	80.47% <85.71%> (+0.22%)	⬆️
pkg/http/api.go	77.95% <93.75%> (-0.35%)	⬇️
pkg/runtime/wfengine/wfengine.go	59.74% <91.66%> (+2.59%)	⬆️
pkg/grpc/universalapi/universalapi.go	33.33% <33.33%> (ø)
pkg/grpc/universalapi/api_actors.go	40.00% <40.00%> (ø)
pkg/runtime/runtime.go	73.14% <58.82%> (+0.13%)	⬆️

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[artursouza]

LGTM