Design the interface that allows multiple versions of Reminders/ Actors implementation

[rabollin]

This is a placeholder to add a design proposal for Interface that allows multiple reminders/ actior implementation(versions) in-tree and implement it in 1.13 once the design proposal is approved

[DeepanshuA]

/assign

[DeepanshuA]

Based on the discussion that I had with Alessandro (and I think he had with Yaron), I have put together this:

Interfaces for Actors and Reminders

There are two places where APIs or functions required for Actors and Reminders are defined.

1. Protos in dapr/proto package.

2. Interfaces in pkg/actors/internal for placement and reminders to be updated

Out of these 2, protos are not to be updated, as every implementation will have their own RPCs defined accordingly.

Only interfaces in pkg/actors/internal have to be updated. Though, a major work has already been done for them in release 1.12 already by segregating actors, reminders and timers interfaces (#6669 ).

Idea behind this is that interfaces defined in pkg/actors/internal will be used to form a Building Block / Client which will implement all these methods defined in these interfaces.

Whereas, methods defined for Services in protos will be implemented by respective new/existing Control Plane Service(s).

And, the methods from these clients will call methods of the servers, wherever required.

Depending on the implementation, it can be decided that what all RPCs need to be defined for a Service. Example, for a Placement Service which just creates and pushes Actor Types v/s Dapr Hosts table information to All sidecars, only table related ReportDaprStatusneeds to be as rpc.

Whereas, for a Service, where Actor information is Pulled by sidecars from Actors Service needs some methods like LookupActor or ReportActorDeactivation, which thus form a part of RPCs.

For some minor changes left in interfaces, #7281 has been raised.

How to build and deploy Actors/Scheduler/Placement Service?

There are two sub-systems that we are dealing with here: Actors and Reminders. And, as we know, going forward, there can be multiple variants of Services providing either of these sub-systems or both.

There will be multiple possibilities, like:

1. Placement Service used for Actors and Distributed Scheduler used for Reminders.

2. ABC Service used for both Actors and Reminders.

3. XYZ used for Actors and Distributed Scheduler used for Reminders.

So, how to decide what all to deploy in what case for user?

Steps:

1. Dapr Configuration changes to depict what services to load for Control Plane and what not.

2. Component loading capability, specifically by operator, for Control Plane. This is required, in case CP requires a component to be loaded.

Step 1: Configuration changes:

A) Changes in charts/dapr/charts/dapr_config/templates/dapr_default_config.yaml

B) Changes in charts/dapr/values.yaml for global values to take inputs from users.

C) Changes in templates/deployment.yaml of particular service in charts to check if it needs to be deployed:

Step 2: Component loading:

A) Take out operator code in a library to specifically load a component, scoped only for a control plane service, in dapr-system namespace.

B) Secrets also to be loaded.

@dapr/maintainers-dapr @dapr/approvers-dapr

[JoshVanL]

Please can you move this proposal into a date/proposal PR so we can track ADR there, and be able to incline comment.

[ItalyPaleAle]

Following up on the design on how it's exposed to end users, as discussed with @yaron2, here's what I am thinking about CLI flags and annotations:

• Injector is configured (via env vars set by Helm) with what actor and/or reminders service to use. Default value is to use placement and no reminders service (so it uses built-in reminders). It is possible to set in the Helm chart values to use different reminders and/or placement services.

For reminders:

• Default is to set nothing in the CLI, which means use the built-in reminders (that use state stores)
• If in the Helm chart the injector is configured with a reminders service, then it adds the CLI flag --reminders-service <name>:<address>, for example --reminders-service scheduler:k8s-svc-name.svc.cluster.local:port (the <name> part tells daprd which implementation to load)

For placement:

• If the user has set the dapr.io/placement-host-address annotation, that forces the injector to set --placement-host-address to the value passed, as is the case today
• Otherwise, if there's no explicit dapr.io/placement-host-address annotation:
  • If the configuration (via Helm) is to use placement, then sets --placement-host-address exactly as today (this is default value)
  • If the configuration is to use another actors service, it sets --actors-service <name>:<address>:<port> following the same logic as the reminders svc)

This is backwards-compatible.

Exception: when --actors-service or --reminders-service are set, if the injector is trying to inject a sidecar that use an oldeer version of Dapr, daprd that will crash because the flag doesn't exist. This is a "feature": if the cluster wants to use a different actors/reminders service, versions of Dapr that don't support that should crash

[yaron2]

Following up on the design on how it's exposed to end users, as discussed with @yaron2, here's what I am thinking about CLI flags and annotations:

• Injector is configured (via env vars set by Helm) with what actor and/or reminders service to use. Default value is to use placement and no reminders service (so it uses built-in reminders). It is possible to set in the Helm chart values to use different reminders and/or placement services.

For reminders:

• Default is to set nothing in the CLI, which means use the built-in reminders (that use state stores)
• If in the Helm chart the injector is configured with a reminders service, then it adds the CLI flag --reminders-service <name>:<address>, for example --reminders-service scheduler:k8s-svc-name.svc.cluster.local:port (the <name> part tells daprd which implementation to load)

For placement:

• If the user has set the dapr.io/placement-host-address annotation, that forces the injector to set --placement-host-address to the value passed, as is the case today

• Otherwise, if there's no explicit dapr.io/placement-host-address annotation:

  • If the configuration (via Helm) is to use placement, then sets --placement-host-address exactly as today (this is default value)
  • If the configuration is to use another actors service, it sets --actors-service <name>:<address>:<port> following the same logic as the reminders svc)

This is backwards-compatible.

Exception: when --actors-service or --reminders-service are set, if the injector is trying to inject a sidecar that use an oldeer version of Dapr, daprd that will crash because the flag doesn't exist. This is a "feature": if the cluster wants to use a different actors/reminders service, versions of Dapr that don't support that should crash

Looks good

[ItalyPaleAle]

I believe we can close this issue as completed. Please confirm!