Authentication from the Application to Dapr runtime

[RicardoNiepel]

In what area(s)?

/area runtime

/area operator
/area placement
/area docs
/area test-and-release

Current situation

Increasing the security of Dapr and applications using Dapr, it would be necessary to secure Dapr-to-app communication. This also relates to /dapr/docs/issues/346

As stated at Security Concepts

"Dapr assumes it runs in the same security domain of the application. So, there are no authentication, authorization or encryption between a Dapr sidecar and the application."

Currently it is normal for the sidecar patterns to assume this (e.g. at Service Meshes).
But with Dapr are coming more possibilites and thus also more responsibilities.

Challenge

It is possible to leverage existing pubsub topics, bindings and even normal service invocations, if the application container itself is compromised or another container in the same pod (Kubernetes env). They all can just talk per localhost with the Dapr sidecar and without any auth and knowing about specific connection string, host addresses etc. they can store/manipulate data into the system.

When executing it standalone the situation is even worse, because no pod as a boundary exists.

Describe the feature

Provide any authentication mechanism between Dapr and the application, e.g. a secret/token.

[youngbupark]

@yaron2 does linkerd and istio encrypt between its proxy sidecar and app ?

[yaron2]

@yaron2 does linkerd and istio encrypt between its proxy sidecar and app ?

No

[yaron2]

That's because they can't: the app doesn't talk to Envoy or Linkerd-proxy directly. the proxies intercept the traffic using iptables, so they can't enforce authentication for it.

But it doesn't mean we can't do token authentication in Dapr..

[yaron2]

Design Proposal

This design proposes the means by which Dapr could allow authenticated calls to its APIs.

API authentication using tokens is a common theme and an overview is out of scope of this proposal.
For more information on token based authentication, please see here and here.

The rational is that Dapr will inspect a token for every incoming request to its public APIs (user facing APIs) for both gRPC and HTTP, and only allow authenticated requests to pass.
A call to Dapr (for example, to v1.0/state) is considered authenticated if a valid token is present on the call from the client side.

Implementation

Since Dapr can call back into the application, we can achieve a more secure architecture where the Dapr sidecar itself generates a new token every time it boots up, and communicates it to the application over the configured application port.

Doing this allows us to leave the token in the Dapr sidecar's memory address space, as opposed to keeping it in a secret store or any other location which increases the attack surface.

Once the application receives the token, it should then include it in the header or metadata of the Dapr API call.
Any calls arriving to Dapr not including the token will be rejected.

Pros

• Token is self-generated in the sidecar and not kept at rest anywhere
• Zero additional ops maintenance for developers and operators
• No dependency on secret or state stores
• Works on every hosting platform

Cons

• Apps that do not need Dapr to call back into them will need to open a web-server to receive the token. After the token is received, the user can close and dispose of the web-server.

Configuration on Kubernetes

API authentication will be turned off by default.
To enable token based authentication, the following annotation will be used:

dapr.io/token-authentication: "true"

Configuration on Self Hosted

daprd will receive an additional flag: --token-authentication true.

Token metadata inspection

Dapr will inspect the token authentication flag when it inits.
If the flag is set to true, Dapr will generate a token based on a sufficiently secure algorithm to represent the API token.

HTTP

Dapr will inspect an incoming call for the API token via the following header:
dapr-api-token: <token>

gRPC

Dapr will inspect an incoming call for the API token on the gRPC metadata for the following value:

dapr-api-token[0].

[youngbupark]

Thanks for writing this spec. Overall LGTM.

I have a few feedbacks:

• Can we use standard header Authorization header?
• In terms of security, App Callback may not accept additional token from dapr sidecar once it gets it at the first time.

[yaron2]

Thanks for writing this spec. Overall LGTM.

I have a few feedbacks:

• Can we use standard header Authorization header?
• In terms of security, App Callback may not accept additional token from dapr sidecar once it gets it at the first time.

Yeah we can use the auth header for HTTP.
Can you elaborate on the AppCallback? Not sure I understand

[youngbupark]

• In terms of security, App Callback may not accept additional token from dapr sidecar once it gets it at the first time.

Yeah we can use the auth header for HTTP.

grpc-gateway project use the same authorization metadata for grpc and asp.net core uses the same authorization header for http and metadata. - https://docs.microsoft.com/en-us/aspnet/core/grpc/authn-and-authz?view=aspnetcore-3.1

Can you elaborate on the AppCallback? Not sure I understand

There is a possibility that attacker can inject wrong token to user app if user app continue to accept the token. So we may need to give user the guideline to ignore additional tokens if user app already got token.

[yaron2]

• In terms of security, App Callback may not accept additional token from dapr sidecar once it gets it at the first time.

Yeah we can use the auth header for HTTP.

grpc-gateway project use the same authorization metadata for grpc and asp.net core uses the same authorization header for http and metadata. - https://docs.microsoft.com/en-us/aspnet/core/grpc/authn-and-authz?view=aspnetcore-3.1

Can you elaborate on the AppCallback? Not sure I understand

There is a possibility that attacker can inject wrong token to user app if user app continue to accept the token. So we may need to give user the guideline to not accept token forever.

Oh ok, yes, certainly.
If that happens BTW, the attacker will not have access to Dapr, since the token he injects for the app will not be recognized in Dapr.

So yes, we can guide the user on accepting the token once, or to check the port from which the call came to make sure that Dapr is the one who called the app.

[youngbupark]

When it comes to GUID generation for token, we need to make sure that the generated token is secured.

According to http://www.ietf.org/rfc/rfc4122.txt,

Do not assume that UUIDs are hard to guess; they should not be used as security capabilities (identifiers whose mere possession grants access), for example.

I do not remember the exact code, but I used the secure random function with HMAC in the previous production code.

[yaron2]

When it comes to GUID generation for token, we need to make sure that the generated token is secured.

According to http://www.ietf.org/rfc/rfc4122.txt,

Do not assume that UUIDs are hard to guess; they should not be used as security capabilities (identifiers whose mere possession grants access), for example.

I do not remember the exact code, but I used the secure random function with HMAC in the previous production code.

Ok I changed the description to a more general notion and removed the GUID reference.

[shalabhms]

Going through the design. It looks good to me. I am finding and reading more references around this flow and will comment if I have something.

[shalabhms]

Reading on as how application containers be compromised, found this one a good reference .

[youngbupark]

@yaron2 one thing that I missed is that user app will be able to pass their own authorization token to service invocation api :) So in this case, your original proposal(dapr-api-token) is better than using Authorization.

[amanbha]

Thanks, @yaron2 LGTM overall. I like that its opt-in feature, this is specially applicable to apps which are purely making client calls to Dapr.

I agree with @youngbupark that its better to use a custom header name.

This feature would need changes in all the SDKs (specially Actors to keep the token and present it on all subsequent calls into Dapr) when this is opted in by user.

Could you add following clarifications to the proposal as well:

1. Specifics of the call, will it be a new http/grpc call into user code?
2. How would dapr side car restarts be handled:
   Consider scenario, where Dapr sent a token, app is using it. Then Dapr restarts, app keeps on using the older token. This would need guidance from our side for users. Even if we document that users must use the latest token, there will be a race for some calls. Lets be clear on what will be Error code in this case and what app needs to do.
3. How would app restarts be handled?
   Consider scenario, where Dapr sent a token, app is using it. Then app restarts, it will not have a token. Can app signal to Dapr by making a call, so that Dapr side car makes another call into user app to send the secret?

[yaron2]

I agree with @youngbupark that its better to use a custom header name.

Yeah it was actually a custom header name in the beginning, I reverted back to a dapr-api-token name.

Specifics of the call, will it be a new http/grpc call into user code?

This is for user code calls into Dapr via HTTP or gRPC.

How would dapr side car restarts be handled:

If Dapr restarts, it will reissue a new token to the app. Since a token will only be issued once on boot, there can be no race conditions here, as the desired behavior is last write wins.

How would app restarts be handled?

We already have health checks in place in the sidecar to verify if the app is healthy or not. we can use those primitives to resend the token once the app becomes healthy again.

[amanbha]

Specifics of the call, will it be a new http/grpc call into user code?

This is for user code calls into Dapr via HTTP or gRPC.

App is not asking for initial token, Dapr is sending it to the app which then app is presenting on future calls. I was referring to specifics of the call on which Dapr will deliver the intial token to app.

How would dapr side car restarts be handled:

If Dapr restarts, it will reissue a new token to the app. Since a token will only be issued once on boot, there can be no race conditions here, as the desired behavior is last write wins.

When app & Dapr comes up, app will get the token from Dapr, app will keep token in memory and use it for all the requests to Dapr. Now consider at time T1 Dapr restarts and it sends a new token at Time T2. To use this new token we will have to share guidance for customers for token caching. Even with guidance, for Calls originating from app between time T1 and T2 will use older token and Dapr should return correct error code so that app can retry it with new token.

How would app restarts be handled?

We already have health checks in place in the sidecar to verify if the app is healthy or not. we can use those primitives to resend the token once the app becomes healthy again.

Leveraging health checks sounds good, but that still leaves a race. App going down and coming back up between 2 heath checks and Dapr wont notice that app went down.

[LMWF]

How will the token be stored in dapr? Will it be encrypted?

[jjcollinge]

Might be slightly tangential to this specific issue but does the App need a mechanism by which it can verify the integrity and identity of Dapr itself - i.e. if the side car injector was compromised and injected a malicious dapr image - could we provide a way that the app can detect it’s not speaking to who it expects to be. Or is the platform and Dapr assumes to be safe and trusted?

[yaron2]

How will the token be stored in dapr? Will it be encrypted?

It remains in-memory, not on disk.

[yaron2]

Might be slightly tangential to this specific issue but does the App need a mechanism by which it can verify the integrity and identity of Dapr itself - i.e. if the side car injector was compromised and injected a malicious dapr image - could we provide a way that the app can detect it’s not speaking to who it expects to be. Or is the platform and Dapr assumes to be safe and trusted?

OCI image integrity is a host platform (or even a layer above it) concern.

[yaron2]

Update

There's one scenario where proposal 1 fails to deliver on: when Dapr is exposed to callers outside of the pod (or local network namespace), it will not be able to communicate the authentication token unless the caller is the app which Dapr knows how to call back into.

A concrete example of this would be Ingress: For example, running Dapr next to Nginx, where Dapr cannot call back into Nginx, or to the actual caller which Nginx is proxying the request on behalf of.

IMO this is enough to rule out proposal 1, unless someone feel differently.

Proposal 2

This proposal builds on proposal 1 for the means of authentication, which is using an API token to validate the caller.

Instead of generating the token from within the Dapr runtime, the API token can optionally be injected into the Dapr runtime via an environment variable.

This allows operators to create the token and deliver it to their applications using their existing CI/CD tools and frameworks.

For standardization purposes, the token itself is a standard JWT token that is generated by the user.
While Dapr is not the JWT token issuer in this case, being explicit about JWT as the standard will help keeping users in the right lane.

Dapr only needs to be injected with this token, and it is the app's responsibility to get hold of the token and deliver it to Dapr.

Kubernetes walkthrough

On Kubernetes, the flow can be the following:

1. Operator generates a JWT token string using their own secret key. The secret key does not need to be stored anywhere in the cluster, and Dapr does not need to know about the secret key.

2. A Kubernetes secret is created to hold the token. The secret is created per namespace.

kubectl create secret generic dapr-api-token --from-literal=token=<JWT-TOKEN>

3. The user tells Dapr to secure its APIs with the token by adding the dapr.io/api-token-secret annotation to the pod.

4. The sidecar injector will add an environment variable (DAPR_API_TOKEN) to the daprd container with the secretKey reference. The pod will then consume the token from the secret specified as the value in the annotation.

Self hosted walkthrough

1. Operator generates a JWT token string using their own secret key. The secret key does not need to be known to Dapr.

2. The user sets the environment variable for the daprd process or session:

export DAPR_API_TOKEN=<JWT-TOKEN>

Token expiry

The operator is free to choose whatever expiry date they want when issuing the token. In order to rotate it, the secret needs to be updated (or recreated) with the new token and the container/process to be restarted.

In clustered environments, token rotation can be done with zero downtime.

Rotation walkthrough on Kubernetes

1. The user updates the secret with the new JWT token

2. The user applies the secret to the namespace: kubectl apply -f token-secret.yaml

3. The user triggers a rolling upgrade of the deployment: kubectl rollout restart deploy/myapp. If the deployment has > 1 replicas, a zero downtime rotation will occur

/cc @amanbha @youngbupark @jjcollinge @LMWF @shalabhms

[jjcollinge]

@yaron2 SGTM, just to clarify in the Kubernetes scenario, would the injector resolve the secret and pass the value in or would it simply pass the secret name so the dapr runtime can resolve it (which could allow token rotation without down time).

[yaron2]

@yaron2 SGTM, just to clarify in the Kubernetes scenario, would the injector resolve the secret and pass the value you in or would it simply pass the secret name so the dapr runtime can resolve it (which would allow token rotation without down time).

It would simply pass the secret name ref, and the Pod will resolve it.

[amanbha]

@yaron2 yes, Proposal 2 sounds lot better than Proposal 1, it doesn't need app to start a server to recieve the token and it solves the issues of sidecar, dapr restarts as mentioned earlier.

The walkthrough is good, it clarifies a lot of things. Could you also add walkthrough steps for token update/rotation, To support rotation with minimal downtime, do we need to introduce Primary and Secondary tokens.

[yaron2]

@yaron2 yes, Proposal 2 sounds lot better than Proposal 1, it doesn't need app to start a server to recieve the token and it solves the issues of sidecar, dapr restarts as mentioned earlier.

The walkthrough is good, it clarifies a lot of things. Could you also add walkthrough steps for token update/rotation, To support rotation with minimal downtime, do we need to introduce Primary and Secondary tokens.

Updated.

[shalabhms]

Thanks @yaron2 , proposal 2 is better. Just curious , is Step 4 actually needed for setting environment variable. Can we just provide the secret config reference [not the key of course] in the annotation and not actually set the environment variable? This is similar to what we do for other configuration like tracing configuration. Is that possible or step 4 is something that can not be avoided?

[yaron2]

Thanks @yaron2 , proposal 2 is better. Just curious , is Step 4 actually needed for setting environment variable. Can we just provide the secret config reference [not the key of course] in the annotation and not actually set the environment variable? This is similar to what we do for other configuration like tracing configuration. Is that possible or step 4 is something that can not be avoided?

Using a secret reference for an environment variable is the idiomatic way to do it, and it won't require Dapr to have extra permissions or code to pull the secret itself.

[shalabhms]

Got it. Thanks for clarification @yaron2 .

[yaron2]

There's now a PR: #1606