The future of actions

[adlerhurst]

Discussion is closed. The epic can be found here: #7235

Thanks to all the participants. ❤️

Actions are a powerful feature to customise the behaviour of ZITADEL.

This discussion summarises previous discussions held in different resources. There are multiple open topics which should be discussed in this discussion. Feel free provide your feedback, thoughts and additions :)

Resources

The following links are meant as knowledge base. Please add comments in this discussion instead of adding comments to the resource, except the comment is not action related.

• [Epic]: Action possibility on each event and api request #5101
• Extending ZITADEL Actions #4265
• https://docs.google.com/document/d/1Y4p1lXE6-4NQa7rcz6bF1PQteZ1E50ZXtDEtxowcSVk/edit#heading=h.s5bnox1o985j
• https://jamboard.google.com/d/1kz646GB_MXmHjX4sq05dXW3IwO2b35Fv3_rIjTMp8Ys/viewer?f=5
• User Schema #6433 (comment)

Use cases

• Log each API request to zitadel
• Log each event created
• Validate API requests
• Manipulate API requests
• Manipulate API responses
• As a user I want react on each request to check if conditions of the request apply to my use cases. E.g. I want to check if the client is in an ip subnet
• rate limit API's (e.g. limit amount token requests)
• rate limit events (e.g. amount of orgs created)
• As a user I want to extend the claims of the access token to add information needed in my app
• As a user I want to extend the claims of the id token to add information needed in my app
• As a user I want to extend the claims of user info to add information needed in my app
• As a user I want to react on the user.selfregistered-event and add a user grant to authorize the user from the beginning.
• As a user I want to react on created events because I want to notify foreign systems on behalf of events
• As a user I want to add additional commands before the events are created. E.g. to add metadata to a user
• user schema: validate profile

#4265 (comment)

Functional requirements

• Allow action managers to write custom code to customize the behavior of ZITADEL.
• same action can run concurrently with different contexts

Configuration

• Define when a action is triggered
  • on resource API request/response
  • on OIDC request/response
  • on SAML request/response
    • TODO: are there more? e.g. LDAP query, current flows, where would i add scopes in oidc?
  • before OIDC/SAML token creation
  • on event
• See which actions run during a request
• Test actions
• Debug actions
• Define actions on instance and org (TODO: needed?) level
  • TODO: instance actions run before or after org actions?
  • TODO: policy to disable org actions?
• Define an order of execution
• Define a timeout per action => should be implemented in an action
  • TODO: Define a timeout per "flow"
• runtime must provide secrets
• runtime must provide variables

Execution

• failure handing is implemented in the action
• get context specific data
  • request payload
    • TODO: do we have to redact fields like pw or disable functionalities?
  • event
  • context
    • headers
    • e.g. instance, org, user, ...
• manipulate payload
• throw errors
  • TODO: errors should be human readable
  • result in correct response codes
• Import external code
  • http client
  • logger
  • zitadel client
  • jwt parser
• use secrets
• use predefined variables
• FUTURE: Call other action @adlerhurst's opinion: providing a way like github action's depends-on would be more clear

Documentation

• Understand the fields of a request
• Understand the fields of an event
• Understand which parameters are set in the context
• Understand provided functions

Non-functional requirements

• latency
  • until execution starts: p99 < 10ms
  • overhead of execution is < 10ms
• amount of actions per "flow" should not exceed 5
• The size of an action should not exceed 1mb
• Developer experience (TODO: how to measure?)

Are actions on orgs still required?

Discussed in #6926 (comment)

=> outcome they are still required on org level. It's easier to have smaller pieces of code for maintenance

Validation of custom runtime

Discussed in #6926 (comment)

=> outcome: ZITADEL will focus on a webhook style approach which gives full flexibility to the maintainers of actions

Validation ECMA-Script

Discussed in #6926 (comment)

=> This discussion is postponed for the moment based on the outcome of [the previous discussion]{#Validatioon-ECMA-Script}.

Vision

This topic describes what further steps can be taken with actions.

What does that mean for settings/policies and proto validations?

Can we get rid of them and define actions for each request? Do we need to provide system level actions?

=> for the moment they will remain as is

Templates

Duplicate of Marketplace?

=> discussion will be picked up after implementation of actions v2

Git integration

Analog github actions, i can run bash in an action or use an already existing action

=> not planned

Marketplace

Analog github actions marketplace

=> discussion will be picked up after implementation of actions v2

Interaction between actions

If we could define inputs/outputs of actions in a "flow" actions could interact with each other

=> not planned

Notes

Disable email sending is assumed to be solved by Session/User API and is therefore ignored in the use cases.

Index

Word	Definition
Action manager	A user of ZITADEL which is allowed to modify actions

Future

Following topics will result out of the discussions

Concept

Currently there are some actions available on ZITADEL. These actions run in an integrated runtime. This concepts describes the next major versions of actions.

Current actions stay available until triggers or migration paths for existing flows are implemented/documented.

Triggers

Actions are can get triggred at the following points

• On Request: Before a request is validated an processed by ZITADEL
  • these actions run synchronously
  • parameters:
    • payload of the request sent by the client
    • context information about the call
    • ...
  • response:
    • manipulated payload of the request
    • error which is returned to the client
    • error if the server is not able to fulfil the request, this will trigger retries
• On Response: Before the response of ZITADEL is sent to the client
  • these actions run synchronously
  • parameters:
    • payload of the response sent to the client
    • context information about the call
    • ...
  • response:
    • manipulated payload of the response
    • error which is returned to the client
    • error if the server is not able to fulfil the request, this will trigger retries
• On Event: After an event is written
  • these actions run asynchronously
  • parameters:
    • payload of the request
    • context information about the request
    • ...
  • response:
    • error if the server is not able to fulfil the request, this will trigger retries
• FUTURE: possibility to complement tokens (and deprecate actions/v1)

Runtime

ZITADEL will not provide a built in runtime for actions.

Arguemnts:

• Coexistence of IAM and runtime in a single container can compromise the availablility of the IAM
• Runtime hosting makes IAM more complex
• DevX of runtime is difficult to achive
  • Do I have to upload precompiled code?
  • How does the compiler of ZITADEL work?
• Development workflow does not integrate developers day-to-day workflow
• Security concerns like memory lookups

API calls

ZITADEL actions will call external API's which process the the input analog to extensions of chat apps. An example flow can be found in this comment. These API calls support REST and GRPC. The request and response messages are defined by ZITADEL. There will be SDKs provided for the most used prgramming languages for easy integration.

Arguments:

• Seperation of concern
• Bugs in actions don't lock out all users as the action be fixed and redeployed outside of ZITADEL
• Users can choose their own runtime or hoster
• Action development integrates existing workflows
• Developers have full flexibility
• Runtime cannot compromise availablity of ZITADEL

Order of execution

As there can be defined actions on organizations and instances the order of execution forces actions of instances to overrule actions of organizations as follows:

• On Request:
  1. Client call
  2. Org actions
  3. Instance actions
  4. ZITADEL logic
• On Response:
  1. ZITADEL logic
  2. Instance actions
  3. Org actions
  4. Response to client
• On Event:
  1. event is stored
  2. Instance actions
  3. Org actions

Verification of API calls

ZITADEL needs to provide a possibility for actions to verify the API call. As this problem is solved by webhook providers ZITADEL will provide request signatures analog to stripe webhooks.

Action object

The action object describes the call to the API and contains the following fields:

• id: unique identifier of the action
• name: human readable identifier of the action (duplicates allowed)
• call
  • applicationType: JSON or GRPC
  • url: API url to call

Actions can be managed with the following calls:

• CreateAction: adds an action
• UpdateAction: allows manipulation of the action
• DeleteAction: removes the action, the action will also be removed from linked triggers

Trigger object

The trigger object describes when actions are executed.

• id: unique identifier
• (name: human readable identifier of the action (duplicates allowed))
• actions: list of actions executed, the actions are executed in the order provided by the list
  • id: unique identifier of the action object
  • state: active or inactive
  • timeout: duration until ZITADEL returns an error to the client
  • maxRetries: amount of retries until the action fails
  • allowFailure: if true zitadel will continue eventough the action call failed
• on: one of the below options
  • request: name of the API call
  • response: name of the API call
  • event: event type

Triggers can be managed with the following calls:

• SetTrigger: Creates or updates a trigger for a specific point in the flow, there can only be one trigger per trigger point
• AddAction: adds an action to a trigger
• UpdateAction: updates an action
• ChangeOrderOfAction: sets the action at a specific position in the execution order //TODO: feels strange
• DeleteTrigger: removes a trigger

Required changes to meet the concept

TODO

[adlerhurst]

Are actions on orgs still required?

Should b2b customers be able to add actions? => no

TODO: discuss why it's risky/not useful

[andar1an]

How do you differentiate org users and external users with triggers in a project?

It seems that this requires a 2 org approach?

[DoGab]

@stephenandary there are different flow types. This is currently already possible. There is an internal authentication flow and an external one. Check the docs https://zitadel.com/docs/apis/actions/introduction#trigger-types.

[andar1an]

@DoGab I did check the docs. From what I can see the only difference between internal and external flow is whether the user uses internal or external authentication flow which is dictated by idp. A customer can sign up with internal flow - that is email sign up, and with external flow - social login. I don't see differentiation between social login and ldap or ad in external flow triggers. Checking pre create user ctx or post create user data doesn't enable you to have different actions for different projects based on email address for example - it is an org wide trigger, you need to create new org per project?

Quite literally above there is an example of someone needing to create 2 organizations to separate customers and organization users, and thus use org actions.

What if you want to perform different actions for an employee vs a customer on the same project?

If Instance overrides Org, than you need to use 2 orgs? Instead of just setting customer actions with project/api actions.

E.g. I want my org users to trigger actions to add different database claims than my customers for an api or app - maybe for various environments like staging and dev, or for experiments, or for admin permissions.

Am I missing something clear in docs?

[adlerhurst]

Thank you all for joining the discussion. We just added the concept to the original post. Feel free to discuss the concept in this thread.

[mffap]

Would it also make sense to have an App/API level (maybe project is right word)?

The question of limiting actions to an application (maybe project) also came up from customer success side.

[adlerhurst]

Validation custom runtime

Is a custom runtime the best way to provide extendability of the behavior of ZITADEL?

Currently ZITADEL uses goja to execute ECMAScript code. The code is stored inside ZITADEL.

There are currently several flaws in DevX and to fulfill the requirements:

• No semantic versioning
• No diffs
• Does not fit well in developer workspaces
• Limited imports
• No secrets
• No variables
• No debug capabilities
• Not testable
• No zitadel client, some functions are provided through ctx
• No typescript support
• No SDK support
• missing information
• Documentation is much overhead
• Code has to be copy pasted if implemented in an IDE
• function name must match the name of the action
• No other programming languages are supported

Developers are nowadays used to work with git, deploy their code through ci/cd and deploy small pieces of software like FaaS. ZITADEL behaves differently which results in an overhead.

[DoGab]

From my experience working a lot with Zitadel actions i saw that we rewrite the whole actions very often for different organizations.

Loading actions from external sources would be a nice approach. I could imagine that there could be an option to load/execute code in different ways:

• Github: loading actions from GitHub (how to store the GitHub credentials?) - how to provide semantic versioning with monorepos?
• From a package registry (like jfrog) which would add semantic versioning
• Execute a webhook and expect a certain schema as a response (f.e. a json schema).

In general the question arises (at least for me) how to make these actions configurable and pass variables/secrets, etc. to these actions so that they are dynamically?

I don't think that having a VSCode editor in Zitadel would be of a benefit for us. I think the nicest approach would be to be able to store the actions as code somewhere in a version control to fulfil the IaC (infrastructure as code) need.

[andar1an]

I quite agree with that sentiment, but to store within an oci registry as a binary would be easier I think. Code doesn't equate to easy integration unless you are going to infer the code in a runtime.

OCI registries can leverage url to pull even from private repos (for example cloudsmith). Personally I will be testing self hosting through gitlab oci, and would prefer the flexibility to deploy my own action binaries in a FaaS style, if that is not what Zitadel will be doing.

[adlerhurst]

From my point of view just passing an URL and pray that ZITADEL compiles the code correctly would break my experience. There would be a lot of black magic in between which I'm not able to control, even if it's just the compiler version.
Even if we would live in a perfect world and I would have the possibility to configure all the parameters in ZITADEL it's for me still bad devex as I need to leave my known environment and keep track of an additional dependency.

It looks different if ZITADEL would pull releases of my compiled code. This allows me to tune the complier parameters as I'd like them to but doesn't allow me to tune or even choose the runtime. From ZITADEL's perspective executing random binaries even if they are sandboxed is a huge gamble if the runtime isn't separated into its own environment. E.g. calculating the first 100'000 digits of PI which is compute intensive but totally valid code and I'm sure there are tons of other possibilities to influence the overall stability of ZITADEL.
Therefore a runtime inside ZITADEL doesn't feel like the correct way to solve this problem as well to me.

The webhook approach doesn't have the floors of previous approaches. It feels natural to me to deploy some endpoint whether as FaaS or inside my server which is capable to answer action requests. I have the control about it, can store my secretes as I always do and don't have to be feared that my PI calculation has any impact on others or my action get's canceled because of too heavy resource usage.
I think it's feasible to require a basic knowledge about API development to an action developer.
To not ruin my devX ZITADEL must provide a simple solution for me to create the API, this includes definitions of the request and response, an easy way to secure my endpoint as discussed previously.

If you have a look at how others solve the problem there are two approaches

• hosted runtime: github actions require typescript, docker containers or shell, I have no evidence but it's quite obvious that they don't run actions within the same containers as the registry
• API: If you have a look at chat systems like discord, slack or google chat you can provide bots, these bots are called through API's, this gives me full flexibility to me.

What both have in common is the differentiation between the core system and extensions, both separate it as much as possible.
Github has different requirements for example their users need system resources which isn't the case for the others.
The requirements of ZITADEL don't rely on system resources but still provide full flexibility which would be given in an API approach.

If I don't have the possibility to host an API on my own I have to live with trade offs as I have to adjust my development lifecycle anyways. I don't have a clear answer to that problem at the moment of writing, I need some more nights to deal that with me :D
I'm currently unsure where it feels best make the hand over from the action to ZITADEL in a hosted scenario. But I will add a comment as soon as I figured out. #6926 (comment) will bring clarity to that.

[andar1an]

I'm all for hosting my own action runtimes and interacting through an interface/API. Just playing OCI vs Code devil. I personally would prefer if these actions were not within Zitadels scope (pretty sure I said that in one of these threads too) because scaling actions would require scaling a monolithic deployment of Zitadel. I would rather have the flexibility to scale my own actions as needed based on Zitadel requests. If you wanted to do a host runtime wasmCloud may give some inspiration.

[adlerhurst]

Thank you all for joining the discussion. We just added the concept to the original post. Feel free to discuss the concept in this thread

[adlerhurst]

Validation ECMA-Script

ZITADEL currently only allows JS as action programming language. Is JS still our preferred language? Should we move to another runtime like WASM?

The discussion is only relevant if ZITADEL is the runtime for actions.

[yordis]

I was about to recommend the same. WASI Preview 2, and wasmtime are a good alternative to what people have done in the past.

[muhlemmer]

There might be a complication with this as wasm binaries (if packed with a lot of dependencies and also depending on the language) tend to be big.

Why do need need lots of dependencies? Actions are there to modify Request and Response data, not implement a complete web application. If there is extended business data to be handled, the action can (and should) call an external API for that. IMO a size limit will correctly push developers in that direction.

Furthermore. depending on the language there are options to optimize for size as well. For Go one could use tinygo or perhaps other languages could maybe use gcc with the -oS (optmize for size) flag. I'm not sure about llvm for rust, but there must be a similar option (or just use the gcc front-end for rust).

But if storing the file is the issue you could implement an option of providing a url (insted of uploading it) for the wasm file. If memory is the concern it might spike.

I do not believe storage is the problem, but rather performance. 1MiB of WASM is a lot of bytecode to compile (or interpret directly). As zitadel cloud runs a backend for many instances, actions will need to be retrieved on a request basis. Compile once and execute many times would not scale well in our case, because that would require lots of RAM if there are many actions.

The size of an action should not exceed 1mb

Perhaps there is some wiggle room to the exact number. We need a practical value. It could be made configurable, where self-hosters and enterprise can raise the limit if required for their use case, while still protecting the performance of cloud. Based on feedback we can then always adjust the number in cloud.

[buehler]

I really like the idea of exitism or WASI runtime. To allow people to use other / faster / their language of choice gives them easy access to these features (actions and validators). As far as I understand, the bytecode (compiled wasm) could be loaded directly for an action and be executed.

My concern would be about security: How do you give such powerful tools without compromising security of the whole system?

I would think of giving a specific input and expecting a specific and well-known output. This is a simple json/protobuf/whatever object (per action) and nothing else. As such, people must implement that given function. I'm thinking about "cloud functions" (or AWS lambda or however you name them). With a system like that, it would greatly increase the developer experience.

Coding wise, you could provide some form of the monaco editor in the GUI (vscode) or use (as I stated in the discussion above) git hooks / repositories to let people provide their code and compiled result via git repositories. With repos, devs can also use CI/CD to test their actions against a mocked instance or something.

[andar1an]

wasm/wasi is sandboxed - based on the varying opinions I think Zitadel should not bring hosting actions within it's scope, and instead create an interface by which to communicate with externally hosted actions.

[adlerhurst]

Thank you all for joining the discussion. We just added the concept to the original post. Feel free to discuss the concept in this thread

[andar1an]

I did not see this use case covered in the beginning, but I am working on using actions to modify a JWT to integrate application databases.

Right now I am planning on authenticating a user on the database with the JWT signed by Zitadel, however, revocation will require an action which can invalidate that token in the database for a specific user.

Generally, I hope that Zitadel can be the single point of truth for any infrastructure integration for the application, and actions may be a crucial part of this.

Are actions limited to modifying JWTs, or manipulating Zitadel data, or can they also be used to make API requests to an external database for example?

It would be really cool if I could investigate if any risk exists when replaying a jwt, and if api actions can remove that risk.

[adlerhurst]

The goal is to allow external calls, it's not clearly described in the requirements but at least a http call should be possible. Depending on the discussions above the runtime should provide the option or if zitadel does only a call to a foreign API you can implement whatever logic you prefer

[adlerhurst]

We just added the concept to the original post. Feel free to discuss the concept in this thread

[ghstahl]

I have an absolute need to at least add to any claim that already exists.
#6979

The point of actions is to be able to change the final outcome of what is present in an id_token and access_token.
Compliment claims really needs to be mutate claims.

Speaking about aud. If I mutate this claim by completely replacing it, that's on me. My business, my use case, breaks no other user of zitadel.

There are some claims that are part of the spec the I should not be able to change. expiration, etc.
I would only allow the following not to be changed.

"exp": 1701106743,
"iat": 1701103143,
"iss": "http://zitadel.blah.com",
"jti": "clicc9tdnc9c73fu7g30",

Everything else should be fair game.

"sub":"my subject, not zitadels"
"aud":"my auds, not zitadels"

[muhlemmer]

If you need a JWT that does not contain ZITADEL's data, but something of your own, why not just create your own JWTs in your own app? What do you need zitadel for?

[ghstahl]

@muhlemmer

What do I need zitadel for: All the authentication flows. Not to be confused with the authorization stuff. I rely on zitadel to provide me with proof-of-life. Zitadel does all the integrations to azure, saml, ldap, oidc, etc. It should have a nice admin portal. It should have nice user console that lets the user change passwords, etc. All those google accounts type of stuff. Linking accounts, setup MFA, emailing users. Authentication should be what Zitadel does best. Authorization is very opinionated hence.... token_exchange

Why not just create your own: That is exactly what I am doing. I have a token_exchange that immediately takes a zitadel token where I only need to know user id and mint my own. Thats works great until....

We started using a third-party app that is very immature in its SSO integration. They expect that the access_token they get from the IDP(Zitadel) has got everything needed for downstream api calls. I can tell my home-grown apps to use the token_exchange, but here I cannot. I have even contemplated spinning up my own OIDC compliant login flow that orchestrates this. i.e. fronting Zitadel for this stupid third-party app, and I have done this before but it's bad form.

Using actions (compliment claims) I am able to add all my claims in, but there is also aud verifications and the zitadel auds are meaningless to my internal apis.

NOTE: zitadels competitors (Auth0) let me add auds, so I expect to not be losing features when moving over to zitadel.

My apis can accept tokens from multiple issuers. Right now they accept tokens issued by Zitadel and my internal OAuth2 service. The access_tokens both provide have the same claims, with the exception of the aud. I have disabled aud verification to get past Zitadel not letting me add to that claim, but I will eventually need to do something to turn aud verification back on to pass compliance. I really don't want to spin up an orchestrator.

[muhlemmer]

On the User V2 API it is already possible to create an user with an optional ID. If omitted there is an ID generated, just like on the V1 API. sub is the ID of the authenticated user.

You mentioned subject before, would it work for you to create the user with the external ID expected by the application?

Likewise, when we add the Project and Application resources to the V2 API we also could allow custom ID settings and subsequently they would be present in the aud.

Perhaps an action wouldn't even be required.

---

Alternatively, we could allow appending to aud. I understood initially you want to replace the aud. I think completely replacing fields such as sub and aud isn't the best idea. How would you handle refresh tokens in that case?

[ghstahl]

refresh tokens should always call the actions.

That would give me a chance to kill the refresh if that user is in bad standings on my side.
I map, using openFGA, the user to permissions on my side, so having the compliment-claims action being called during a refresh is essential. This allows me to give a new set of claims if the user's roles changes.

Note: I don't use zitadel to manage roles. Just proof-of-life.

Right now, I don't have the need to swap the subject, and I only put is as something that should be allowed.
append to AUD would be great.

I have a custom claim right now for subject: my_real_user_id = {{blah}}

[t3cneo]

Well I both agree and disagree.

If you take a look at OpenID Connect specifications, here is what you'll read regarding the token specs about audiences :

aud, REQUIRED. Audience(s) that this ID Token is intended for. It MUST contain the OAuth 2.0 client_id of the Relying Party as an audience value. It MAY also contain identifiers for other audiences. In the general case, the aud value is an array of case-sensitive strings. In the common special case when there is one audience, the aud value MAY be a single case-sensitive string.

If you scroll further down to Token Validation specs, here is what you'll find about audiences :

2, The Client MUST validate that the aud (audience) Claim contains its client_id value registered at the Issuer identified by the iss (issuer) Claim as an audience. The aud (audience) Claim MAY contain an array with more than one element. The ID Token MUST be rejected if the ID Token does not list the Client as a valid audience, or if it contains additional audiences not trusted by the Client.

That being said and because I think it's in Zitadel's (and as users, our) interest to stick to the specifications by default, I think the audience should be populated by default only with Client ID.

I also think it should be possible to add anything we'd like in the audience part of a token with custom actions (add the Project ID for instance, or any other attribute for the matter) without being able to remove the Client ID which is required by the specs.

By the way I was brought here by the fact you can't setup Zitadel with Proxmox.

As of now Proxmox apparently uses openidconnect-rs project which, I think, just follows Open ID connect specs. (-> reject the token because Zitadel's project ID which is unknown to proxmox is in the aud array)

So being able to add something to audience with action -> yes, it's not against the specs
Being able to remove Client ID from audience with action -> IMO no, it would be against the specs
In the same time, having a special function to strip down audience to it's required minimum, Client ID, with action -> yes please, might be easier than modifying how token are forged

[adlerhurst]

Concept

If you have questions or proposals for the concept of the original post feel free to share them in this thread :)