How we are using CEL in agentgateway

[howardjohn]

I figured the most useful thing an OSS project can get is feedback, so wanted to share how we are using cel-rust in https://github.com/agentgateway/agentgateway/. A quick tl;dr of agentgateway is it is a reverse proxy that is focusing on AI use cases and Kubernetes.

We use CEL throughout the project, and are constantly expanding our usage. Some examples:

• Authorization policies
• Customizing logging/tracing fields
• Rate limiting (selecting which attributes to rate limit based on)
• Request transformation (e.g. set x-user: 'jwt.claims.sub')

On a single request we may execute 10s or even 100s of CEL expressions, and we aim to make the gateway as high performance as possible, serving ~50k QPS per core. As such, performance of evaluation is critical.

One challenge we have around this is giving the expressions access to the right information in a performant way. Throughout a given request we may evaluate CEL policies in different stages. For example, I will evaluate authz policies based on the request, but a log after the entire response is complete. If a user wants to do something like log a request header, we need to make sure they have access to these (when otherwise we would discard these, as they had already been sent to the destination). Or a more extreme example, the request body, which requires buffering up a full copy of the request body in memory.

To handle this, we incrementally build up a CEL context and conditional copy data into the context based on which fields are required by expressions. To do this, we evaluate the AST to extract variable references (inspired by Kuadrant, thanks @alexsnaps) and then conditionally copy in attributes.

One area of particular interest to our project is the performance of trivial variable access operations, such as jwt.claims. These are very common for our use cases, but are relatively costly at ~50-100ns. Given our performance targets, we have about 20000ns total for a complete request, so each CEL expression we evaluate eats up 1/200th of our budget.

So far we are really enjoying cel-rust and looking forward to making more contributions!

[clarkmcc]

@howardjohn thanks for sharing your use-case and the context is very helpful to understand.

To do this, we evaluate the AST to extract variable references

How we could improve the existing API for this to meet your needs.

[howardjohn]

For the expression request.headers.map(h, h) || request.body || response.status, our current impl returns request, request.body, response, h. references().variables() returns ["request", "h", "response"]. So we don't get the nested expression lookups.

(note: our implementation specifically ignores some nested fields but on others , which is why we get request.body but not request.headers)

Notably, both of these probably shouldn't include h for our use case, but its ok for us to have false positives.

[cgettys-microsoft]

I do pretty similar things for different reasons (effectively short-circuiting ahead of time). But I don't currently need nesting today (but I was mulling grouping some of the fields into "structs", at which point I well, would.

[cgettys-microsoft]

FWIW @howardjohn - #171 makes the Program expose the Expression so that you can walk it if you need to.