[v0.1] Envoy ExtProc integration for AI gateways

[rootfs]

Acceptance:

• ExtProc header/body mutation consistent with GIE/LLM-d/Envoy AI Gateway filter chains (documented setup for each)
• Example Envoy configs for common patterns (e.g., A/B test, canary routing)

[LYZJU2019]

I can take a look at this!

[rootfs]

Thanks! If possible, can you outline a plan and we can loop in other folks too? Since this is very community collaborative, it would be great if your plan can be shared with GIE/llm-d/AI GW/etc communities.

[LYZJU2019]

Sure. I plan to proceed with llm-d first with two stages:

Stage 1: Documentation

• Show the typical chain for LLM-d (auth → rate-limit → ext_proc → router).
• Emphasize ext_proc’s role: normalize request/response, set routing metadata, and ensure consistency before endpoint selection.
• Document which headers/metadata are expected downstream (e.g., x-ig-model, x-ig-criticality, dynamic metadata ig.model.id, ig.session.affinity_key).

Stage 2: Implement llm-d ext_proc

• Provide a minimal Go gRPC server using Envoy’s ext_proc API link
• Normalization request into a canonical schema (e.g., OpenAI chat style).
• Compute lightweight token estimates, set metadata (model.id, criticality, lora_ids, session.affinity_key, cost.estimated_tokens).
• Mirror key fields into x-ig-* headers for observability/debugging.

Let me know how this plan sounds!

[Xunzhuo]

I think our ext proc should be firstly integrated with an AI GW, not integrated with an existing extproc, I suggest having a try with envoy ai gw, then integrate with inference framework with ai gw

[rootfs]

@LYZJU2019 thanks! looking forward to your progress.

[rootfs]

@srampal @Xunzhuo @LYZJU2019 Thanks for contributing to this task. For clarity, I've created sub issues for each of these integration targets. If you are working on these gateway targets, please share your plans on the related sub issues. Thanks