Enable conformance tests for Inference Extension #4079
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: To support the full Gateway API Inference Extension, we need to be able to extract the model name from the client request body in certain situations. Solution: Add a basic NJS module to extract the model name. This module will be enhanced (I've added notes) to be included in the full solution. On its own, it is not yet used.
This commit adds support for the control plane to watch InferencePools. A feature flag has been added to enable/disable processing these resources. By default, it is disabled. When an HTTPRoute references an InferencePool, we will create a headless Service associated with that InferencePool, and reference it internally in the graph config for that Route. This allows us to use all of our existing logic to get the endpoints and build the proper nginx config for those endpoints. In a future commit, the nginx config will be updated to handle the proper load balancing for the AI workloads, but for now we just use our default methods by proxy_passing to the upstream.
Problem: In order for NGINX to get the endpoint of the AI workload from the EndpointPicker, it needs to send a gRPC request using the proper protobuf protocol. Solution: A simple Go server is injected as an additional container when the inference extension feature is enabled, that will listen for a request from our (upcoming) NJS module, and forward to the configured EPP to get a response in a header.
Problem: We need to connect NGINX to the Golang shim that talks to the EndpointPicker, and then pass client traffic to the proper inference workload. Solution: Write an NJS module that will query the local Go server to get the AI endpoint to route traffic to. Then redirect the original client request to an internal location that proxies the traffic to the chosen endpoint. The location building gets a bit complicated especially when using both HTTP matching conditions and inference workloads. It requires 2 layers of internal redirects. I added lots of comments to hopefully clear up how we build these locations to perform all the routing steps.
Update the inference extension design doc to specify different status that needs to be set on Inference Pools to understand its state
…4006) Update gateway inference extension proposal on inability to provide a secure TLS connection to EPP.
Add status to Inference Pools Problem: Users want to see the current status of their Inference pools Solution: Add status for inference pools
Proposed changes Problem: Want to collect number of referenced InferencePools in cluster. Solution: Collect the count of referenced InferencePools. Testing: Unit tests and manually verified collection via debug logs.
github-actions
bot
added
the
documentation
salonichf5
force-pushed
the
tests/conformance-inference-extension
branch
from
4f307c1 to
0bf576c
Compare
6 tasks
salonichf5
changed the title
salonichf5
force-pushed
the
tests/conformance-inference-extension
branch
from
cb8e184 to
5a30751
Compare
salonichf5
commented
Oct 15, 2025
salonichf5
commented
Oct 16, 2025
salonichf5
force-pushed
the
tests/conformance-inference-extension
branch
from
5a30751 to
ddb3ae3
Compare
ciarams87
requested changes
Oct 16, 2025
cmd/gateway/commands.go
Outdated
Comment on lines
779
to
784
| cmd.Flags().BoolVar( | ||
| &endpointPickerEnableTLS, | ||
| endpointPickerEnableTLSFlag, | ||
| false, | ||
| "Enable TLS for gRPC connections to the EndpointPicker extension.", | ||
| ) |
There was a problem hiding this comment.
We need to provide some mechanism to actually expose and then set these flags. As we deploy this as a sidecar in the NGINX pod, I think exposing these options in the NginxProxy makes most sense. This can then be propagated to the cli args in the pod spec in objects.go. We can do this in a separate PR, but at the very least for now we need to default this flag to true for the conformance tests
There was a problem hiding this comment.
Yeah currently the flags disable TLS and enable InsecureSkipVerify. Yeah that seems like a better idea let me do that
There was a problem hiding this comment.
so I have been trying to set the flags but its seeming a bit trickier than i initially thought.
As per our offline discussion I will
I will create a story to make these configurable in the future. But by default I will enable TLS and secure skip verify.
Also, I am squash these commits into one and make sue I co-author you so its easier and faster to rebase
github-project-automation
bot
moved this from 🆕 New
to 🏗 In Progress
in NGINX Gateway Fabric
ciarams87
force-pushed
the
feat/inference-extension
branch
from
52fb31b to
ba7bd6d
Compare
ciarams87
force-pushed
the
tests/conformance-inference-extension
branch
from
ddb3ae3 to
e13a1ec
Compare
salonichf5
force-pushed
the
tests/conformance-inference-extension
branch
from
e13a1ec to
2ccea14
Compare
github-actions
bot
added
dependencies
salonichf5
force-pushed
the
tests/conformance-inference-extension
branch
from
2ccea14 to
16cb93a
Compare
github-project-automation
bot
moved this from 🏗 In Progress
to ✅ Done
in NGINX Gateway Fabric
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Write a clear and concise description that helps reviewers understand the purpose and impact of your changes. Use the
following format:
Problem: Users want to enable conformance tests to ensure correct checks are in place
Solution: Enables conformance test
Testing: Manually being tested still.
Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.
Closes #3842
Checklist
Before creating a PR, run through this checklist and mark each as complete.
Release notes
If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.