Feature: allow mixed IP and UDS endpoints in backend route references

[stekole]

What type of PR is this?
/kind feature

What this PR does / why we need it:
Backend resources with mixed IP and Unix Domain Socket (UDS) endpoints were incorrectly rejected as unsupported mixed address types. Additionally, UDS endpoints were blocked entirely in route backend references.

• Classifies IP + UDS combinations as IP (static) instead of MIXED
• Allows UDS endpoints in route Backend references
• Only rejects MIXED when FQDN is involved with IP or UDS

Which issue(s) this PR fixes:
Fixes #8229

Release Notes: Yes

[netlify]

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name	Link
🔨 Latest commit	de0232e
🔍 Latest deploy log	https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/69cc9d563c78bd0008d9a797
😎 Deploy Preview	https://deploy-preview-8530--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.38%. Comparing base (4dcb964) to head (de0232e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/gatewayapi/listener.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8530      +/-   ##
==========================================
- Coverage   74.41%   74.38%   -0.03%     
==========================================
  Files         243      243              
  Lines       38336    38333       -3     
==========================================
- Hits        28526    28514      -12     
- Misses       7814     7822       +8     
- Partials     1996     1997       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zirain]

IIRC, we're find to remove the last one parameter from function validateBackendRefBackend.

[stekole]

Makes sense - Thanks - Addressed.

[zirain]

is it possible to add an e2e test to ensure that a mixed backend worked as expected?

[stekole]

Yes - I can write something to validate the route is accepted (not rejected as "MIXED") and traffic flows through. If we want a full HTTP on UDS and serve traffic that may require additional containers and complexity.

I think ensuring the route is created is a good first step. Let me know if you'd like to see something else. We can discuss further or happy to address it in a follow-up.

[zirain]

/retest

[arkodg]

how would admins ensure proper access control if app developers can route back into the proxy ?

[arkodg]

cc @guydc

[stekole]

Good question. I may not have all the answers but I will references a few things I know.

rbac on Backend resources - the docs already reference restricting who can create Backend CRs, consistent with guidance on CVE-2021-25740. An app developer without RBAC to create Backend resources can't reference UDS: https://github.com/envoyproxy/gateway/blob/main/site/content/en/latest/tasks/traffic/backend.md#L14-L21

UDS also requires a mounted socket in the proxy pod - UDS path in a Backend spec is a no-op unless the socketfile is actually present in the proxy pods filesystem. Mounting it requires an envoyproxy infrastructure patch, which is an admin resource that app developers shouldnt control.
ref: https://github.com/envoyproxy/gateway/blob/main/site/content/en/contributions/design/backend.md#L129-L131
Task doc: https://github.com/envoyproxy/gateway/blob/main/site/content/en/latest/tasks/traffic/backend.md#L37

[stekole]

loopback routing also looks to be disabled - https://github.com/envoyproxy/gateway/blob/main/internal/gatewayapi/backend.go#L198

[stekole]

Checking-in if there are any outstanding concerns?

[zirain]

@guydc can you take a look?

[arkodg]

hey @stekole will bring this up in the community meeting tomorrow to help make a decision on this feature

[zhaohuabing]

Hi, I've changed the title to feature since it changed the route behavior to allow UDS. Calling this fix is misleading.

[zhaohuabing]

For reference, EG also rejects localhost for dynamic resolvers because of similar concerns.

gateway/internal/xds/translator/testdata/out/xds-ir/http-route-dynamic-resolver-with-host-rewriting.routes.yaml

Lines 34 to 119 in c7e21fa

	envoy.filters.http.rbac.dfp_loopback_deny:
	'@type': type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute
	rbac:
	matcher:
	matcherList:
	matchers:
	- onMatch:
	action:
	name: deny-loopback-host
	typedConfig:
	'@type': type.googleapis.com/envoy.config.rbac.v3.Action
	action: DENY
	name: DENY
	predicate:
	orMatcher:
	predicate:
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^127\.0\.0\.1(?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^localhost(?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^localhost\.localdomain(?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^ip6-localhost(?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^ip6-loopback(?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^\[::1\](?::\d+)?$
	- singlePredicate:
	input:
	name: http_header
	typedConfig:
	'@type': type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput
	headerName: x-dynamic-host-header-1
	valueMatch:
	safeRegex:
	googleRe2: {}
	regex: ^::1(?::\d+)?$

[guydc]

I don't think that there's a significant loopback risk via the UDS endpoint, unless users patch/mutate XDS:

• By default, EG does not configure envoy to listen on UDS addresses (e.g. for admin endpoint, probe endpoint, etc.). These endpoints are only available on TCP listeners.
• There is also no way for users to define UDS listeners, as that's not support by GW-API at this time.

I also don't think that we can create a reasonable deny-list of well-known privileged sockets that should not be targeted (e.g. /var/run/docker.sock), like we do for FQDN endpoints.

However, I do have a more practical concern: UDS service are often treated as-if they were "in-process", and often do not implement security measures like TLS and AuthN/Z. The common assumption is that this interface is purely local and has no network exposure. With this change, we will make it possible for app developers that have Backend RBAC to expose these UDS services to the internet.

In the EG context, ext-auth, ext-proc and SDS servers are often integrated using UDS. The exposure of these auxiliary services to direct end-user calls can create availability and confidentiality concerns, such as:

• Ext-Auth is DoSed via a route exposed by an App Developer that does not include circuit breakers, causing other traffic to be processed according to fail-open configuration (passthrough without security checks or deny all).
• End users can directly call the SDS server exposed by a route and access credentials such as TLS keys.

This change might surprise operators, that may not realize that such existing mounted sockets may now be exposed by app developers.

The backend resource was created, first and foremost, to support routing to cluster-external targets. It's not uncommon for operators to allow app developers to create Backend resources, while also using additional tools like network policies to restrict access to sensitive in-cluster resources. I prefer that this usage pattern would continue to be supported by default without the possible compromise of UDS services.

So, the way I see it, we can implement this feature, but we should create some security measures:

• Make this feature an opt-in feature, to avoid surprises for operators that follow current EG integration patterns.
• Create an allow-list of operator-defined "unprivileged" sockets that can be referenced by App Developers. Maybe users can access a static file server but not an SDS server.

[guydc]

@codex

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[stekole]

So, the way I see it, we can implement this feature, but we should create some security measures:

• Make this feature an opt-in feature, to avoid surprises for operators that follow current EG integration patterns.
• Create an allow-list of operator-defined "unprivileged" sockets that can be referenced by App Developers. Maybe users can access a static file server but not an SDS server.

Thanks @guydc for the analysis.
I agree this can be opt-in. I'll add a boolean and default to false for backwards compatibility.

Regarding the allow-list of socket paths, I've been thinking about this and I wonder if the complexity outweighs the benefit at this stage.

• There's no equivalent allow-list for IP or FQDN Backend endpoints today AFIAK. Adding one only for UDS could be more confusing to operators.
• RBAC already controls who can create Backend resources. Operators who are concerned about app developers exposing sensitive sockets can scope Backend permissions accordingly
• You made a good point earlier that a reasonable deny-list is hard to define. I think an allow-list has similar challenges
• socket paths can be dynamic depending on volume mounts and sidecar configurations, making a static list hard to maintain

The opt-in gate combined with clear documentation about the security implications gives operators the control they need without introducing API surface that I can see being hard to maintain long term.

Happy to continue the discussion and adjust as needed.

[guydc]

There's no equivalent allow-list for IP or FQDN Backend endpoints today AFIAK. Adding one only for UDS could be more confusing to operators.

Yes, but other measures exist, like network security policies, that can block TCP traffic from reaching unwanted sensitive resources like other microservices in the cluster that are not end-user facing.

In essence, I'd like to recreate that level of control also for the UDS case, albeit via EG.

You made a good point earlier that a reasonable deny-list is hard to define. I think an allow-list has similar challenges

IMHO, an operator that changed EnvoyProxy config to mount the sockets into pod is also well-positioned to decide which of these sockets should go into an allowlist maintained in EnvoyProxy.

cc @envoyproxy/gateway-reviewers @envoyproxy/gateway-maintainers please chime in

[arkodg]

There's no equivalent allow-list for IP or FQDN Backend endpoints today AFIAK. Adding one only for UDS could be more confusing to operators.

Yes, but other measures exist, like network security policies, that can block TCP traffic from reaching unwanted sensitive resources like other microservices in the cluster that are not end-user facing.

In essence, I'd like to recreate that level of control also for the UDS case, albeit via EG.

You made a good point earlier that a reasonable deny-list is hard to define. I think an allow-list has similar challenges

IMHO, an operator that changed EnvoyProxy config to mount the sockets into pod is also well-positioned to decide which of these sockets should go into an allowlist maintained in EnvoyProxy.

cc @envoyproxy/gateway-reviewers @envoyproxy/gateway-maintainers please chime in

sgtm, what shape are you thinking @guydc

[guydc]

sgtm, what shape are you thinking @guydc

probably something like

spec:
  allowUnixSocketReferences: bool
  allowedUnixSockets:
  - /path/to/socket

Alternatively, the allow list could be a list of named Backend resources. But, that could be a bit problematic:

• Do we want to allow to App Developers to create as many backends as they want referencing these sockets? Creates a bit of scale problem...
• Who ensures that the Backend is not admitted and later the App Developer changes the socket address to a forbidden path...

IMHO, an explicit list of paths is the best option, though it's not a very nice UX.