egress: add egress gateway support - atenet-egress by LiorLieberman · Pull Request #693 · agent-substrate/substrate

Lior Lieberman (LiorLieberman) · 2026-08-02T03:48:28Z

Added pluggable egress PEP support. (Feedback on atenet-egress name is welcomed - will open a separate PR to rename atenet-router to atenet-ingress )

#559 shipped the actor egress data path without any egress gateway. This adds an Envoy deployment that terminates actor CONNECTs. It requires downstream mTLS, so only a worker's atunnel can reach it.

The gateway consists of an Envoy and an atenet router --standalone ext_proc sidecar.

The same ext_proc binary now serves both directions. Direction is decided by
the accepting listener via the xds.listener_name CEL attribute, not by
anything in the request, so an ingress client cannot reach the egress handler
by crafting a CONNECT (and maybe thats not ideal? feedback is welcomed!).

handleEgressRequestHeaders checks the method,
validates the worker-asserted actor identity headers, authenticates them
against the control plane with GetActor, and rejects actors that are not
RUNNING or whose asserted version is stale.

With that change, atelet's Run/Restore requests grow two fields it now
populates on the way down to ateom:

egress_gateway_address, from the new cluster-wide
--egress-gateway-address flag, which the manifests point at
atenet-egress.ate-system.svc:443. This is the value that arms the
nftables REDIRECT; from here actor TCP egress leaves through the gateway
rather than the worker's masquerade.
actor_version, the Actor resource version ate-api observed when it
assigned the worker. atunnel asserts it to the gateway, and the gateway
fails closed when its own GetActor read is older, so a stale control-plane
view cannot authorize egress for an Actor whose assignment has since
moved. atelet rejects Run/Restore without it.

Tests pass
Appropriate changes to documentation are included in the PR

The gateway terminates actor's CONNECT request. It requires downstream mTLS, so only a worker's atunnel can reach it. The gateway consists of an Envoy and an `atenet router --standalone` ext_proc sidecar.

Lior Lieberman (LiorLieberman) · 2026-08-02T03:58:34Z

FYI Dmitry Berkovich (@dberkov)

demos/egress is a small Actor that fetches a URL it is given and echoes the upstream status and body back, which makes the egress path observable from outside the sandbox. hack/install-demo-egress.sh registers it as a --deploy-demo-egress fixture and hack/verify-egress-demo.sh drives it and checks the atenet-egress logs for the corresponding authorized CONNECT. TestActorEgress in the networking suite covers the same path automatically: it creates an Actor from the demo template, POSTs a fetch request through atenet-router, and asserts 200. The suite's actor helper is parameterised by template so the ingress test keeps using the counter fixture.

haiyanmeng · 2026-08-04T01:11:51Z

+      # Envoy calls it over localhost to authenticate actor identity against the
+      # ate API on every CONNECT. This mirrors the ingress gateway topology
+      # (Envoy + ext_proc in one pod); a shared/standalone ext_proc is a future step.
+      - name: ext-proc


Let us make ext-proc a native sidecar (initContainers + restartPolicy: Always).
Regular containers get SIGTERM together, ext_proc has no drain and will exit first,
and with failure_mode_allow: false every CONNECT arriving during Envoy's drain
window fails closed with a 503.

good point. created an issue for that - #760

I've actually opened a PR to handle the graceful termination of the atenet-router (#774, which should help mitigate the Envoy drain window issue mentioned here.

haiyanmeng · 2026-08-04T01:14:00Z

+      # Co-located ext_proc server (the atenet router, ext_proc-only). The egress
+      # Envoy calls it over localhost to authenticate actor identity against the
+      # ate API on every CONNECT. This mirrors the ingress gateway topology
+      # (Envoy + ext_proc in one pod); a shared/standalone ext_proc is a future step.


a shared/standalone ext_proc is a future step: can we first move the exe_proc logic into a separate binary?

ext_proc is already separate

I don't think so. See Line 268 in this file:

image: ko://github.com/agent-substrate/substrate/cmd/atenet

with what we have now - atenet == extproc server
atenet-egress deployment == envoy + atenet (handling egress releated extproc).

atenet-ingress deployment == envoy + atenet(handling ingress releated extproc).

Bowei Du (bowei) · 2026-08-04T21:07:17Z

implementation itself looks ok, but would like to fix a few organization things:

Make it so that egress, ingress for ext_proc can be enabled separately (add flags and gated deployment). We currently deploy both in the same deployment, but at scale, you would probably want to have separate ingress and egress deployments as they will be allocated resources very differently.
Put ingress,egress logic in .../atenet/router/{ingress,egress} packages.
router/ext_proc just has mux logic

probably something like:

switch {
case EgressEnabled && IsEgressRequest(...):
case IngressEnabled:
default:
   // send 4xx nothing is enabled.
}

Bowei Du (bowei)

You need to scrub through all of the comments -- there seems to some internal notes to yourself and your friend Claude.

Bowei Du (bowei) · 2026-08-04T16:51:33Z

 	localhostRegistryReplacement = pflag.String("localhost-registry-replacement", "", "The replacement registry endpoint for localhost and/or loopback IP addresses, useful for local development. for example kind-registry:5000")
 	imageCacheDir                = pflag.String("image-cache-dir", ateompath.ImageCacheDir, "Directory for the node-local OCI image layer cache. Must be on the volume shared with the ateom pods (the cached layers are their overlay lowerdirs), and on a disk sized for both capacity and IOPS: unpack throughput is gated by the volume's IOPS.")

+	// SEE(lior): both sides kept — main added --log-level here while this branch


What is this?

Bowei Du (bowei) · 2026-08-04T16:53:24Z

+	// When set, actors whose Run/Restore request does not already carry an egress
+	// gateway address have this address injected, causing ateom to redirect actor
+	// TCP egress through atunnel to the egress gateway. Empty keeps egress off.
+	egressGatewayAddress = pflag.String("egress-gateway-address", "", "Address (host:port) of the egress gateway. When set, actor TCP egress is transparently tunneled through atunnel to this gateway. Empty disables egress.")


Your comment and the help text are different.

As I understand it -- this sets a default egress if not specified on the Actor resource explicitly?

Can you fix it to match up?

not directly. We are using this flag because we have not yet designed the egress API properly and we havent wired it up yet.

When we do, we dont need any egressgateway flag

BTW Eitan took some of it to his identity PR so if that one ends up in first, I will remove it from here and rebase

Bowei Du (bowei) · 2026-08-04T20:43:53Z

 		case *extprocv3.ProcessingRequest_RequestHeaders:
 			start := time.Now()
-			hResponse, rqm, target, tmplNs, tmplName, resumeOutcome, err := s.handleRequestHeaders(stream.Context(), reqType.RequestHeaders)
+			// One ext_proc server handles both directions: actor egress


Let's factor this so you could run it in this way, but in a real deployment, I think you would keep them as separate deployments as they will likely have different scaling parameters.

Comment is confusing, they are currently separate deployments but egress and ingress is supported with the same atenet binary (different extproc handlers) in the same binary. I will reword the comment and will also make a mode to be able to launch the atenet in ingress only or egress only mode.

Bowei Du (bowei) · 2026-08-05T06:06:31Z

Needs rebase

Eitan Yarmush (EItanya)

Offline we discussed cleaning up the locations in which the egressgateway address needs to be set, right now it's set on quite a few pods, but we can definitely simplify.

Eitan Yarmush (EItanya) · 2026-08-06T16:08:47Z

+	// assigned, and atunnel documents it as a lower bound on trustworthy actor
+	// metadata. If our authoritative view is older than what the worker asserts,
+	// we cannot yet vouch for the identity, so reject rather than allow blindly.
+	if assertedVersion != "" {


Will not be necessary once my PR goes in, this version is removed

Eitan Yarmush (EItanya) · 2026-08-06T17:09:42Z

Here is the list of specific PEP todos as a result of #708

  - Require mTLS client certificates signed by the actor-identity CA.
  - Verify validity period, ClientAuth EKU, and IsCA == false.
  - Require exactly one valid ActorIdentity extension.
  - Require non-empty atespace, actor name, and actor UID.
  - Require Purpose == "atunnel"; reject generic, missing, or unknown purposes.
  - Authorize using the verified actor UID—not CONNECT headers or actor-provided metadata.

egress: add an Envoy egress gateway.

7a02250

The gateway terminates actor's CONNECT request. It requires downstream mTLS, so only a worker's atunnel can reach it. The gateway consists of an Envoy and an `atenet router --standalone` ext_proc sidecar.

Lior Lieberman (LiorLieberman) requested review from Bowei Du (bowei) and John Howard (howardjohn) August 2, 2026 03:48

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from 305b232 to 598a548 Compare August 3, 2026 19:02

haiyanmeng reviewed Aug 4, 2026

View reviewed changes

Bowei Du (bowei) reviewed Aug 4, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) mentioned this pull request Aug 5, 2026

Move atenet to a native sidecar container #760

Open

fixes

c133167

Eitan Yarmush (EItanya) reviewed Aug 6, 2026

View reviewed changes

Uh oh!

Conversation

Lior Lieberman (LiorLieberman) commented Aug 2, 2026

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrutiyam-glitch Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Aug 4, 2026

Uh oh!

Bowei Du (bowei) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Bowei Du (bowei) commented Aug 5, 2026

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shrutiyam-glitch Aug 6, 2026 •

edited

Loading

Lior Lieberman (LiorLieberman) Aug 5, 2026 •

edited

Loading