atunnel: ingress CONNECT support by keithmattix · Pull Request #993 · agent-substrate/substrate

Keith Mattix II (keithmattix) · 2026-08-17T16:20:27Z

Finishes up the vision from #715 to have atunnel serve CONNECT on the ingress path. This will give us the option to hit actors on other ports besides 80. I haven't wired up atenet router yet because it's nontrivial; we should do that in a second step so we can have a baseline for performance

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Keith Mattix II (keithmattix) · 2026-08-18T17:20:40Z

 type conflictStore struct {
 	store.Interface
 	conflictTriggered atomic.Bool
+	shouldInject      func(worker *ateapipb.Worker) bool


This was added to try to reduce some flakiness with OnUpdate being written to/read across tests

Yuan Gao (ygao-g) · 2026-08-18T18:52:45Z

Heads up: --atunnel-connect-listen-address defaults to 0.0.0.0:444, which adds a third instance of #943. It is a declared containerPort that the router dials for CONNECT, so on an IPv6-only cluster nothing is listening on the worker's v6 PodIP — the same failure as 443. :444 binds all address families and still accepts IPv4.

It also collides textually with #978 and #1042: all three edit the same flag var blocks in cmd/ateom-gvisor/main.go and cmd/ateom-microvm/main.go, and the same arg list in workerpool_apply.go.

Keith Mattix II (keithmattix) · 2026-08-18T21:42:44Z

Yuan Gao (@ygao-g) I'm not sure I understand; if 0.0.0.0:443 binds dualstack by default, then an ipv6 cluster should still work with it right? The main difference is that #943 was on the egress path and required IP6T_SO_ORIGINAL_DST. The status quo should work with ipv6 only clusters

Yuan Gao (ygao-g) · 2026-08-19T02:58:01Z

You're right, and my heads-up was wrong.

net.Listen("tcp", "0.0.0.0:444") doesn't bind IPv4-only. Go treats an unspecified address as a wildcard on a listen, so favoriteAddrFamily picks AF_INET6 with IPV6_V6ONLY=0 and the socket is [::]:444, accepting both families — the router reaching the worker's v6 PodIP on 444 works as written. Your read of #943 is the right one: there the dual-stack bind is the bug rather than the fix, because the accepted v6 connection has no IPv4 SO_ORIGINAL_DST to recover. CONNECT takes its target from the request authority and never needs that lookup.

One thing still applies: #1042 changes atunnel-listen-address to :443 in the same var block in both main.gos, so whichever lands second gets a conflict. I'll take the rebase on my side.

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Bowei Du (bowei) · 2026-08-19T15:27:25Z

I see this was merged, but looking at some of the commits, they look like intermediary commits. In the future, it would be useful to squash this right before the merge to keep the commits meaningful for the overall history...

Benjamin Elder (BenTheElder) · 2026-08-19T15:28:01Z

Alternatively: We have squash merge enabled as the default, if the PR doesn't have clean commit history then you can just squash it on merge.

Eitan Yarmush (EItanya) · 2026-08-19T15:57:17Z

I see this was merged, but looking at some of the commits, they look like intermediary commits. In the future, it would be useful to squash this right before the merge to keep the commits meaningful for the overall history...

According to the commit history this was squashed and merged already?

https://github.com/agent-substrate/substrate/commits/main/
b9e42db

Bowei Du (bowei) · 2026-08-19T16:05:16Z

Ok, sorry, was looking at this PR itself. No worries. We should make the guidance more clear -- I've been looking at some of the other changes that have been going in.

Benjamin Elder (BenTheElder) · 2026-08-19T16:17:29Z

According to the commit history this was squashed and merged already?

Thanks Eitan Yarmush (@EItanya), and the recent the log looks clean to me ... even passed tests after merge this time 🤞

We should make the guidance more clear -- I've been looking at some of the other changes that have been going in.

Yeah, it should be relatively uncommon that we have a PR large enough to warrant multiple commits at this point, wondering if we even want any other mode ...

Lior Lieberman (LiorLieberman) · 2026-08-19T21:42:08Z

 	secondFileCounterDirectory := pflag.String("second-file-counter-directory", "", "Directory for a second file counter; empty disables it. Used to exercise an Actor with more than one durable volume")
 	validateExistingFilePath := pflag.String("validate-existing-file-path", "", "Path to existing file to validate reading")
 	extraPort := pflag.Int("extra-port", 0, "Additional port to listen on, for exercising atenet-router's arbitrary-port ingress support; 0 disables it")
+	tcpPort := pflag.Int("tcp-port", 0, "Plain TCP echo port for exercising atunnel CONNECT ingress; 0 disables it")


not sure I follow what is it for?

Oh sorry I was testing a branch for adding TCP support and this snuck into the final version. I can remove if we'd like, but it's probably nice to keep around

Lior Lieberman (LiorLieberman) · 2026-08-19T21:43:06Z

 const (
+	// DefaultConnectPort is the worker port on which atunnel accepts inbound
+	// mTLS CONNECT tunnels from the ingress router.
+	DefaultConnectPort = 444


nit: any reason 444 is used? e.g why not 8443? I understand this is from ingress --> atunnel but IIRC what what we used in #715 was 8443 for client --> ingress for CONNECT?

Is the plan to migrate this to be served on the same 443 long term or leave it as two distinct listeners?

The plan long term will be to migrate to a single port and only support connect, but AFAIU envoy will need some changes to make that happen.

its still a nit but why 444 and not same connect protocol as we are doing for client-->atunnel? just simpler to remember

Do we want to reuse a port from the lower 0-1023 range?

https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers#Well-known_ports

This is currently occupied by an actual protocol assignment.

Pagers on Kubernetes!

Seriously, good find though; I can change this to the router connect port

Lior Lieberman (LiorLieberman) · 2026-08-19T21:45:52Z

 	return err
 }

+// ServeConnectHTTP accepts a router-authenticated CONNECT request and relays


what does router-authenticated mean?

The router accepts the request and forwards it securely. We don't have authn/authz in the router today so maybe a tad misleading

Ah sorry, I missed this in a rename. There was a function called authorize() but all it did was reserve/allocate a worker. Must've added this comment before the rename

Lior Lieberman (LiorLieberman) · 2026-08-19T21:54:40Z

+
+	_, port, err := net.SplitHostPort(r.Host)
+	if err != nil {
+		http.Error(w, "CONNECT authority must include a port", http.StatusBadRequest)


so we are taking the port from authority, and we are not using any connect metadata?

Not sure what you mean by CONNECT metadata? This is a new TCP connection/HTTP request to atunnel on a worker, so we have no envoy metadata. We could add extra headers, but :authority is built to solve this problem, so I used that

yeah I meant CONNECT headers. the requests goes to one port on atunnel, (I assumed it is through authority port) but then needs to go to a different port on the actor (which is taken from the originally dialed dns)? how does this actor port persisted?

Might be helpful to walk through the full flow:

Request gets to atenet router

atenet router inspects the request to figure out what actor to use (currently via Host header) as well as what port to go to (also via host header)

atenet router sends CONNECT to atunnel with the port communicated via :authority

atunnel forwards traffic to the actor based on the port in that authority

The key thing about getting that port from the atenet router is that it gives us a future where, after we stop using Host header for actor identification, we can still know exactly what port on the actor we intend to talk to without having to parse the original request (assuming that's possible; it might be TCP). In that scenario (i.e. non-HTTP) CONNECT to atenet router tells us what the port should be and we propagate that to the CONNECT to atunnel.

/cc Eitan Yarmush (@EItanya)

Lior Lieberman (LiorLieberman) · 2026-08-19T21:57:28Z

        value:
          name: agentgateway
-          image: cr.agentgateway.dev/agentgateway:v1.4.1
+          image: cr.agentgateway.dev/agentgateway:v0.0.0-alpha.a6c0e366


Whats is the expectation here wrt to pre-released versions? (given that its in the main project). Ideally we avoid putting non stable versions here. We have been trying to avoid doing this with envoy.

Honestly I don't think that's a realistic expectation right now. This project is so unstable itself, and is moving so quickly that I think it's ok. Especially given that we have promised an immediate turnaround on all agentgateway fixes

I am open for thoughts. Also fine saying this is something that will be changed moving forward when substrate is more stable but we need to think about that.

We avoided doing it with envoy - we probably need to be opinionated whether we should embrace un-released versions of the proxies (and other dependencies) or no

I think it's fair game for everything; if we're going to iterate quickly before we're stable, prerelease seems expected IMO

Keith Mattix II (keithmattix) · 2026-08-19T22:32:46Z

Bowei Du (@bowei) - good call on the squashed commits. I typically try to be good about doing another push after and approval, but I'll be honest in saying that flaky CI made me hesitant to push again 😅 I'll do better about this

Benjamin Elder (BenTheElder) · 2026-08-20T04:52:25Z

The flakes are really bad, standing offer to prioritize reviewing flake fixes...

NekoPunch (orangeCatDeveloper) mentioned this pull request Aug 17, 2026

E2E test flake: TestWorkerPoolPodTemplateUpdate fails on optimistic-concurrency conflict #994

Closed

Keith Mattix II (keithmattix) force-pushed the atunnel-ingress-connect branch from f9e4121 to d216b54 Compare August 17, 2026 18:40

Keith Mattix II (keithmattix) added 5 commits August 17, 2026 22:41

atunnel: ingress CONNECT support

2151d1e

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Fix some flakes

b34e0cc

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

ALPN change was overbroad

73b8bd7

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Fix CONNECT test flake

c3fd5b5

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Remove unused helper

165454a

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Keith Mattix II (keithmattix) force-pushed the atunnel-ingress-connect branch from d85d3f0 to 165454a Compare August 18, 2026 03:49

Eitan Yarmush (EItanya) reviewed Aug 18, 2026

View reviewed changes

Comment thread cmd/atecontroller/internal/controllers/workerpool_apply.go

Fix context propagation and TCP half-close

e6bc515

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Eitan Yarmush (EItanya) reviewed Aug 18, 2026

View reviewed changes

Comment thread internal/atunnel/egress.go Outdated

Comment thread internal/atunnel/egress.go Outdated

Comment thread internal/atunnel/egress.go Outdated

Revert ingress changes

4ce7ac1

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Keith Mattix II (keithmattix) commented Aug 18, 2026

View reviewed changes

Keith Mattix II (keithmattix) requested a review from Eitan Yarmush (EItanya) August 18, 2026 17:37

Bump agentgateway

50535a7

Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>

Keith Mattix II (keithmattix) mentioned this pull request Aug 19, 2026

Multiple flaky networking tests #1050

Open

Lior Lieberman (LiorLieberman) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) August 19, 2026 03:53

Bowei Du (bowei) self-assigned this Aug 19, 2026

John Howard (howardjohn) approved these changes Aug 19, 2026

View reviewed changes

Eitan Yarmush (EItanya) approved these changes Aug 19, 2026

View reviewed changes

Eitan Yarmush (EItanya) merged commit b9e42db into agent-substrate:main Aug 19, 2026
9 checks passed

Lior Lieberman (LiorLieberman) reviewed Aug 19, 2026

View reviewed changes

Yuan Gao (ygao-g) mentioned this pull request Aug 19, 2026

ci: IPv6-only kind e2e plus the actor veth [DO NOT MERGE — signal only] #1059

Closed

Uh oh!

Conversation

Keith Mattix II (keithmattix) commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yuan Gao (ygao-g) commented Aug 18, 2026

Uh oh!

Keith Mattix II (keithmattix) commented Aug 18, 2026

Uh oh!

Yuan Gao (ygao-g) commented Aug 19, 2026

Uh oh!

Uh oh!

Bowei Du (bowei) commented Aug 19, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 19, 2026

Uh oh!

Eitan Yarmush (EItanya) commented Aug 19, 2026

Uh oh!

Bowei Du (bowei) commented Aug 19, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Keith Mattix II (keithmattix) commented Aug 19, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Keith Mattix II (keithmattix) commented Aug 17, 2026 •

edited

Loading

Eitan Yarmush (EItanya) Aug 19, 2026 •

edited

Loading

Lior Lieberman (LiorLieberman) Aug 19, 2026 •

edited

Loading