fix(sqs,lambda): mint queue URLs per caller so SQS clients in Lambda reach Overcast - #318
Merged
Merged
Conversation
Neaox
force-pushed
the
claude/aws-endpoint-resolution-lambdas-71bc53
branch
2 times, most recently
from
July 27, 2026 19:56
af4664a to
545e385
Compare
…reach Overcast AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client configuration. @aws-sdk/middleware-sdk-sqs's queueUrlMiddleware replaces the resolved endpoint with the queue URL's origin whenever the two differ and the client was not constructed with an explicit `endpoint`; .NET and Java v1 use the queue URL as the request URI outright, a leftover from the Query protocol. AWS_ENDPOINT_URL does not suppress this: it resolves through the endpoint ruleset's Endpoint parameter and never lands on config.endpoint, which is what the middleware checks. A queue URL carrying localhost:4566 therefore sent a Lambda function's SQS client to its own container, since Lambda containers are siblings of Overcast rather than children of it. Three mechanisms, one per way a URL reaches a caller: - Queue URLs are minted per request on the origin the caller reached Overcast on (internal/middleware/clientendpoint.go). Requests arriving on a real AWS hostname fall back to the configured origin, so host-based addressing cannot mint a URL nobody can dial. The stored form stays canonical. - Loopback origins on Overcast's port in Lambda function environment are re-pointed at the container-reachable endpoint at container start, covering URLs baked in by a host-side deploy (a CDK stack passing queue.queueUrl). - Split-horizon hostnames — localhost.overcast.sh, localhost.localstack.cloud, localhost.floci.io, OVERCAST_HOSTNAME when it is a DNS name, and the new OVERCAST_SPLIT_HORIZON_HOSTS — are mapped to Overcast's address in each Lambda container's /etc/hosts, so one URL form works from both sides. Behaviour change: OVERCAST_HOSTNAME is now the fallback for callers without a usable origin rather than a server-wide override. This also fixes the mirror image of the same bug, where the documented compose setup handed host-side clients an `overcast:4566` queue URL that does not resolve on the host. Queue URLs minted on any origin are still accepted, since only the queue name is read from them. ECS task containers have the same root cause and are not covered here; sharing the helpers requires extracting them into their own package.
Neaox
force-pushed
the
claude/aws-endpoint-resolution-lambdas-71bc53
branch
2 times, most recently
from
July 27, 2026 20:15
e4d36f6 to
848d781
Compare
runtimeAPIContainerAddr kept its own copy of the container-vs-host address logic, including a hostReachableIP that returned whatever non-loopback IPv4 address net.InterfaceAddrs listed first. That copy is the one internal/ containerendpoint was extracted from, and #319 fixed two defects in it that this one still had. net.InterfaceAddrs makes no ordering promise and a development machine is routinely multi-homed. Measured on a Windows Docker Desktop host, three of six non-loopback candidates were unreachable from a sibling container: two 169.254/16 addresses left by adapters that failed DHCP, and a Hyper-V host-only switch. The routable address came back first only by luck. The shared resolver prefers the interface carrying the default route and skips link-local addresses, and its host.docker.internal last resort is paired with a matching /etc/hosts entry so it resolves on native Linux rather than only on the platform where the fallback is never reached. ResolveHost returns the host without a port, because Lambda pairs it with two: the Runtime API port and the emulator API port.
Neaox
added a commit
that referenced
this pull request
Jul 27, 2026
…rt (#331) A container cannot see its own port mapping from the inside, and RewriteURLs built its match origin from cfg.Port — the port Overcast listens on. Started with remapped ports (docker run -p 4580:4566, which scripts/run-test-instance.sh does by design), Overcast listens on 4566 while a host-side deploy mints queue URLs on 4580. Those URLs were not recognised as Overcast's own, so a queue URL baked into Lambda function or ECS task environment was passed through verbatim and the container's SQS client dialled its own loopback on a port nothing was listening on: connect ECONNREFUSED 127.0.0.1:4580. The other half of #318 was unaffected throughout — a URL the function fetches itself is minted per request on the origin the caller dialled, so it already pointed at a reachable address in both configurations. PublishedPort landed in #328 for the web UI; this reuses it. It is now resolved once at startup and carried on config.Config, so the single Docker inspect serves the web UI and both container-endpoint mappers instead of each doing its own — and Lambda's runtime constructor does no blocking work, per AGENTS.md. Verified end-to-end with a Lambda whose handler runs a real @aws-sdk/client-sqs SendMessage, counting messages that actually arrive: remapped -p 4580:4566 before: 1/2 delivered (baked-in URL ECONNREFUSED) after: 2/2 1:1 -p 4590:4590 before: 2/2 after: 2/2 (no regression)
This was referenced Jul 27, 2026
Neaox
added a commit
that referenced
this pull request
Jul 28, 2026
…oint Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL as the request URI. A queue URL minted on an origin the caller cannot dial therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1 port mapping mask it further. That is not something a coded test can carry, so it goes in a guide agents can find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md. It leads with the invariant everything else follows from -- inside a container localhost is that container, never Overcast -- then the CLI caveat, the deployment and client shapes worth covering, the split-horizon /etc/hosts rewriting and why bare localhost cannot be hijacked, the reserved ports rule, and the CDK specifics that caught #335 and #353. Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for path-style URLs was tried and reverted: with a split-horizon hostname the configured name is better than the caller's raw address, because it resolves from both sides of the container boundary.
Neaox
added a commit
that referenced
this pull request
Jul 28, 2026
…#353) * fix(cloudformation): re-mint stack output URLs on the caller's origin A stack output naming one of Overcast's own path-style URLs was returned with the origin it was built from at provisioning time. Provisioning runs through internal requests, so resource handlers have no caller to derive an origin from and use cfg.ExternalBaseURL(); that value was then handed unchanged to every later caller, including ones that reached Overcast somewhere else. With the API port remapped -- docker run -p 4600:4566, which is what scripts/run-test-instance.sh does -- a stack's queue-URL output named 4566 while the same server's GetQueueUrl returned 4600. The output was undialable: AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client configuration, so a JS SDK client reading the stack output got the queue URL's origin substituted for its endpoint and failed against a closed port. The AWS CLI hid it by honouring AWS_ENDPOINT_URL, and passing --endpoint-url hides it too, so it only shows up with a default-configured SDK client. #351 already re-mints host-routed outputs on the caller's origin. This extends that to path-style URLs whose origin is recognisably Overcast's own -- the configured external base, or loopback on its listen port -- and leaves everything else alone, the same restraint the host-routed branch shows. ARNs, ECR URIs and third-party endpoints are untouched. Verified against a container published on 4600: the output moved from localhost:4566 to localhost:4600, matching GetQueueUrl, and the host-side JS SDK case that previously failed now succeeds. docs/plans/cfn-output-url-origin.md records the investigation, including why the AWS CLI cannot reproduce it. * docs: manual testing guide, and why the AWS CLI cannot verify an endpoint Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL as the request URI. A queue URL minted on an origin the caller cannot dial therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1 port mapping mask it further. That is not something a coded test can carry, so it goes in a guide agents can find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md. It leads with the invariant everything else follows from -- inside a container localhost is that container, never Overcast -- then the CLI caveat, the deployment and client shapes worth covering, the split-horizon /etc/hosts rewriting and why bare localhost cannot be hijacked, the reserved ports rule, and the CDK specifics that caught #335 and #353. Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for path-style URLs was tried and reverted: with a split-horizon hostname the configured name is better than the caller's raw address, because it resolves from both sides of the container boundary. * docs: drop the manual-testing pointer from tests/AGENTS.md That file is about writing and maintaining coded tests; how to smoke test by hand is a different concern and already reachable from the root AGENTS.md. * docs: record that wildcard hostnames do not resolve inside containers /etc/hosts is an exact-match table, so the split-horizon entries Overcast injects cover only the apex names. A subdomain — virtual-hosted S3, an execute-api invoke URL — misses the table and falls through to public DNS, which answers 127.0.0.1: the container itself. The lookup succeeds, so it presents as a connection failure rather than a resolution one. Enumeration cannot fix it. The names are created after the container starts and ExtraHosts is create-time only. Measured cost rules file size out as the constraint (100 entries adds ~9us per lookup), so the failure is coverage, not speed. Records the mechanism that does work, verified with a throwaway resolver: Docker's embedded DNS forwards unanswered names to --dns upstreams while keeping 127.0.0.11 in resolv.conf, so container-name discovery is unaffected. Not an alpha.26 regression; the server-side host routing already exists and is simply unreachable from function code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An
SQSClientinside a Lambda function could not reach the emulator: it resolved the endpoint from theQueueUrland ignoredAWS_ENDPOINT_URLentirely.That is deliberate SDK behaviour.
@aws-sdk/middleware-sdk-sqs'squeueUrlMiddlewareruns after endpoint resolution and swaps in the queue URL's origin:The trap is
!endpoint: that is the client config field, andAWS_ENDPOINT_URLnever reaches it — it resolves throughserviceConfiguredEndpoint→ the ruleset'sEndpointparameter, so the field staysundefinedand the queue URL wins. Onlynew SQSClient({ endpoint })passed literally suppresses it. .NET and Java v1 use the queue URL as the request URI outright, so no client-side flag covers every runtime.Combined with
queueURL()minting from a single server-wideExternalBaseURL(), a Lambda container — a sibling of Overcast, not a child — receivedhttp://localhost:4566/...and sent its SQS traffic to its own loopback.Relationship to #319
#319 (ECS) landed first and extracted
internal/containerendpoint, holding the container-side half of this fix. This PR is rebased onto it and uses that package rather than duplicating it: theSplitHorizonHostsconfig field, thedocker.HostConfig.ExtraHostsfield, and the URL-rewriting//etc/hostshelpers are all #319's now. What remains here is the half #319 could not do —internal/containerendpoint's own doc comment defers to it: "Minted at runtime by the container itself (CreateQueue, GetQueueUrl, ListQueues) — handled server-side, see internal/middleware."What this PR adds
1. Per-request queue URLs (the server-side half). New
middleware.ClientEndpointstamps the origin the caller dialled, and SQS renders every wire-facing queue URL through it.X-Forwarded-Hostaware. Requests arriving on a real AWS hostname fall through to the configured origin, so host-based addressing (#313) cannot mint a URL nobody on the machine can resolve. The stored form stays canonical, so persisted state does not inherit one caller's view.2. Lambda wired onto
internal/containerendpoint.ContainerRuntimenow holds a*containerendpoint.Mapper, so function environment goes throughRewriteURLsand containers are created withExtraHosts()— the same treatment ECS tasks got in #319. The Runtime API address is already routable from Lambda containers, so it is passed toNewdirectly rather than re-resolved.Together these cover both ways a URL reaches a function: one it fetches itself now matches its own endpoint, and one baked in by a host-side deploy is rewritten or resolves via
/etc/hosts.Behaviour change
OVERCAST_HOSTNAMEbecomes the fallback for callers without a usable origin rather than a server-wide override. This is what makes the fix hold when it is set, and it fixes the mirror image of the same bug: the compose setup indocs/README.mdrecommendsOVERCAST_HOSTNAME=overcast, which was handing host-side clients anovercast:4566queue URL that does not resolve on the host.Queue URLs minted on any origin are still accepted — only the queue name is read from them — so a queue created on one origin can be addressed from another. The CloudFormation integration test exercises exactly that.
Tests
Failing-first, then green:
tests/integration/sqs/endpoint_addressing_test.go— caller-origin minting across three origins, cross-originGetQueueUrl/ListQueues, AWS-hostname fallback.internal/services/lambda/container_endpoint_test.go— that the runtime actually routes env and container creation through the mapper. The mechanisms themselves areinternal/containerendpoint's own tests, so these stay at the wiring level rather than duplicating them.Two existing tests encoded the old server-wide semantics and were updated to the new contract, both keeping their original intent:
TestCreateQueue_honorsHostnamenow pins fallback and caller-origin behaviour, andTestCreateStack_SQSQueueRefOutputIsUsableQueueURLcompares the queue path rather than the exact origin — it still proves a stack-output URL from a different origin works.Green locally on the rebased branch: full
./internal/..., full./tests/...,make docs-check, andgolangci-lint(v1.64.8, matching CI) over./....Drive-by
docs/services/lambda.mdhad duplicated frontmatter onmain— a Markdown parser reads the first block, so the real description was shadowed by a degeneratedescription: "Lambda"and the second block rendered as body text. I scanned every doc; it is the only affected file, so a one-off rather than a tooling regression. Removed, since this PR already edits that file. Verifiedcapgen --write-docsdoes not re-add it.