Skip to content

fix(sqs,lambda): mint queue URLs per caller so SQS clients in Lambda reach Overcast - #318

Merged
Neaox merged 2 commits into
mainfrom
claude/aws-endpoint-resolution-lambdas-71bc53
Jul 27, 2026
Merged

fix(sqs,lambda): mint queue URLs per caller so SQS clients in Lambda reach Overcast#318
Neaox merged 2 commits into
mainfrom
claude/aws-endpoint-resolution-lambdas-71bc53

Conversation

@Neaox

@Neaox Neaox commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Problem

An SQSClient inside a Lambda function could not reach the emulator: it resolved the endpoint from the QueueUrl and ignored AWS_ENDPOINT_URL entirely.

That is deliberate SDK behaviour. @aws-sdk/middleware-sdk-sqs's queueUrlMiddleware runs after endpoint resolution and swaps in the queue URL's origin:

if (!endpoint && input.QueueUrl && resolvedEndpoint && useQueueUrlAsEndpoint) {
  const queueUrlOrigin = new URL(new URL(input.QueueUrl).origin);
  if (resolvedEndpoint.url.origin !== queueUrlOrigin.origin) {
    context.endpointV2 = { ...resolvedEndpoint, url: queueUrlOrigin };
  }
}

The trap is !endpoint: that is the client config field, and AWS_ENDPOINT_URL never reaches it — it resolves through serviceConfiguredEndpoint → the ruleset's Endpoint parameter, so the field stays undefined and the queue URL wins. Only new SQSClient({ endpoint }) passed literally suppresses it. .NET and Java v1 use the queue URL as the request URI outright, so no client-side flag covers every runtime.

Combined with queueURL() minting from a single server-wide ExternalBaseURL(), a Lambda container — a sibling of Overcast, not a child — received http://localhost:4566/... and sent its SQS traffic to its own loopback.

Relationship to #319

#319 (ECS) landed first and extracted internal/containerendpoint, holding the container-side half of this fix. This PR is rebased onto it and uses that package rather than duplicating it: the SplitHorizonHosts config field, the docker.HostConfig.ExtraHosts field, and the URL-rewriting//etc/hosts helpers are all #319's now. What remains here is the half #319 could not do — internal/containerendpoint's own doc comment defers to it: "Minted at runtime by the container itself (CreateQueue, GetQueueUrl, ListQueues) — handled server-side, see internal/middleware."

What this PR adds

1. Per-request queue URLs (the server-side half). New middleware.ClientEndpoint stamps the origin the caller dialled, and SQS renders every wire-facing queue URL through it. X-Forwarded-Host aware. Requests arriving on a real AWS hostname fall through to the configured origin, so host-based addressing (#313) cannot mint a URL nobody on the machine can resolve. The stored form stays canonical, so persisted state does not inherit one caller's view.

2. Lambda wired onto internal/containerendpoint. ContainerRuntime now holds a *containerendpoint.Mapper, so function environment goes through RewriteURLs and containers are created with ExtraHosts() — the same treatment ECS tasks got in #319. The Runtime API address is already routable from Lambda containers, so it is passed to New directly rather than re-resolved.

Together these cover both ways a URL reaches a function: one it fetches itself now matches its own endpoint, and one baked in by a host-side deploy is rewritten or resolves via /etc/hosts.

Behaviour change

OVERCAST_HOSTNAME becomes the fallback for callers without a usable origin rather than a server-wide override. This is what makes the fix hold when it is set, and it fixes the mirror image of the same bug: the compose setup in docs/README.md recommends OVERCAST_HOSTNAME=overcast, which was handing host-side clients an overcast:4566 queue URL that does not resolve on the host.

Queue URLs minted on any origin are still accepted — only the queue name is read from them — so a queue created on one origin can be addressed from another. The CloudFormation integration test exercises exactly that.

Tests

Failing-first, then green:

  • tests/integration/sqs/endpoint_addressing_test.go — caller-origin minting across three origins, cross-origin GetQueueUrl/ListQueues, AWS-hostname fallback.
  • internal/services/lambda/container_endpoint_test.go — that the runtime actually routes env and container creation through the mapper. The mechanisms themselves are internal/containerendpoint's own tests, so these stay at the wiring level rather than duplicating them.

Two existing tests encoded the old server-wide semantics and were updated to the new contract, both keeping their original intent: TestCreateQueue_honorsHostname now pins fallback and caller-origin behaviour, and TestCreateStack_SQSQueueRefOutputIsUsableQueueURL compares the queue path rather than the exact origin — it still proves a stack-output URL from a different origin works.

Green locally on the rebased branch: full ./internal/..., full ./tests/..., make docs-check, and golangci-lint (v1.64.8, matching CI) over ./....

Drive-by

docs/services/lambda.md had duplicated frontmatter on main — a Markdown parser reads the first block, so the real description was shadowed by a degenerate description: "Lambda" and the second block rendered as body text. I scanned every doc; it is the only affected file, so a one-off rather than a tooling regression. Removed, since this PR already edits that file. Verified capgen --write-docs does not re-add it.

@Neaox
Neaox force-pushed the claude/aws-endpoint-resolution-lambdas-71bc53 branch 2 times, most recently from af4664a to 545e385 Compare July 27, 2026 19:56
…reach Overcast

AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client
configuration. @aws-sdk/middleware-sdk-sqs's queueUrlMiddleware replaces the
resolved endpoint with the queue URL's origin whenever the two differ and the
client was not constructed with an explicit `endpoint`; .NET and Java v1 use the
queue URL as the request URI outright, a leftover from the Query protocol.

AWS_ENDPOINT_URL does not suppress this: it resolves through the endpoint
ruleset's Endpoint parameter and never lands on config.endpoint, which is what
the middleware checks. A queue URL carrying localhost:4566 therefore sent a
Lambda function's SQS client to its own container, since Lambda containers are
siblings of Overcast rather than children of it.

Three mechanisms, one per way a URL reaches a caller:

- Queue URLs are minted per request on the origin the caller reached Overcast
  on (internal/middleware/clientendpoint.go). Requests arriving on a real AWS
  hostname fall back to the configured origin, so host-based addressing cannot
  mint a URL nobody can dial. The stored form stays canonical.
- Loopback origins on Overcast's port in Lambda function environment are
  re-pointed at the container-reachable endpoint at container start, covering
  URLs baked in by a host-side deploy (a CDK stack passing queue.queueUrl).
- Split-horizon hostnames — localhost.overcast.sh, localhost.localstack.cloud,
  localhost.floci.io, OVERCAST_HOSTNAME when it is a DNS name, and the new
  OVERCAST_SPLIT_HORIZON_HOSTS — are mapped to Overcast's address in each
  Lambda container's /etc/hosts, so one URL form works from both sides.

Behaviour change: OVERCAST_HOSTNAME is now the fallback for callers without a
usable origin rather than a server-wide override. This also fixes the mirror
image of the same bug, where the documented compose setup handed host-side
clients an `overcast:4566` queue URL that does not resolve on the host. Queue
URLs minted on any origin are still accepted, since only the queue name is read
from them.

ECS task containers have the same root cause and are not covered here; sharing
the helpers requires extracting them into their own package.
@Neaox
Neaox force-pushed the claude/aws-endpoint-resolution-lambdas-71bc53 branch 2 times, most recently from e4d36f6 to 848d781 Compare July 27, 2026 20:15
runtimeAPIContainerAddr kept its own copy of the container-vs-host address
logic, including a hostReachableIP that returned whatever non-loopback IPv4
address net.InterfaceAddrs listed first. That copy is the one internal/
containerendpoint was extracted from, and #319 fixed two defects in it that
this one still had.

net.InterfaceAddrs makes no ordering promise and a development machine is
routinely multi-homed. Measured on a Windows Docker Desktop host, three of
six non-loopback candidates were unreachable from a sibling container: two
169.254/16 addresses left by adapters that failed DHCP, and a Hyper-V
host-only switch. The routable address came back first only by luck. The
shared resolver prefers the interface carrying the default route and skips
link-local addresses, and its host.docker.internal last resort is paired
with a matching /etc/hosts entry so it resolves on native Linux rather than
only on the platform where the fallback is never reached.

ResolveHost returns the host without a port, because Lambda pairs it with
two: the Runtime API port and the emulator API port.
@Neaox
Neaox merged commit e2dd93c into main Jul 27, 2026
39 checks passed
@Neaox
Neaox deleted the claude/aws-endpoint-resolution-lambdas-71bc53 branch July 27, 2026 20:44
Neaox added a commit that referenced this pull request Jul 27, 2026
…rt (#331)

A container cannot see its own port mapping from the inside, and RewriteURLs
built its match origin from cfg.Port — the port Overcast listens on. Started
with remapped ports (docker run -p 4580:4566, which scripts/run-test-instance.sh
does by design), Overcast listens on 4566 while a host-side deploy mints queue
URLs on 4580. Those URLs were not recognised as Overcast's own, so a queue URL
baked into Lambda function or ECS task environment was passed through verbatim
and the container's SQS client dialled its own loopback on a port nothing was
listening on: connect ECONNREFUSED 127.0.0.1:4580.

The other half of #318 was unaffected throughout — a URL the function fetches
itself is minted per request on the origin the caller dialled, so it already
pointed at a reachable address in both configurations.

PublishedPort landed in #328 for the web UI; this reuses it. It is now resolved
once at startup and carried on config.Config, so the single Docker inspect
serves the web UI and both container-endpoint mappers instead of each doing its
own — and Lambda's runtime constructor does no blocking work, per AGENTS.md.

Verified end-to-end with a Lambda whose handler runs a real @aws-sdk/client-sqs
SendMessage, counting messages that actually arrive:

  remapped -p 4580:4566   before: 1/2 delivered (baked-in URL ECONNREFUSED)
                           after: 2/2
  1:1 -p 4590:4590         before: 2/2    after: 2/2 (no regression)
Neaox added a commit that referenced this pull request Jul 28, 2026
…oint

Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a
representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK
derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL
as the request URI. A queue URL minted on an origin the caller cannot dial
therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1
port mapping mask it further.

That is not something a coded test can carry, so it goes in a guide agents can
find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md.

It leads with the invariant everything else follows from -- inside a container
localhost is that container, never Overcast -- then the CLI caveat, the
deployment and client shapes worth covering, the split-horizon /etc/hosts
rewriting and why bare localhost cannot be hijacked, the reserved ports rule,
and the CDK specifics that caught #335 and #353.

Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for
path-style URLs was tried and reverted: with a split-horizon hostname the
configured name is better than the caller's raw address, because it resolves
from both sides of the container boundary.
Neaox added a commit that referenced this pull request Jul 28, 2026
…#353)

* fix(cloudformation): re-mint stack output URLs on the caller's origin

A stack output naming one of Overcast's own path-style URLs was returned with
the origin it was built from at provisioning time. Provisioning runs through
internal requests, so resource handlers have no caller to derive an origin
from and use cfg.ExternalBaseURL(); that value was then handed unchanged to
every later caller, including ones that reached Overcast somewhere else.

With the API port remapped -- docker run -p 4600:4566, which is what
scripts/run-test-instance.sh does -- a stack's queue-URL output named 4566
while the same server's GetQueueUrl returned 4600. The output was undialable:
AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client
configuration, so a JS SDK client reading the stack output got the queue URL's
origin substituted for its endpoint and failed against a closed port. The AWS
CLI hid it by honouring AWS_ENDPOINT_URL, and passing --endpoint-url hides it
too, so it only shows up with a default-configured SDK client.

#351 already re-mints host-routed outputs on the caller's origin. This extends
that to path-style URLs whose origin is recognisably Overcast's own -- the
configured external base, or loopback on its listen port -- and leaves
everything else alone, the same restraint the host-routed branch shows. ARNs,
ECR URIs and third-party endpoints are untouched.

Verified against a container published on 4600: the output moved from
localhost:4566 to localhost:4600, matching GetQueueUrl, and the host-side JS
SDK case that previously failed now succeeds.

docs/plans/cfn-output-url-origin.md records the investigation, including why
the AWS CLI cannot reproduce it.

* docs: manual testing guide, and why the AWS CLI cannot verify an endpoint

Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a
representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK
derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL
as the request URI. A queue URL minted on an origin the caller cannot dial
therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1
port mapping mask it further.

That is not something a coded test can carry, so it goes in a guide agents can
find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md.

It leads with the invariant everything else follows from -- inside a container
localhost is that container, never Overcast -- then the CLI caveat, the
deployment and client shapes worth covering, the split-horizon /etc/hosts
rewriting and why bare localhost cannot be hijacked, the reserved ports rule,
and the CDK specifics that caught #335 and #353.

Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for
path-style URLs was tried and reverted: with a split-horizon hostname the
configured name is better than the caller's raw address, because it resolves
from both sides of the container boundary.

* docs: drop the manual-testing pointer from tests/AGENTS.md

That file is about writing and maintaining coded tests; how to smoke test by
hand is a different concern and already reachable from the root AGENTS.md.

* docs: record that wildcard hostnames do not resolve inside containers

/etc/hosts is an exact-match table, so the split-horizon entries Overcast
injects cover only the apex names. A subdomain — virtual-hosted S3, an
execute-api invoke URL — misses the table and falls through to public DNS,
which answers 127.0.0.1: the container itself. The lookup succeeds, so it
presents as a connection failure rather than a resolution one.

Enumeration cannot fix it. The names are created after the container starts
and ExtraHosts is create-time only. Measured cost rules file size out as the
constraint (100 entries adds ~9us per lookup), so the failure is coverage,
not speed.

Records the mechanism that does work, verified with a throwaway resolver:
Docker's embedded DNS forwards unanswered names to --dns upstreams while
keeping 127.0.0.11 in resolv.conf, so container-name discovery is unaffected.

Not an alpha.26 regression; the server-side host routing already exists and
is simply unreachable from function code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant