Skip to content

fix(cloudformation): return stack output URLs on a reachable origin - #351

Merged
Neaox merged 3 commits into
mainfrom
claude/cfn-urlsuffix-audit
Jul 28, 2026
Merged

fix(cloudformation): return stack output URLs on a reachable origin#351
Neaox merged 3 commits into
mainfrom
claude/cfn-urlsuffix-audit

Conversation

@Neaox

@Neaox Neaox commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Completes the host-addressing plan: H4 (CloudFormation stack output URLs) and H6 (documentation sweep). Follows #345, #350.

The bug

CDK composes an API Gateway invoke URL in the template itself:

["https://", {"Ref":"Api"}, ".execute-api.us-east-1.", {"Ref":"AWS::URLSuffix"}, "/", {"Ref":"Stage"}, "/"]

So DescribeStacks handed back https://abc.execute-api.us-east-1.amazonaws.com/prod/ — a URL that resolves to real AWS.

Why not substitute AWS::URLSuffix

The plan originally proposed resolving AWS::URLSuffix to the configured hostname. That cannot work: the scheme is a literal and there is no port, and the pseudo-parameter can express neither. A value containing :4566 would also make it lie about what it is. It still resolves to amazonaws.com, faithful to AWS at the template level.

The correction belongs at output emission, where Overcast has already assembled the finished string and still holds the request context:

m, ok := middleware.ParseHostRoute(u.Host)   // the grammar that ROUTES inbound
if !ok { return value }                       // not ours — untouched
serviceutil.HostRoutedURLFromBase(h.clientBaseURL(ctx), m.Label, m.ID, m.Region, u.Path)
  • DRY. No regex, no new knowledge of AWS hostname shapes. It reuses ParseHostRoute (which decides inbound routing) and HostRoutedURLFromBase (the helper every service already mints through), so the grammar is stated once and applied in both directions.
  • Scheme and port come free, from the caller's own origin.
  • Conservative. Only a registered host-route label is claimed, so ECR registry URIs, S3 URLs, ARNs and plain strings pass through untouched — Overcast does not serve those hostnames and must not claim to. Pinned by a test.
  • Context-based rather than request-based so the Query and typed (Smithy) paths share one implementation; the typed path has no *http.Request.

clientBaseURL mirrors serviceutil.ClientBaseURL's precedence — a configured OVERCAST_HOSTNAME wins over the address the caller dialled, but the caller's scheme and port are kept. That lookup lives in the CloudFormation package because middleware imports serviceutil, not the reverse.

Two corrections to the plan, both from evidence

The IAM service-principal hazard does not exist. The plan warned that substituting the suffix would corrupt principals built as Fn::Join ["", ["states.", {"Ref": "AWS::URLSuffix"}]]. Synthesised compat/suites/cdk plus a fixture covering three ServicePrincipal forms, ECR, S3 website and virtual-hosted URLs, CloudFront, Cognito hosted UI and API Gateway — for us-east-1, cn-north-1 and region-agnostic:

Stack AWS::URLSuffix sites Service principals
AuditAws 3 — all Outputs all literal
AuditCn 3 — same all literal
AuditAgnostic 3 — same all literal

Zero principal use sites in any partition, including with an explicit region on the principal. CDK v2 resolves them through its region-info database. S3 website, CloudFront and Cognito URLs never touch the suffix — they come from Fn::GetAtt on real attributes.

The nested-stack TemplateURL was never broken. I had called it a latent bug. Both fetchers — resolveTemplateBody and nestedStackHandler.fetchTemplate — parse the URL and dispatch u.Path internally, discarding the host. Overcast never dials it, and the nested-stack tests pass.

H6 — documentation sweep

Every doc stating which hostnames or URL forms Overcast supports, reviewed. Two were actively wrong:

  • migration-from-localstack.md said virtual-hosted style "requires DNS resolution of *.localhost which doesn't work without extra configuration" and told readers to force path-style. Both forms work, the bare one needs no s3. prefix (unlike LocalStack), and OVERCAST_HOSTNAME makes it resolve everywhere.
  • sdk-cli.md linked to #s3-asset-upload-fails-on-windows-or-macos; the heading is #s3-asset-upload-fails-on-windows, so the link was dead.

apigateway.md and appsync.md gained the endpoint-URL sections they never had — the host-routed forms, what apiEndpoint and the uris/dns maps report and why they differ, and why dns.REALTIME reports the appsync-api host. Both inserted before the generated capability block, which is untouched.

networking.md is also realigned: #350 added appsync-realtime-api and cloudfront to the inventory table but not to the "what works today" table, the precedence rule, or the reserved-label note.

Verification

gofmt, go build -tags slim ./..., go vet -tags slim ./..., full ./internal/... and ./tests/integration/..., make lint-go (golangci-lint v2.8.0), and docs-index --check all clean on main @ 76d2b5ba.

Written test-first: the failing test asserted a dialable URL and untouched non-URL outputs before any implementation existed.

Neaox added 3 commits July 28, 2026 23:11
H4's gate was to classify every AWS::URLSuffix use site in synthesised CDK
templates before touching the resolver. Done, and the result contradicts the
hazard the plan recorded: there are no IAM service-principal use sites.

Synthesised compat/suites/cdk (aws-cdk-lib 2.220.0, CDK CLI 2.1133.0) -- S3,
SQS, SNS, Lambda, IAM roles and managed policies, API Gateway REST,
EventBridge, Step Functions, nested stack -- and walked the template for every
{"Ref": "AWS::URLSuffix"} node. Exactly two, both URL hosts: the API Gateway
endpoint output and a nested stack's TemplateURL. Principals are emitted as
literal strings ("lambda.amazonaws.com", "states.amazonaws.com"), because CDK
v2 resolves them through its region-info database rather than joining over the
suffix.

The TemplateURL site is in fact already broken -- it resolves to
https://s3.us-east-1.amazonaws.com/..., which Overcast cannot fetch -- so
nested stacks need the substitution rather than merely tolerating it. And
EnforceIAM defaults to false, so a mis-substituted principal would be inert
even in the residual hand-written-template case.

Also realigns docs/networking.md, which #350 left half-updated: the
appsync-realtime-api and cloudfront labels were added to the new inventory
table but not to the "what works today" table, the precedence rule, or the
reserved-label note.
CDK composes an API Gateway invoke URL in the template itself:

  ["https://", {"Ref":"Api"}, ".execute-api.us-east-1.",
   {"Ref":"AWS::URLSuffix"}, "/", {"Ref":"Stage"}, "/"]

The scheme is a literal and there is no port, so DescribeStacks handed back
https://abc.execute-api.us-east-1.amazonaws.com/prod/ -- a URL that resolves to
real AWS. AWS::URLSuffix cannot express a scheme or a port, so substituting it
can never produce a dialable URL, and a value containing ":4566" would make the
pseudo-parameter lie about what it is. It still resolves to amazonaws.com, as
it does on AWS.

The correction belongs at output emission, where Overcast has already assembled
the finished string and still holds the request context. Handler.reachableURL
parses with middleware.ParseHostRoute -- the same grammar that decides inbound
routing -- and re-mints through serviceutil.HostRoutedURLFromBase, the helper
every service handing back such a URL already uses. The grammar is stated once
and applied in both directions, so a rewritten output is by construction a URL
this router accepts.

Only a registered host-route label is claimed. ECR registry URIs, S3 URLs, ARNs
and plain strings pass through untouched, because Overcast does not serve those
hostnames and must not pretend to. Both the Query and typed (Smithy) paths share
one implementation.

Two corrections to the plan, both from evidence:

  - The IAM service-principal hazard does not exist. Synthesising for
    us-east-1, cn-north-1 and region-agnostic shows every URLSuffix use site is
    a stack Output, and principals are emitted as literals in all three -- even
    with an explicit region on the principal.
  - The nested-stack TemplateURL was called a latent bug. It is not: both
    fetchers dispatch u.Path internally and discard the host, so Overcast never
    dials it.
H6. Every doc that states which hostnames or URL forms Overcast supports,
reviewed against what it now does.

Two were actively wrong rather than merely incomplete:

  - migration-from-localstack.md said virtual-hosted style "requires DNS
    resolution of *.localhost which doesn't work without extra configuration",
    and told readers to force path-style. Both forms work, the bare one needs
    no s3. prefix (unlike LocalStack), and OVERCAST_HOSTNAME makes it resolve
    everywhere.
  - sdk-cli.md linked to #s3-asset-upload-fails-on-windows-or-macos; the
    heading is #s3-asset-upload-fails-on-windows, so the link was dead.

apigateway.md and appsync.md gained the endpoint-URL sections they never had:
the host-routed forms, what apiEndpoint and the uris/dns maps report and why
they differ, and why dns.REALTIME reports the appsync-api host. Both inserted
before the generated capability block, which is left untouched.

The reserved-label exception is now stated wherever bucket naming is
documented, not only in networking.md -- a user meets it at CreateBucket, not
while reading about DNS.
@Neaox
Neaox merged commit 202b68f into main Jul 28, 2026
42 checks passed
@Neaox
Neaox deleted the claude/cfn-urlsuffix-audit branch July 28, 2026 11:24
Neaox added a commit that referenced this pull request Jul 28, 2026
…#353)

* fix(cloudformation): re-mint stack output URLs on the caller's origin

A stack output naming one of Overcast's own path-style URLs was returned with
the origin it was built from at provisioning time. Provisioning runs through
internal requests, so resource handlers have no caller to derive an origin
from and use cfg.ExternalBaseURL(); that value was then handed unchanged to
every later caller, including ones that reached Overcast somewhere else.

With the API port remapped -- docker run -p 4600:4566, which is what
scripts/run-test-instance.sh does -- a stack's queue-URL output named 4566
while the same server's GetQueueUrl returned 4600. The output was undialable:
AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client
configuration, so a JS SDK client reading the stack output got the queue URL's
origin substituted for its endpoint and failed against a closed port. The AWS
CLI hid it by honouring AWS_ENDPOINT_URL, and passing --endpoint-url hides it
too, so it only shows up with a default-configured SDK client.

#351 already re-mints host-routed outputs on the caller's origin. This extends
that to path-style URLs whose origin is recognisably Overcast's own -- the
configured external base, or loopback on its listen port -- and leaves
everything else alone, the same restraint the host-routed branch shows. ARNs,
ECR URIs and third-party endpoints are untouched.

Verified against a container published on 4600: the output moved from
localhost:4566 to localhost:4600, matching GetQueueUrl, and the host-side JS
SDK case that previously failed now succeeds.

docs/plans/cfn-output-url-origin.md records the investigation, including why
the AWS CLI cannot reproduce it.

* docs: manual testing guide, and why the AWS CLI cannot verify an endpoint

Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a
representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK
derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL
as the request URI. A queue URL minted on an origin the caller cannot dial
therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1
port mapping mask it further.

That is not something a coded test can carry, so it goes in a guide agents can
find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md.

It leads with the invariant everything else follows from -- inside a container
localhost is that container, never Overcast -- then the CLI caveat, the
deployment and client shapes worth covering, the split-horizon /etc/hosts
rewriting and why bare localhost cannot be hijacked, the reserved ports rule,
and the CDK specifics that caught #335 and #353.

Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for
path-style URLs was tried and reverted: with a split-horizon hostname the
configured name is better than the caller's raw address, because it resolves
from both sides of the container boundary.

* docs: drop the manual-testing pointer from tests/AGENTS.md

That file is about writing and maintaining coded tests; how to smoke test by
hand is a different concern and already reachable from the root AGENTS.md.

* docs: record that wildcard hostnames do not resolve inside containers

/etc/hosts is an exact-match table, so the split-horizon entries Overcast
injects cover only the apex names. A subdomain — virtual-hosted S3, an
execute-api invoke URL — misses the table and falls through to public DNS,
which answers 127.0.0.1: the container itself. The lookup succeeds, so it
presents as a connection failure rather than a resolution one.

Enumeration cannot fix it. The names are created after the container starts
and ExtraHosts is create-time only. Measured cost rules file size out as the
constraint (100 entries adds ~9us per lookup), so the failure is coverage,
not speed.

Records the mechanism that does work, verified with a throwaway resolver:
Docker's embedded DNS forwards unanswered names to --dns upstreams while
keeping 127.0.0.11 in resolv.conf, so container-name discovery is unaffected.

Not an alpha.26 regression; the server-side host routing already exists and
is simply unreachable from function code.
Neaox added a commit that referenced this pull request Jul 29, 2026
With OVERCAST_HOSTNAME set and the API port remapped (docker run -p 4652:4566),
AppSync's uris, API Gateway v2's apiEndpoint, Lambda function URLs and the
Cognito issuer all carried the listen port and were undialable from the host —
while SQS queue URLs on the same instance were correct, because they mint per
caller. The repo had four hand-kept base-URL precedences (serviceutil, SQS,
CloudFormation, Cognito's typed path); two disagreed on the port, which was
exactly the bug.

One implementation now: serviceutil.ClientBaseURLFromOrigin — configured
hostname (authoritative, #351), the caller's port (their request is the only
proof of a dialable one; Overcast cannot see its own port mapping), and a
TLS-aware scheme that upgrades and never downgrades. ClientBaseURL wraps it for
request-shaped callers; CloudFormation's and Cognito's private copies become
one-line delegations, so the typed CBOR path can no longer drift from the JSON
path — it had, to the config port, making a caller's wire protocol change the
issuer their token carried.

The Cognito issuer following the caller's port is what OIDC Discovery 1.0 §4.3
requires: issuer MUST equal the URL the configuration was retrieved from, so
the old config-port issuer failed spec-compliant validation for every
remapped-port caller and pointed jwks_uri at a port they could not dial.
Overcast's own validation is port-agnostic — the pool ID is read from the
issuer path, never compared literally — and that accommodation is now guarded
by comment and test so a tidy-up cannot silently revert it.

Two deliberate divergences survive, each commented in place with its functional
reason: SQS wire responses echo the caller's exact origin (SDKs dial the
QueueUrl itself; substitution would amplify the OVERCAST_HOSTNAME=localhost
misconfiguration into a container-breaking failure), and ECR's repositoryUri
stays canonical (the docker daemon, not the API caller, dials it).

containerendpoint correspondingly rewrites split-horizon hostnames carrying the
published port to the listen port on the way into function/task environment and
invoke payloads — the name resolves inside a container, but that port is bound
only on the host. Patterns are precomputed in WithPublishedPort because the
rewrite runs per invoke payload: measured 313 ns and zero allocations on a 1 KiB
miss. URL minting itself measures 274 ns per minted URL, response-path only.

Verified live on OVERCAST_HOSTNAME=localhost.overcast.sh with -p 4652:4566:
every symptom-table URL carries :4652 and answers; discovery issuer equals the
fetch origin and jwks_uri returns 200; the in-Lambda probe passes (bare
virtual-hosted S3, SQS via env/discovery/payload, wildcard DNS); a :4652 URL
baked into function env arrives inside as :4566 and works. Full go test ./...
passes, golangci-lint clean, CDK lifecycle suite 35 passed / 0 failed.

Design, per-service requirements and constraints, and the methodology for new
services: docs/plans/client-facing-url-minting.md. User-facing summary:
docs/networking.md. The one irreducible caveat — no single host:port is
dialable from both sides of a port remap, so cross-boundary literal iss
comparison under a remap fails under every possible policy — is documented in
both, with the 1:1 mapping workaround.
Neaox added a commit that referenced this pull request Jul 29, 2026
* docs(plans): record that ClientBaseURL cannot serve both an issuer and a resource URL

With OVERCAST_HOSTNAME set and the API port remapped, AppSync's uris, API
Gateway v2's apiEndpoint and Lambda function URLs carry the listen port and are
undialable from the host. SQS queue URLs are correct on the same instance,
because they mint per caller.

serviceutil.ClientBaseURL takes cfg.Port unconditionally once a hostname is set,
and cfg.PublishedPort is never referenced in the package at all.

Taking the caller's port instead — matching CloudFormation's own clientBaseURL
and SQS — fixes those URLs and passes the whole suite except Cognito, whose OIDC
issuer was deliberately changed in this same release to come from the configured
origin rather than the caller's Host. A per-caller iss breaks token validation
across the container boundary, which is the failure that fix removed.

One helper, two opposite requirements. Attempted, measured, reverted rather than
shipped: splitting the two notions of "the external base" is a design decision,
not a release patch.

Pre-existing — the responsible line is in v0.0.1-alpha.25, so not an alpha.26
regression, and the default configuration is unaffected.

* fix(urls): mint every client-facing URL on the caller's port

With OVERCAST_HOSTNAME set and the API port remapped (docker run -p 4652:4566),
AppSync's uris, API Gateway v2's apiEndpoint, Lambda function URLs and the
Cognito issuer all carried the listen port and were undialable from the host —
while SQS queue URLs on the same instance were correct, because they mint per
caller. The repo had four hand-kept base-URL precedences (serviceutil, SQS,
CloudFormation, Cognito's typed path); two disagreed on the port, which was
exactly the bug.

One implementation now: serviceutil.ClientBaseURLFromOrigin — configured
hostname (authoritative, #351), the caller's port (their request is the only
proof of a dialable one; Overcast cannot see its own port mapping), and a
TLS-aware scheme that upgrades and never downgrades. ClientBaseURL wraps it for
request-shaped callers; CloudFormation's and Cognito's private copies become
one-line delegations, so the typed CBOR path can no longer drift from the JSON
path — it had, to the config port, making a caller's wire protocol change the
issuer their token carried.

The Cognito issuer following the caller's port is what OIDC Discovery 1.0 §4.3
requires: issuer MUST equal the URL the configuration was retrieved from, so
the old config-port issuer failed spec-compliant validation for every
remapped-port caller and pointed jwks_uri at a port they could not dial.
Overcast's own validation is port-agnostic — the pool ID is read from the
issuer path, never compared literally — and that accommodation is now guarded
by comment and test so a tidy-up cannot silently revert it.

Two deliberate divergences survive, each commented in place with its functional
reason: SQS wire responses echo the caller's exact origin (SDKs dial the
QueueUrl itself; substitution would amplify the OVERCAST_HOSTNAME=localhost
misconfiguration into a container-breaking failure), and ECR's repositoryUri
stays canonical (the docker daemon, not the API caller, dials it).

containerendpoint correspondingly rewrites split-horizon hostnames carrying the
published port to the listen port on the way into function/task environment and
invoke payloads — the name resolves inside a container, but that port is bound
only on the host. Patterns are precomputed in WithPublishedPort because the
rewrite runs per invoke payload: measured 313 ns and zero allocations on a 1 KiB
miss. URL minting itself measures 274 ns per minted URL, response-path only.

Verified live on OVERCAST_HOSTNAME=localhost.overcast.sh with -p 4652:4566:
every symptom-table URL carries :4652 and answers; discovery issuer equals the
fetch origin and jwks_uri returns 200; the in-Lambda probe passes (bare
virtual-hosted S3, SQS via env/discovery/payload, wildcard DNS); a :4652 URL
baked into function env arrives inside as :4566 and works. Full go test ./...
passes, golangci-lint clean, CDK lifecycle suite 35 passed / 0 failed.

Design, per-service requirements and constraints, and the methodology for new
services: docs/plans/client-facing-url-minting.md. User-facing summary:
docs/networking.md. The one irreducible caveat — no single host:port is
dialable from both sides of a port remap, so cross-boundary literal iss
comparison under a remap fails under every possible policy — is documented in
both, with the 1:1 mapping workaround.

* docs: regenerate the docs search index for the networking.md addition

make docs-check gates on the committed index matching the docs content; the
new 'Which host and port a URL carries' section made it stale.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant