fix(cloudformation): return stack output URLs on a reachable origin - #351
Merged
Conversation
H4's gate was to classify every AWS::URLSuffix use site in synthesised CDK
templates before touching the resolver. Done, and the result contradicts the
hazard the plan recorded: there are no IAM service-principal use sites.
Synthesised compat/suites/cdk (aws-cdk-lib 2.220.0, CDK CLI 2.1133.0) -- S3,
SQS, SNS, Lambda, IAM roles and managed policies, API Gateway REST,
EventBridge, Step Functions, nested stack -- and walked the template for every
{"Ref": "AWS::URLSuffix"} node. Exactly two, both URL hosts: the API Gateway
endpoint output and a nested stack's TemplateURL. Principals are emitted as
literal strings ("lambda.amazonaws.com", "states.amazonaws.com"), because CDK
v2 resolves them through its region-info database rather than joining over the
suffix.
The TemplateURL site is in fact already broken -- it resolves to
https://s3.us-east-1.amazonaws.com/..., which Overcast cannot fetch -- so
nested stacks need the substitution rather than merely tolerating it. And
EnforceIAM defaults to false, so a mis-substituted principal would be inert
even in the residual hand-written-template case.
Also realigns docs/networking.md, which #350 left half-updated: the
appsync-realtime-api and cloudfront labels were added to the new inventory
table but not to the "what works today" table, the precedence rule, or the
reserved-label note.
CDK composes an API Gateway invoke URL in the template itself:
["https://", {"Ref":"Api"}, ".execute-api.us-east-1.",
{"Ref":"AWS::URLSuffix"}, "/", {"Ref":"Stage"}, "/"]
The scheme is a literal and there is no port, so DescribeStacks handed back
https://abc.execute-api.us-east-1.amazonaws.com/prod/ -- a URL that resolves to
real AWS. AWS::URLSuffix cannot express a scheme or a port, so substituting it
can never produce a dialable URL, and a value containing ":4566" would make the
pseudo-parameter lie about what it is. It still resolves to amazonaws.com, as
it does on AWS.
The correction belongs at output emission, where Overcast has already assembled
the finished string and still holds the request context. Handler.reachableURL
parses with middleware.ParseHostRoute -- the same grammar that decides inbound
routing -- and re-mints through serviceutil.HostRoutedURLFromBase, the helper
every service handing back such a URL already uses. The grammar is stated once
and applied in both directions, so a rewritten output is by construction a URL
this router accepts.
Only a registered host-route label is claimed. ECR registry URIs, S3 URLs, ARNs
and plain strings pass through untouched, because Overcast does not serve those
hostnames and must not pretend to. Both the Query and typed (Smithy) paths share
one implementation.
Two corrections to the plan, both from evidence:
- The IAM service-principal hazard does not exist. Synthesising for
us-east-1, cn-north-1 and region-agnostic shows every URLSuffix use site is
a stack Output, and principals are emitted as literals in all three -- even
with an explicit region on the principal.
- The nested-stack TemplateURL was called a latent bug. It is not: both
fetchers dispatch u.Path internally and discard the host, so Overcast never
dials it.
H6. Every doc that states which hostnames or URL forms Overcast supports,
reviewed against what it now does.
Two were actively wrong rather than merely incomplete:
- migration-from-localstack.md said virtual-hosted style "requires DNS
resolution of *.localhost which doesn't work without extra configuration",
and told readers to force path-style. Both forms work, the bare one needs
no s3. prefix (unlike LocalStack), and OVERCAST_HOSTNAME makes it resolve
everywhere.
- sdk-cli.md linked to #s3-asset-upload-fails-on-windows-or-macos; the
heading is #s3-asset-upload-fails-on-windows, so the link was dead.
apigateway.md and appsync.md gained the endpoint-URL sections they never had:
the host-routed forms, what apiEndpoint and the uris/dns maps report and why
they differ, and why dns.REALTIME reports the appsync-api host. Both inserted
before the generated capability block, which is left untouched.
The reserved-label exception is now stated wherever bucket naming is
documented, not only in networking.md -- a user meets it at CreateBucket, not
while reading about DNS.
This was referenced Jul 28, 2026
Neaox
added a commit
that referenced
this pull request
Jul 28, 2026
…#353) * fix(cloudformation): re-mint stack output URLs on the caller's origin A stack output naming one of Overcast's own path-style URLs was returned with the origin it was built from at provisioning time. Provisioning runs through internal requests, so resource handlers have no caller to derive an origin from and use cfg.ExternalBaseURL(); that value was then handed unchanged to every later caller, including ones that reached Overcast somewhere else. With the API port remapped -- docker run -p 4600:4566, which is what scripts/run-test-instance.sh does -- a stack's queue-URL output named 4566 while the same server's GetQueueUrl returned 4600. The output was undialable: AWS SDKs resolve the SQS endpoint from the QueueUrl rather than from client configuration, so a JS SDK client reading the stack output got the queue URL's origin substituted for its endpoint and failed against a closed port. The AWS CLI hid it by honouring AWS_ENDPOINT_URL, and passing --endpoint-url hides it too, so it only shows up with a default-configured SDK client. #351 already re-mints host-routed outputs on the caller's origin. This extends that to path-style URLs whose origin is recognisably Overcast's own -- the configured external base, or loopback on its listen port -- and leaves everything else alone, the same restraint the host-routed branch shows. ARNs, ECR URIs and third-party endpoints are untouched. Verified against a container published on 4600: the output moved from localhost:4566 to localhost:4600, matching GetQueueUrl, and the host-side JS SDK case that previously failed now succeeds. docs/plans/cfn-output-url-origin.md records the investigation, including why the AWS CLI cannot reproduce it. * docs: manual testing guide, and why the AWS CLI cannot verify an endpoint Two endpoint bugs (#318, #353) hid behind the same thing: the AWS CLI is not a representative client. botocore honours AWS_ENDPOINT_URL, while the JS SDK derives the SQS endpoint from the QueueUrl and .NET/Java v1 use the queue URL as the request URI. A queue URL minted on an origin the caller cannot dial therefore passes the CLI and fails a real SDK. Passing --endpoint-url and a 1:1 port mapping mask it further. That is not something a coded test can carry, so it goes in a guide agents can find: docs/dev/manual-testing.md, linked from AGENTS.md and tests/AGENTS.md. It leads with the invariant everything else follows from -- inside a container localhost is that container, never Overcast -- then the CLI caveat, the deployment and client shapes worth covering, the split-horizon /etc/hosts rewriting and why bare localhost cannot be hijacked, the reserved ports rule, and the CDK specifics that caught #335 and #353. Also records in the handover doc that flipping OVERCAST_HOSTNAME precedence for path-style URLs was tried and reverted: with a split-horizon hostname the configured name is better than the caller's raw address, because it resolves from both sides of the container boundary. * docs: drop the manual-testing pointer from tests/AGENTS.md That file is about writing and maintaining coded tests; how to smoke test by hand is a different concern and already reachable from the root AGENTS.md. * docs: record that wildcard hostnames do not resolve inside containers /etc/hosts is an exact-match table, so the split-horizon entries Overcast injects cover only the apex names. A subdomain — virtual-hosted S3, an execute-api invoke URL — misses the table and falls through to public DNS, which answers 127.0.0.1: the container itself. The lookup succeeds, so it presents as a connection failure rather than a resolution one. Enumeration cannot fix it. The names are created after the container starts and ExtraHosts is create-time only. Measured cost rules file size out as the constraint (100 entries adds ~9us per lookup), so the failure is coverage, not speed. Records the mechanism that does work, verified with a throwaway resolver: Docker's embedded DNS forwards unanswered names to --dns upstreams while keeping 127.0.0.11 in resolv.conf, so container-name discovery is unaffected. Not an alpha.26 regression; the server-side host routing already exists and is simply unreachable from function code.
Neaox
added a commit
that referenced
this pull request
Jul 29, 2026
With OVERCAST_HOSTNAME set and the API port remapped (docker run -p 4652:4566), AppSync's uris, API Gateway v2's apiEndpoint, Lambda function URLs and the Cognito issuer all carried the listen port and were undialable from the host — while SQS queue URLs on the same instance were correct, because they mint per caller. The repo had four hand-kept base-URL precedences (serviceutil, SQS, CloudFormation, Cognito's typed path); two disagreed on the port, which was exactly the bug. One implementation now: serviceutil.ClientBaseURLFromOrigin — configured hostname (authoritative, #351), the caller's port (their request is the only proof of a dialable one; Overcast cannot see its own port mapping), and a TLS-aware scheme that upgrades and never downgrades. ClientBaseURL wraps it for request-shaped callers; CloudFormation's and Cognito's private copies become one-line delegations, so the typed CBOR path can no longer drift from the JSON path — it had, to the config port, making a caller's wire protocol change the issuer their token carried. The Cognito issuer following the caller's port is what OIDC Discovery 1.0 §4.3 requires: issuer MUST equal the URL the configuration was retrieved from, so the old config-port issuer failed spec-compliant validation for every remapped-port caller and pointed jwks_uri at a port they could not dial. Overcast's own validation is port-agnostic — the pool ID is read from the issuer path, never compared literally — and that accommodation is now guarded by comment and test so a tidy-up cannot silently revert it. Two deliberate divergences survive, each commented in place with its functional reason: SQS wire responses echo the caller's exact origin (SDKs dial the QueueUrl itself; substitution would amplify the OVERCAST_HOSTNAME=localhost misconfiguration into a container-breaking failure), and ECR's repositoryUri stays canonical (the docker daemon, not the API caller, dials it). containerendpoint correspondingly rewrites split-horizon hostnames carrying the published port to the listen port on the way into function/task environment and invoke payloads — the name resolves inside a container, but that port is bound only on the host. Patterns are precomputed in WithPublishedPort because the rewrite runs per invoke payload: measured 313 ns and zero allocations on a 1 KiB miss. URL minting itself measures 274 ns per minted URL, response-path only. Verified live on OVERCAST_HOSTNAME=localhost.overcast.sh with -p 4652:4566: every symptom-table URL carries :4652 and answers; discovery issuer equals the fetch origin and jwks_uri returns 200; the in-Lambda probe passes (bare virtual-hosted S3, SQS via env/discovery/payload, wildcard DNS); a :4652 URL baked into function env arrives inside as :4566 and works. Full go test ./... passes, golangci-lint clean, CDK lifecycle suite 35 passed / 0 failed. Design, per-service requirements and constraints, and the methodology for new services: docs/plans/client-facing-url-minting.md. User-facing summary: docs/networking.md. The one irreducible caveat — no single host:port is dialable from both sides of a port remap, so cross-boundary literal iss comparison under a remap fails under every possible policy — is documented in both, with the 1:1 mapping workaround.
Neaox
added a commit
that referenced
this pull request
Jul 29, 2026
* docs(plans): record that ClientBaseURL cannot serve both an issuer and a resource URL With OVERCAST_HOSTNAME set and the API port remapped, AppSync's uris, API Gateway v2's apiEndpoint and Lambda function URLs carry the listen port and are undialable from the host. SQS queue URLs are correct on the same instance, because they mint per caller. serviceutil.ClientBaseURL takes cfg.Port unconditionally once a hostname is set, and cfg.PublishedPort is never referenced in the package at all. Taking the caller's port instead — matching CloudFormation's own clientBaseURL and SQS — fixes those URLs and passes the whole suite except Cognito, whose OIDC issuer was deliberately changed in this same release to come from the configured origin rather than the caller's Host. A per-caller iss breaks token validation across the container boundary, which is the failure that fix removed. One helper, two opposite requirements. Attempted, measured, reverted rather than shipped: splitting the two notions of "the external base" is a design decision, not a release patch. Pre-existing — the responsible line is in v0.0.1-alpha.25, so not an alpha.26 regression, and the default configuration is unaffected. * fix(urls): mint every client-facing URL on the caller's port With OVERCAST_HOSTNAME set and the API port remapped (docker run -p 4652:4566), AppSync's uris, API Gateway v2's apiEndpoint, Lambda function URLs and the Cognito issuer all carried the listen port and were undialable from the host — while SQS queue URLs on the same instance were correct, because they mint per caller. The repo had four hand-kept base-URL precedences (serviceutil, SQS, CloudFormation, Cognito's typed path); two disagreed on the port, which was exactly the bug. One implementation now: serviceutil.ClientBaseURLFromOrigin — configured hostname (authoritative, #351), the caller's port (their request is the only proof of a dialable one; Overcast cannot see its own port mapping), and a TLS-aware scheme that upgrades and never downgrades. ClientBaseURL wraps it for request-shaped callers; CloudFormation's and Cognito's private copies become one-line delegations, so the typed CBOR path can no longer drift from the JSON path — it had, to the config port, making a caller's wire protocol change the issuer their token carried. The Cognito issuer following the caller's port is what OIDC Discovery 1.0 §4.3 requires: issuer MUST equal the URL the configuration was retrieved from, so the old config-port issuer failed spec-compliant validation for every remapped-port caller and pointed jwks_uri at a port they could not dial. Overcast's own validation is port-agnostic — the pool ID is read from the issuer path, never compared literally — and that accommodation is now guarded by comment and test so a tidy-up cannot silently revert it. Two deliberate divergences survive, each commented in place with its functional reason: SQS wire responses echo the caller's exact origin (SDKs dial the QueueUrl itself; substitution would amplify the OVERCAST_HOSTNAME=localhost misconfiguration into a container-breaking failure), and ECR's repositoryUri stays canonical (the docker daemon, not the API caller, dials it). containerendpoint correspondingly rewrites split-horizon hostnames carrying the published port to the listen port on the way into function/task environment and invoke payloads — the name resolves inside a container, but that port is bound only on the host. Patterns are precomputed in WithPublishedPort because the rewrite runs per invoke payload: measured 313 ns and zero allocations on a 1 KiB miss. URL minting itself measures 274 ns per minted URL, response-path only. Verified live on OVERCAST_HOSTNAME=localhost.overcast.sh with -p 4652:4566: every symptom-table URL carries :4652 and answers; discovery issuer equals the fetch origin and jwks_uri returns 200; the in-Lambda probe passes (bare virtual-hosted S3, SQS via env/discovery/payload, wildcard DNS); a :4652 URL baked into function env arrives inside as :4566 and works. Full go test ./... passes, golangci-lint clean, CDK lifecycle suite 35 passed / 0 failed. Design, per-service requirements and constraints, and the methodology for new services: docs/plans/client-facing-url-minting.md. User-facing summary: docs/networking.md. The one irreducible caveat — no single host:port is dialable from both sides of a port remap, so cross-boundary literal iss comparison under a remap fails under every possible policy — is documented in both, with the 1:1 mapping workaround. * docs: regenerate the docs search index for the networking.md addition make docs-check gates on the committed index matching the docs content; the new 'Which host and port a URL carries' section made it stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the host-addressing plan: H4 (CloudFormation stack output URLs) and H6 (documentation sweep). Follows #345, #350.
The bug
CDK composes an API Gateway invoke URL in the template itself:
So
DescribeStackshanded backhttps://abc.execute-api.us-east-1.amazonaws.com/prod/— a URL that resolves to real AWS.Why not substitute
AWS::URLSuffixThe plan originally proposed resolving
AWS::URLSuffixto the configured hostname. That cannot work: the scheme is a literal and there is no port, and the pseudo-parameter can express neither. A value containing:4566would also make it lie about what it is. It still resolves toamazonaws.com, faithful to AWS at the template level.The correction belongs at output emission, where Overcast has already assembled the finished string and still holds the request context:
ParseHostRoute(which decides inbound routing) andHostRoutedURLFromBase(the helper every service already mints through), so the grammar is stated once and applied in both directions.*http.Request.clientBaseURLmirrorsserviceutil.ClientBaseURL's precedence — a configuredOVERCAST_HOSTNAMEwins over the address the caller dialled, but the caller's scheme and port are kept. That lookup lives in the CloudFormation package becausemiddlewareimportsserviceutil, not the reverse.Two corrections to the plan, both from evidence
The IAM service-principal hazard does not exist. The plan warned that substituting the suffix would corrupt principals built as
Fn::Join ["", ["states.", {"Ref": "AWS::URLSuffix"}]]. Synthesisedcompat/suites/cdkplus a fixture covering threeServicePrincipalforms, ECR, S3 website and virtual-hosted URLs, CloudFront, Cognito hosted UI and API Gateway — forus-east-1,cn-north-1and region-agnostic:AWS::URLSuffixsitesAuditAwsOutputsAuditCnAuditAgnosticZero principal use sites in any partition, including with an explicit
regionon the principal. CDK v2 resolves them through its region-info database. S3 website, CloudFront and Cognito URLs never touch the suffix — they come fromFn::GetAtton real attributes.The nested-stack
TemplateURLwas never broken. I had called it a latent bug. Both fetchers —resolveTemplateBodyandnestedStackHandler.fetchTemplate— parse the URL and dispatchu.Pathinternally, discarding the host. Overcast never dials it, and the nested-stack tests pass.H6 — documentation sweep
Every doc stating which hostnames or URL forms Overcast supports, reviewed. Two were actively wrong:
migration-from-localstack.mdsaid virtual-hosted style "requires DNS resolution of*.localhostwhich doesn't work without extra configuration" and told readers to force path-style. Both forms work, the bare one needs nos3.prefix (unlike LocalStack), andOVERCAST_HOSTNAMEmakes it resolve everywhere.sdk-cli.mdlinked to#s3-asset-upload-fails-on-windows-or-macos; the heading is#s3-asset-upload-fails-on-windows, so the link was dead.apigateway.mdandappsync.mdgained the endpoint-URL sections they never had — the host-routed forms, whatapiEndpointand theuris/dnsmaps report and why they differ, and whydns.REALTIMEreports theappsync-apihost. Both inserted before the generated capability block, which is untouched.networking.mdis also realigned: #350 addedappsync-realtime-apiandcloudfrontto the inventory table but not to the "what works today" table, the precedence rule, or the reserved-label note.Verification
gofmt,go build -tags slim ./...,go vet -tags slim ./..., full./internal/...and./tests/integration/...,make lint-go(golangci-lint v2.8.0), anddocs-index --checkall clean onmain@76d2b5ba.Written test-first: the failing test asserted a dialable URL and untouched non-URL outputs before any implementation existed.