RFC: Storage-backed image optimization

[ascorbic]

RFC: Storage-backed image optimization (Access-safe, per-service strategy)

Summary

EmDash already routes locally/R2-stored media through Astro's image service for responsive srcsets (#1438). That optimization is silently broken on Cloudflare: behind Cloudflare Access (or with global_fetch_strictly_public), Astro's image endpoint must fetch() the media source over HTTP, and that self-referential subrequest gets an Access login redirect instead of image bytes — surfacing as a 404 from /_image.

This RFC proposes fixing it by choosing the correct Astro image strategy per service type rather than applying one strategy everywhere. Astro image services are either local (sharp, Cloudflare's binding service — they load source bytes through an endpoint and transform server-side) or external (Netlify, Vercel — they emit a URL that the platform's edge optimizer fetches). For local services, EmDash wraps the image endpoint so source bytes come straight from the storage adapter, never over HTTP. For external services, EmDash does nothing — the edge optimizer already fetches the (public) media URL. The result is one Astro pipeline, no parallel transform route, no per-image component branching, and the Access 404 disappears on the one platform where it actually occurs.

This supersedes the binding-only approach in #1494, reusing its Cloudflare IMAGES-binding transformer while discarding its parallel route and component-level branching.

Example

Before (Image.astro / EmDashImage.astro, today): the component absolutizes the media URL, authorizes the origin via image.remotePatterns, and calls getImage. On Cloudflare behind Access this 404s; #1494 added a second code path (a parallel /_emdash/api/media/transform/{key} route plus component branching) to work around it.

After: the component always hands the relative internal media URL to <Image>/getImage — no absolutize, no branching:

---
import { getImage } from "astro:assets";
const optimized = await getImage({
  src: "/_emdash/api/media/file/01J....webp", // relative, same-origin
  width: renderWidth,
  height: renderHeight,
  widths: responsiveWidths(renderWidth),
});
---
<img src={optimized.src} srcset={optimized.srcSet.attribute} sizes={sizes} ... />

What happens to that request is decided once, at config time, by the configured image service:

Platform / service	Type	What EmDash does	Source bytes loaded by
sharp (Node default)	local	wrap image.endpoint	storage adapter (no HTTP)
Cloudflare cloudflare-binding	local	wrap image.endpoint	storage adapter → env.IMAGES
Netlify	external	nothing	Netlify edge fetches the URL
Vercel	external	nothing	Vercel edge fetches the URL

User-facing config is unchanged: the adapter's imageService, Astro's image block, and <Image> props own all behaviour. There is no new EmDash config field.

Background & Motivation

How Astro loads image source bytes

A local image service (sharp; Cloudflare's cloudflare-binding service) routes through an endpoint. baseService.getURL builds the endpoint URL from imageConfig.endpoint.route (astro/assets/services/service.ts), e.g. /_image?href=<src>&w=…. The endpoint then loads the source and runs the transform.

Astro's generic endpoint (astro/assets/endpoint/generic.ts) loads bytes like this:

const isRemoteImage = isRemotePath(transform.src);
inputBuffer = await loadImage(
  sourceUrl,
  isRemoteImage ? new Headers() : request.headers, // <- forwards cookies only for same-origin
  imageConfig, isRemoteImage,
);

For a relative (same-origin) src it forwards the request headers, including the Access cookie. For an absolute src it strips them.

EmDash's responsive path (media/responsive.ts) deliberately absolutizes the media URL (toAbsoluteMediaUrl) and authorizes the origin via image.remotePatterns, because that was the recipe for getting same-origin media optimized. On Cloudflare that absolutize step flips Astro into the no-cookie remote branch → the loopback subrequest carries no Access cookie → Access returns a login redirect → /_image 404s. The bug is applying the external strategy (absolutize + remotePatterns) to a local-service platform.

Note that even keeping the URL relative (so the cookie rides along) is not a robust fix: a Worker fetching its own origin is a subrequest blocked by global_fetch_strictly_public, and edge self-fetches do not reliably carry Access cookies. The durable answer is to not fetch over HTTP at all — read bytes from the storage adapter.

How the four target services actually behave

• sharp (Node default): local. Generic endpoint fetches src; sharp transforms. Works today via loopback (no Access on Node), but pays an HTTP round-trip.
• Cloudflare cloudflare-binding (adapter default): local. The workerd service's transform is a no-op passthrough; the real resize happens in the endpoint (image-transform-endpoint.ts calls env.IMAGES directly, loading local bytes via env.ASSETS.fetch). EmDash media is not in env.ASSETS, so the adapter's endpoint can't see it — and the absolutize fallback hits the Access wall. Broken today.
• Netlify: external (ExternalImageService). getURL → /.netlify/images?url=<src>; the Netlify edge fetches the URL. No endpoint involved.
• Vercel: external. getURL → /_vercel/image?url=<src>; the Vercel edge fetches the URL. No endpoint involved.

External optimizers fetch the source over HTTP from the deployment. With no Access in play (Access is Cloudflare-specific), that fetch of a public EmDash media URL succeeds. They cannot reach private media — and EmDash cannot help them, because there is no endpoint to hook.

Integration ordering

runHookConfigSetup does settings.config.integrations.unshift(settings.config.adapter) — the adapter's config:setup runs first, EmDash's runs after, and updateConfig deep-merges. EmDash can therefore read the resolved image.service.entrypoint and override image.endpoint deterministically.

Goals

• EmDash media renders an optimized, responsive srcset on sharp and Cloudflare cloudflare-binding, including when the site is behind Cloudflare Access or global_fetch_strictly_public.
• EmDash media continues to optimize on Netlify and Vercel for publicly served media, with no EmDash intervention.
• No source bytes are loaded over HTTP for local services — they come from the storage adapter.
• One image pipeline: <Image>/getImage for every image, with Astro generating the srcset. No parallel transform route, no per-image component branching.
• Zero new user-facing configuration; default-on.
• Non-EmDash images (bundled src/assets, allowed remote) behave exactly as the platform's stock endpoint produces them.

Non-Goals

• Optimizing private/auth-gated media on Netlify or Vercel. External optimizers must fetch the source over HTTP; EmDash has no endpoint to substitute storage reads. Such media falls back to an unoptimized <img> served from storage.
• Supporting the Cloudflare external image service (imageService: "cloudflare", /cdn-cgi/image). It is endpoint-less and edge-fetches the source, so it cannot reach private media; out of scope here.
• Arbitrary third-party global image services (e.g. a user setting image.service to Cloudinary). EmDash defers to them untouched (unoptimized <img> fallback for EmDash media). The supported way to use such a CDN is an EmDash media provider, which bypasses this path entirely.
• Provider-based external media (mediaProviders). It already resolves via getEmbed/getSrc and never touches astro:assets.

Prior Art

• feat: responsive srcset for media via astro:assets #1438 introduced responsive srcset generation for EmDash media via astro:assets. This RFC fixes the platform where that feature silently fails.
• feat(media): binding-based image transforms for same-origin media (Access-safe) #1494 ("binding-based image transforms for same-origin media") diagnosed the Access 404 and shipped a working Cloudflare fix: a parallel /_emdash/api/media/transform/{key} route reading bytes from storage and resizing with the IMAGES binding, selected by component-level branching. This RFC keeps feat(media): binding-based image transforms for same-origin media (Access-safe) #1494's IMAGES-binding transformer (packages/cloudflare/src/media/transform-runtime.ts) and its key-safety validation, but replaces the parallel route + branching with an image.endpoint wrapper so Astro's own pipeline drives everything.
• @astrojs/cloudflare image-transform-endpoint.ts is the model for the local Cloudflare endpoint: load bytes by binding (env.ASSETS.fetch there; the storage adapter here), transform with env.IMAGES, cache immutable. We mirror its structure and delegate non-EmDash hrefs back to it.

Detailed Design

1. Service-type classification

EmDash classifies the resolved image.service.entrypoint (read in config:setup, after the adapter) against a small known-entrypoint map:

entrypoint	class
astro/assets/services/sharp	local
astro/assets/services/noop (passthrough)	local
@astrojs/cloudflare/image-service-workerd	local
@astrojs/netlify/image-service	external
@astrojs/vercel/build-image-service (and the dev variant)	external
@astrojs/cloudflare/image-service (/cdn-cgi/image)	external (out of scope)
anything else	unknown

• local → install the EmDash endpoint wrapper (§2).
• external / unknown → do nothing; the component still emits the relative media URL, and the platform optimizer fetches it (external) or it falls back to an unoptimized <img> (unknown).

A static map (not runtime sniffing of an imported module) keeps config:setup cheap and the supported matrix explicit. New adapter entrypoints are a one-line map addition. The map errs safe: an unrecognized service is never wrapped.

2. The EmDash image endpoint

EmDash sets image.endpoint.entrypoint to a platform-specific module, leaving image.endpoint.route at the default (/_image). It does this only when (a) the service is local and (b) the current image.endpoint.entrypoint is a known stock endpoint (astro/assets/endpoint/generic, the Node endpoint, or @astrojs/cloudflare/image-transform-endpoint). If the user configured a custom endpoint, EmDash backs off and logs a warning — their pipeline is their responsibility (they may call an exported EmDash helper if they want storage-backed loading).

The endpoint module:

export const GET: APIRoute = async (ctx) => {
  const href = new URL(ctx.request.url).searchParams.get("href");
  const key = matchInternalMediaKey(href); // /_emdash/api/media/file/{key}, validated
  if (key) return transformFromStorage(ctx, key); // storage.download → transform
  return delegateToStockEndpoint(ctx);            // bundled / allowed-remote, unchanged
};

• matchInternalMediaKey strips INTERNAL_MEDIA_PREFIX and validates the key with the existing SAFE_STORAGE_KEY / isSafeTransformKey regex (no slashes, traversal, query/fragment characters).
• transformFromStorage calls emdash.storage.download(key) (the same call the /file/{key} route makes), checks the source is image/*, then runs the transform:
  • Cloudflare: env.IMAGES (reusing feat(media): binding-based image transforms for same-origin media (Access-safe) #1494's transform-runtime.ts). The binding name is the adapter's, exposed via globalThis.__ASTRO_IMAGES_BINDING_NAME (default IMAGES).
  • Node: getConfiguredImageService().transform(bytes, opts, imageConfig) (sharp, or whatever local service is configured).
  • If no transformer is available, stream the original bytes through with the correct content type — this still fixes the 404; it just isn't resized.
  • Response headers: Cache-Control: public, max-age=31536000, immutable, X-Content-Type-Options: nosniff.
• delegateToStockEndpoint forwards the whole request to the platform's stock endpoint GET (@astrojs/cloudflare/image-transform-endpoint on CF, astro/assets/endpoint/generic on Node), so bundled and allowed-remote images get byte-identical stock behaviour.

The endpoint entrypoint lives where its platform deps are importable: a Node variant in @emdash-cms/core, the Cloudflare variant in @emdash-cms/cloudflare (alongside the existing transformer). EmDash core references the Cloudflare entrypoint by string when it detects the Cloudflare adapter — the same indirection #1494 already uses for the transformer descriptor, so core stays portable.

3. Component unification

Image.astro and EmDashImage.astro stop branching on platform/service. For local/R2 media they emit the relative internal URL (/_emdash/api/media/file/{key}) and call <Image>/getImage with known width/height/widths. Astro's getSrcSet generates the candidates.

• local service: getURL → /_image?href=/_emdash/api/media/file/{key}&w=…; the EmDash endpoint loads bytes from storage.
• external service: getURL → /.netlify/images?url=/_emdash/api/media/file/{key}&w=… (or /_vercel/image?…); the edge fetches the relative same-origin URL.

Removed: toAbsoluteMediaUrl, buildResponsiveImage (the absolutize/remote-pattern dance), and #1494's buildTransformUrl / buildTransformSrcset / buildTransformedImage plus the parallel route. responsiveWidths / responsiveSizes are retained.

Because same-origin media is now passed relative, the image.remotePatterns entries EmDash injects for media (buildImageRemotePatterns) are no longer required for the media path. They are retained only if still needed for publicUrl-hosted (CDN-origin) media.

4. Provider and publicUrl media (unchanged)

• Media served via a configured provider (mediaProviders) is handled by the provider branch in the components and never reaches astro:assets.
• Media with a configured publicUrl (R2 custom domain, S3 CDN) resolves to a genuinely public external URL. On local services it falls through delegateToStockEndpoint (allowed-remote fetch); on external services the edge fetches it. Either way it is not an EmDash media key and is not storage-loaded.

Security Model

• Key validation / traversal: href is accepted only when it matches INTERNAL_MEDIA_PREFIX + a key passing SAFE_STORAGE_KEY (/^[A-Za-z0-9._-]+$/). No slashes, .., ?, #, or %, so the key cannot traverse or reroute on the storage backend. Identical to the existing /file/{key} and feat(media): binding-based image transforms for same-origin media (Access-safe) #1494 route guards.
• SSRF: local-service media URLs are relative and same-origin; the endpoint never fetches an attacker-influenced absolute URL for the EmDash branch (bytes come from storage.download). The delegate branch reuses the stock endpoint's existing remote-allow checks (isRemoteAllowed) unchanged.
• Content type: only image/* sources are transformed; anything else is refused (SVG/PDF/etc. are never fed to the binding). Responses carry X-Content-Type-Options: nosniff.
• No new authorization surface: media authorization is unchanged; this RFC changes only where source bytes come from, not who may read them.

Testing Strategy

• Unit — service-type classification map; matchInternalMediaKey accept/reject (traversal, query chars, bare id vs key); transform param validation; "no transformer → stream original" fallback.
• Integration (describeEachDialect) — endpoint resolves a known key to bytes via the storage adapter and returns a transformed/streamed image; non-EmDash hrefs delegate.
• e2e (Cloudflare) — EMDASH_E2E_TARGET=cloudflare against e2e/fixture-cloudflare: render a page with EmDash media and assert a real srcset (multiple distinct candidate URLs) and a 200 (not a login redirect) for a transform request. A regression guard for the original Access 404 — ideally exercised with a simulated gate or the global_fetch_strictly_public flag, since stock e2e cannot reproduce Access.
• Snapshot — pnpm query-counts to confirm no added per-render queries.

Drawbacks

• EmDash takes ownership of the global image.endpoint for local services. The delegate path keeps non-EmDash images byte-identical, but it is a larger footprint than feat(media): binding-based image transforms for same-origin media (Access-safe) #1494's opt-in side route. Mitigated by the known-stock-endpoint guard and the custom-endpoint back-off.
• The known-entrypoint map needs maintenance if an adapter renames its service entrypoint. Mitigated by erring safe (unknown → no wrap) and a test asserting the current entrypoints.
• Private media on Netlify/Vercel is not optimized. This is inherent to external optimizers and documented as a non-goal.

Alternatives

• feat(media): binding-based image transforms for same-origin media (Access-safe) #1494 as-is (parallel route + component branching). Works on Cloudflare but maintains a second image pipeline alongside Astro's, reimplements srcset/breakpoint/format logic, and special-cases components. Rejected in favour of folding back into Astro's single pipeline.
• Forward cookies by keeping the URL relative. Lets the Access cookie ride the loopback for sharp/Cloudflare-dev, but a Worker self-fetch is still blocked by global_fetch_strictly_public and edge self-fetches don't reliably carry the cookie. Fragile; rejected.
• Replace image.service entirely with an EmDash local service. Heavier; fights the adapters (which set the service) and changes the transform backend for all images. Wrapping only the endpoint is the minimal change that keeps the adapter's service authoritative.

Adoption Strategy

Additive and default-on. No migration: it makes the existing #1438 behaviour correct on Cloudflare and removes an HTTP round-trip on Node, while leaving Netlify/Vercel and non-EmDash images unchanged. An emdash({ images: false }) opt-out skips the endpoint wrap for anyone running their own image pipeline. EmDash is pre-1.0; this ships as a fix with a changeset describing the observable effect (EmDash media now produces a responsive srcset on Cloudflare behind Access).

Implementation Plan

1. Service classification + Node endpoint + component unification. Add the entrypoint map and config:setup wiring; ship the Node endpoint (storage + getConfiguredImageService().transform + delegate to generic); switch Image.astro/EmDashImage.astro to the uniform relative-URL path; remove toAbsoluteMediaUrl/buildResponsiveImage. Tests for the Node/local path.
2. Cloudflare endpoint. Ship the @emdash-cms/cloudflare endpoint reusing transform-runtime.ts/IMAGES; wire entrypoint selection on adapter detection; e2e on fixture-cloudflare.
3. Remove feat(media): binding-based image transforms for same-origin media (Access-safe) #1494's parallel surface. Delete the transform route and buildTransformUrl/buildTransformSrcset/buildTransformedImage; reconcile image.remotePatterns injection to the publicUrl-only case. Changeset.

Unresolved Questions

• Should EmDash assert/require cloudflare-binding when the Cloudflare adapter is detected with the external service, or just warn and fall back to unoptimized? (Out of scope to fix, but the diagnostic matters.)
• Exact entrypoint strings for the current Vercel/Netlify adapter versions to seed the map (build vs dev variants).
• Whether to keep image.remotePatterns injection at all once same-origin media is relative, or scope it strictly to publicUrl media.
• image.endpoint.route stays /_image (wrapping the stock route) vs a dedicated EmDash route — /_image is simplest and matches the adapter; confirm no conflict with the delegate.

Future Possibilities

• A documented hook so a custom local image service can opt into EmDash's storage-backed loading without EmDash owning image.endpoint.
• Extending the storage-backed endpoint to serve blurhash/dominant-color placeholders from stored metadata in the same request family.
• Signed, time-bound transform URLs if private-media optimization is ever wanted on external platforms (would require an EmDash-served, authorized proxy the external optimizer can reach).

[marcusbellamyshaw-cell]

This direction is great for us — we're hit by exactly the Access / global_fetch_strictly_public 404 from /_image that this RFC describes, on Cloudflare.

One question on the lookup path: does the wrapped (local-service) endpoint resolve a media key purely from storage.download(key), or does it first require a matching DB media record for that key?

Why it matters for us: in one of our plugins (a community photo-submission widget) we write uploads directly to the R2 bucket via the Worker binding and never create a DB media row — because ctx.media.upload() is currently unavailable to in-process plugins on the R2 binding (#1313). Those photos are served today via GET /_emdash/api/media/file/{key}, which reads R2 directly with no DB check. If the new optimized endpoint keys off a DB record rather than the storage adapter, optimization would silently skip exactly these images even though the raw /file/{key} path keeps working.

Two related notes if the storage-adapter path is the intent:

• It'd be ideal if SAFE_STORAGE_KEY accepts plain {ulid}.{ext} keys (e.g. 01JXYZ….webp) so binding-written keys pass validation.
• Whatever bucket/prefix the adapter's download(key) reads from needs to match what a raw binding put(key, …) writes to, otherwise optimized rendering would miss binding-written media while /file/{key} still resolves.

Happy to test against our R2 setup once there's a branch. Thanks for tackling this.