Skip to content

Cloud Run /us/metadata is 3x slower than App Engine: GZipMiddleware compresses 70MB at level 9 in-request #3734

Description

@anth-volk

Finding (Stage 2 warm-parity gate failure)

Stage 2 of the Cloud Run cutover measures warm parity between platforms (capture_migration_baseline.py --repetitions 10, same commit, same hour, both warm). The gate (CR p95 ≤ GAE p95 × 1.20) failed at 3.25×, localized entirely to /us/metadata:

Probe (p95) Cloud Run App Engine
liveness 353 ms 673 ms
readiness 391 ms 620 ms
us_metadata 13,151 ms 4,332 ms

Trivial endpoints are faster on Cloud Run — the regression is metadata-specific.

Diagnosis

/us/metadata is ~70 MB raw (8.96 MB gzipped). Timing split via curl: CR TTFB 9.8 s / total 11.4 s vs GAE TTFB 2.1 s / total 3.2 s — the delta is server-side compute, not transfer. On App Engine flex, nginx gzips the response natively at a low compression level. On Cloud Run, starlette.middleware.gzip.GZipMiddleware (added in asgi_factory.py with default compresslevel=9, zlib's slowest/max setting) compresses the 70 MB inside the request, through the a2wsgi bridge. Evidence of the level difference: CR's compressed output is 8.96 MB vs GAE's 10.95 MB for the same payload — CR compresses harder, and pays ~8 s for it.

Fix

Set an explicit moderate compresslevel on the middleware (level 4: roughly 4× faster compression for ~10–15% larger output — ~0.2 s extra transfer vs ~6+ s saved compute). Expected to bring CR metadata TTFB to roughly GAE parity; verified empirically by re-running the Stage 2 Phase A parity capture after deploy.

Related but explicitly out of scope (per maintainer): the 70 MB payload itself (metadata slimming is planned separately), and the Accept-Encoding: identity → 500 on Cloud Run (~32 MB response cap; single known API consumer always sends gzip, and metadata slimming will moot it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions