Two issues found running a Serverless (osls v4) app locally against Floci
Filing both together since they were found in the same session and share the same environment, happy to split into separate issues if preferred.
Shared environment
- Floci image:
floci/floci:1.5.34
- Docker: 29.6.2 (build dfc4efb), Docker Desktop for Mac
- Host: macOS (Darwin 25.5.0, arm64 / Apple Silicon)
- Docker daemon platform:
linux/arm64
- Lambda runtime under test:
public.ecr.aws/lambda/python:3.14 (managed runtime, not container image)
- Deploy tool:
osls (Serverless Framework fork) v4.0.0, via serverless-python-requirements
- API Gateway: REST API (
AWS::ApiGateway::RestApi, not HTTP API), invoked via /restapis/{id}/{stage}/_user_request_/...
- 4 distinct Lambda functions involved,
individually: false (single shared deployment package), no authorizer configured on any HTTP route (default NONE auth type)
Bug 1: API Gateway REST emulation rejects any request with an Authorization header, even on routes with no authorizer configured
Summary
Any HTTP request to a Floci-emulated API Gateway REST API endpoint that includes an Authorization header, regardless of its value/format, and regardless of whether the target endpoint has an authorizer configured at all, fails with:
{"message":"Invalid API id specified"}
HTTP status: 404.
The exact same request, with the Authorization header removed and every other header unchanged, succeeds normally (200, correct response body).
This strongly suggests Floci's API Gateway REST emulation unconditionally attempts to interpret any Authorization header as an AWS SigV4 signature (i.e., IAM-authenticated request routing/validation), fails to parse it as such, and returns a generic/misleading "Invalid API id specified" error instead of either (a) ignoring the header since the route's authorizer type is NONE, or (b) passing it through untouched to the Lambda proxy integration, which is what real AWS API Gateway does for NONE-auth routes (the Authorization header is just forwarded to the Lambda like any other header, with zero validation).
Steps to reproduce
- Deploy a Serverless app with a Lambda-proxy HTTP endpoint that has no authorizer configured (default), e.g.:
functions:
myFunc:
handler: main.my_handler
events:
- http:
path: api/v1/example
method: get
- Get Floci's REST API invoke URL for that stage:
http://localhost:4566/restapis/{api-id}/{stage}/_user_request_
- Call the endpoint without any
Authorization header:
curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:4566/restapis/{api-id}/{stage}/_user_request_/api/v1/example"
# -> 200
- Call the exact same endpoint, adding an arbitrary
Authorization header (any scheme, we tested Bearer <opaque-token>, but presumably any non-empty value triggers it):
curl -s "http://localhost:4566/restapis/{api-id}/{stage}/_user_request_/api/v1/example" \
-H "Authorization: Bearer some-arbitrary-token"
# -> 404 {"message":"Invalid API id specified"}
Actual output from our repro
$ curl -s -o /dev/null -w "no Authorization: %{http_code}\n" "$URL"
no Authorization: 200
$ curl -s -o /dev/null -w "with Authorization: %{http_code}\n" "$URL" -H "Authorization: Bearer <redacted>"
with Authorization: 404
$ curl -s "$URL" -H "Authorization: Bearer <redacted>"
{"message":"Invalid API id specified"}
Isolated further: adding only the Authorization header (no other custom headers) is sufficient to trigger the failure. Other custom headers (e.g. an arbitrary X-* header, a custom Accept value) do not trigger it, only Authorization does.
Also confirmed this happens hitting Floci directly, with no reverse proxy or dev-server proxy in between, it's not related to any client-side tooling, purely a Floci-side behavior.
Expected behavior
For a REST API route with no authorizer configured (NONE auth type, the default), Floci should forward the Authorization header to the backend Lambda unchanged, exactly like every other header, the same as real AWS API Gateway does. IAM/SigV4 validation of the Authorization header should only apply to routes explicitly configured with authorizer: aws_iam (or equivalent), not globally to every request that happens to carry that header name.
Impact
Any application that uses its own bearer-token/JWT scheme (Cognito, custom OAuth2, etc.) and sends it via a standard Authorization: Bearer <token> header, the overwhelmingly common convention, cannot exercise any authenticated request against a Floci-emulated REST API. Every such call fails at the API Gateway layer before it ever reaches the Lambda, with a confusing, unrelated-looking error message that gives no hint the Authorization header is the cause.
For an SPA hitting several authenticated endpoints on page load, this manifests as the entire page failing to load any data, with only generic 404s in the network tab and no clear error pointing back to the header.
Workaround we're using
Stripping the Authorization header specifically at our local dev proxy layer before it reaches Floci, and relying on our backend's own dev-only fallback for user attribution in logs, since the JWT/bearer token itself is only needed for real deployments (not for local Floci testing). This is a workaround at the client/proxy layer, not a fix, it means we can't actually exercise our real auth-header code path against Floci locally.
Questions for maintainers
- Is the API Gateway REST emulation intentionally treating any
Authorization header as a SigV4 signature attempt, regardless of the route's configured authorizer type?
- Is there a way to configure Floci (env var, per-route setting) to skip this check for
NONE-auth routes, or to disable IAM/SigV4 header interception globally for local dev use cases where it's not needed?
Bug 2: Concurrent Lambda cold starts across different functions serialize/bottleneck (one call took 31s vs <1s warm)
Summary
When multiple different Lambda functions are invoked concurrently for the first time after a fresh Floci start (all cold, no warm containers yet), overall latency degrades far more than what per-function cold-start overhead alone would suggest. One function that normally responds in <1s took 31.7s when invoked concurrently alongside 5 other cold functions. The same 6 concurrent calls, once all functions are warm, complete in well under 1s total.
This looks like invocation/container-provisioning across different functions is being serialized (or heavily contended) rather than handled in parallel, which is surprising since these are unrelated functions (different code, different containers), not concurrent invocations of the same function (where serialization due to a single execution environment would be expected/correct).
Steps to reproduce
- Start Floci fresh (no prior state):
docker compose down -v --remove-orphans # or: floci reset / equivalent
docker compose up -d floci
- Deploy a serverless app with several distinct HTTP-triggered Lambda functions, all pointing at the same Floci instance/API Gateway emulation. In our repro we had 4 functions behind an API Gateway REST API emulation, one of them (call it
funcB) noticeably larger/slower to cold-start than the others.
- Immediately after deploy (before any invocation has happened, so every function is genuinely cold), fire all endpoints concurrently from a single shell, e.g.:
time (
curl -s -o /dev/null -w "funcA: %{http_code} (%{time_total}s)\n" "$BASE/funcA" &
curl -s -o /dev/null -w "funcB: %{http_code} (%{time_total}s)\n" "$BASE/funcB" &
curl -s -o /dev/null -w "funcC: %{http_code} (%{time_total}s)\n" "$BASE/funcC" &
curl -s -o /dev/null -w "funcD: %{http_code} (%{time_total}s)\n" "$BASE/funcD" &
curl -s -o /dev/null -w "funcE: %{http_code} (%{time_total}s)\n" "$BASE/funcE" &
curl -s -o /dev/null -w "funcF: %{http_code} (%{time_total}s)\n" "$BASE/funcF" &
wait
)
- Compare against the same 6 calls fired again immediately after (now warm).
Actual output from our repro (function names anonymized, real timings)
6 concurrent calls, but only 4 distinct Lambda functions (funcB is hit 3x with different query params, same underlying function/container):
Cold (fresh deploy, first invocation of every function, all fired concurrently):
funcA: 200 (7.939875s)
funcB(1): 200 (14.078408s)
funcB(2): 200 (14.090147s)
funcB(3): 200 (14.091175s)
funcC: 200 (14.095592s)
funcD: 200 (31.689977s)
Wall-clock (time) for the whole batch: ~31.7s.
Warm (same 6 calls, fired again right after):
funcB(3): 200 (0.039291s)
funcB(1): 200 (0.040236s)
funcC: 200 (0.043519s)
funcB(2): 200 (0.044213s)
funcA: 200 (0.389826s)
funcD: 200 (0.887255s)
Wall-clock for the whole batch: ~0.9s.
Expected behavior
Cold-starting 4 independent, unrelated Lambda functions concurrently should scale roughly with the slowest individual cold start (each function has its own container, its own code volume, no shared state), i.e., total wall-clock should be close to max(individual cold start times), not sum-ish/serialized. If a single function's cold start (image pull + code volume population + container start) normally takes e.g. 5-14s in isolation, 4 of them in parallel should still finish in roughly that same ballpark, not 31s+.
Impact
Real-world SPA dashboards commonly fire several API calls in parallel on page load (Promise.all / similar) hitting different backend Lambda-backed endpoints. Right after a fresh local environment reset, this pattern reliably triggers the worst case above, a dashboard with a client-side retry budget of ~12-16s (a few retries a few seconds apart) can time out and show a "server not responding" / stuck-loading state on the very first load, even though every individual endpoint is perfectly healthy and fast once warm.
Workaround we're using
Sequentially (not concurrently) invoking every distinct Lambda function once right after deploy finishes, before opening the frontend. This "pre-warms" all functions so the first real concurrent burst from the browser hits already-warm containers. Works, but obviously isn't a real fix, and the same slowdown would presumably resurface after Floci's warm-pool idle eviction (we saw a Warm pool idle eviction enabled: timeout=300s log line) if the app sits idle for 5+ minutes and then gets hit with a concurrent burst again.
Possible root cause (speculation, untested against Floci's source)
Something in the code path that provisions a new Lambda execution container (Docker image pull check / code-volume population / container create+start) may be behind a global lock, single-threaded queue, or a semaphore with concurrency=1, causing concurrent cold-start requests for different functions to queue up rather than run in parallel. Given the per-call timings above roughly step up (~8s, ~14s, ~14s, ~14s, ~14s, ~32s) rather than clustering near a single max, this looks consistent with requests being serviced roughly one-at-a-time with some overlap, not fully parallel provisioning.
Questions for maintainers
- Is Lambda container provisioning (across different functions) intentionally serialized anywhere (e.g., a shared lock around Docker API calls, image-cache checks, or code-volume population)?
- Is there a concurrency/parallelism setting for cold-start provisioning that we could tune, or is this a known limitation?
Two issues found running a Serverless (osls v4) app locally against Floci
Filing both together since they were found in the same session and share the same environment, happy to split into separate issues if preferred.
Shared environment
floci/floci:1.5.34linux/arm64public.ecr.aws/lambda/python:3.14(managed runtime, not container image)osls(Serverless Framework fork) v4.0.0, viaserverless-python-requirementsAWS::ApiGateway::RestApi, not HTTP API), invoked via/restapis/{id}/{stage}/_user_request_/...individually: false(single shared deployment package), noauthorizerconfigured on any HTTP route (defaultNONEauth type)Bug 1: API Gateway REST emulation rejects any request with an
Authorizationheader, even on routes with no authorizer configuredSummary
Any HTTP request to a Floci-emulated API Gateway REST API endpoint that includes an
Authorizationheader, regardless of its value/format, and regardless of whether the target endpoint has an authorizer configured at all, fails with:{"message":"Invalid API id specified"}HTTP status:
404.The exact same request, with the
Authorizationheader removed and every other header unchanged, succeeds normally (200, correct response body).This strongly suggests Floci's API Gateway REST emulation unconditionally attempts to interpret any
Authorizationheader as an AWS SigV4 signature (i.e., IAM-authenticated request routing/validation), fails to parse it as such, and returns a generic/misleading "Invalid API id specified" error instead of either (a) ignoring the header since the route's authorizer type isNONE, or (b) passing it through untouched to the Lambda proxy integration, which is what real AWS API Gateway does forNONE-auth routes (theAuthorizationheader is just forwarded to the Lambda like any other header, with zero validation).Steps to reproduce
Authorizationheader:Authorizationheader (any scheme, we testedBearer <opaque-token>, but presumably any non-empty value triggers it):Actual output from our repro
Isolated further: adding only the
Authorizationheader (no other custom headers) is sufficient to trigger the failure. Other custom headers (e.g. an arbitraryX-*header, a customAcceptvalue) do not trigger it, onlyAuthorizationdoes.Also confirmed this happens hitting Floci directly, with no reverse proxy or dev-server proxy in between, it's not related to any client-side tooling, purely a Floci-side behavior.
Expected behavior
For a REST API route with no authorizer configured (
NONEauth type, the default), Floci should forward theAuthorizationheader to the backend Lambda unchanged, exactly like every other header, the same as real AWS API Gateway does. IAM/SigV4 validation of theAuthorizationheader should only apply to routes explicitly configured withauthorizer: aws_iam(or equivalent), not globally to every request that happens to carry that header name.Impact
Any application that uses its own bearer-token/JWT scheme (Cognito, custom OAuth2, etc.) and sends it via a standard
Authorization: Bearer <token>header, the overwhelmingly common convention, cannot exercise any authenticated request against a Floci-emulated REST API. Every such call fails at the API Gateway layer before it ever reaches the Lambda, with a confusing, unrelated-looking error message that gives no hint theAuthorizationheader is the cause.For an SPA hitting several authenticated endpoints on page load, this manifests as the entire page failing to load any data, with only generic 404s in the network tab and no clear error pointing back to the header.
Workaround we're using
Stripping the
Authorizationheader specifically at our local dev proxy layer before it reaches Floci, and relying on our backend's own dev-only fallback for user attribution in logs, since the JWT/bearer token itself is only needed for real deployments (not for local Floci testing). This is a workaround at the client/proxy layer, not a fix, it means we can't actually exercise our real auth-header code path against Floci locally.Questions for maintainers
Authorizationheader as a SigV4 signature attempt, regardless of the route's configured authorizer type?NONE-auth routes, or to disable IAM/SigV4 header interception globally for local dev use cases where it's not needed?Bug 2: Concurrent Lambda cold starts across different functions serialize/bottleneck (one call took 31s vs <1s warm)
Summary
When multiple different Lambda functions are invoked concurrently for the first time after a fresh Floci start (all cold, no warm containers yet), overall latency degrades far more than what per-function cold-start overhead alone would suggest. One function that normally responds in <1s took 31.7s when invoked concurrently alongside 5 other cold functions. The same 6 concurrent calls, once all functions are warm, complete in well under 1s total.
This looks like invocation/container-provisioning across different functions is being serialized (or heavily contended) rather than handled in parallel, which is surprising since these are unrelated functions (different code, different containers), not concurrent invocations of the same function (where serialization due to a single execution environment would be expected/correct).
Steps to reproduce
docker compose down -v --remove-orphans # or: floci reset / equivalent docker compose up -d flocifuncB) noticeably larger/slower to cold-start than the others.Actual output from our repro (function names anonymized, real timings)
6 concurrent calls, but only 4 distinct Lambda functions (funcB is hit 3x with different query params, same underlying function/container):
Cold (fresh deploy, first invocation of every function, all fired concurrently):
Wall-clock (
time) for the whole batch: ~31.7s.Warm (same 6 calls, fired again right after):
Wall-clock for the whole batch: ~0.9s.
Expected behavior
Cold-starting 4 independent, unrelated Lambda functions concurrently should scale roughly with the slowest individual cold start (each function has its own container, its own code volume, no shared state), i.e., total wall-clock should be close to
max(individual cold start times), notsum-ish/serialized. If a single function's cold start (image pull + code volume population + container start) normally takes e.g. 5-14s in isolation, 4 of them in parallel should still finish in roughly that same ballpark, not 31s+.Impact
Real-world SPA dashboards commonly fire several API calls in parallel on page load (
Promise.all/ similar) hitting different backend Lambda-backed endpoints. Right after a fresh local environment reset, this pattern reliably triggers the worst case above, a dashboard with a client-side retry budget of ~12-16s (a few retries a few seconds apart) can time out and show a "server not responding" / stuck-loading state on the very first load, even though every individual endpoint is perfectly healthy and fast once warm.Workaround we're using
Sequentially (not concurrently) invoking every distinct Lambda function once right after
deployfinishes, before opening the frontend. This "pre-warms" all functions so the first real concurrent burst from the browser hits already-warm containers. Works, but obviously isn't a real fix, and the same slowdown would presumably resurface after Floci's warm-pool idle eviction (we saw aWarm pool idle eviction enabled: timeout=300slog line) if the app sits idle for 5+ minutes and then gets hit with a concurrent burst again.Possible root cause (speculation, untested against Floci's source)
Something in the code path that provisions a new Lambda execution container (Docker image pull check / code-volume population / container create+start) may be behind a global lock, single-threaded queue, or a semaphore with concurrency=1, causing concurrent cold-start requests for different functions to queue up rather than run in parallel. Given the per-call timings above roughly step up (~8s, ~14s, ~14s, ~14s, ~14s, ~32s) rather than clustering near a single max, this looks consistent with requests being serviced roughly one-at-a-time with some overlap, not fully parallel provisioning.
Questions for maintainers