feat(lambda): ESM FilterCriteria + partial-batch failure + StartingPosition by vieiralucas · Pull Request #740 · faiscadev/fakecloud

vieiralucas · 2026-04-24T22:49:17Z

Summary

Closes the long-standing event source mapping correctness gaps for SQS / Kinesis / DDB Streams pollers.

FilterCriteria — new `fakecloud-lambda::filter::FilterSet` implements the EventBridge-style JSON pattern subset documented for Lambda ESM: exact equality, array-of-scalars (or), `exists` / `prefix` / `suffix` / `equals-ignore-case` / `anything-but` / `numeric` operators, and the SQS body-decode special case (patterns rooted at `body` are matched against the JSON-parsed message body). Wired into all three pollers; non-matching records are dropped and the checkpoint/queue advances past them.
Partial-batch failure (SQS) — when `FunctionResponseTypes=[ReportBatchItemFailures]` is set, the SQS poller parses `{"batchItemFailures":[{"itemIdentifier":""}]}` from the Lambda response and only retries the failed messages.
MaximumBatchingWindowInSeconds (SQS) — partial batches are held open until the window expires, then dispatched.
StartingPosition — Kinesis (TRIM_HORIZON / LATEST / AT_TIMESTAMP) and DDB Streams (TRIM_HORIZON / LATEST) seed the per-shard checkpoint on first poll.
EventSourceMapping struct + Create/Update/Get serializers updated to round-trip `FilterCriteria`, `FunctionResponseTypes`, `StartingPosition`, `StartingPositionTimestamp`, `MaximumBatchingWindowInSeconds`, `ParallelizationFactor`.

Test plan

`cargo test -p fakecloud-lambda --lib filter` — 7 unit tests on FilterSet operator coverage
`cargo test -p fakecloud --bin fakecloud sqs_lambda_poller` — 3 unit tests on `batchItemFailures` parsing
`cargo test -p fakecloud-e2e --test lambda_esm_filter_and_failures` — SQS FilterCriteria end-to-end (matching message routed; non-matching acked off the queue)
`cargo clippy --workspace --all-targets -- -D warnings` clean
`cargo fmt --check` clean

Docs / website / README updated to advertise the new ESM behavior.

Summary by cubic

Adds FilterCriteria matching, SQS partial-batch failures, and StartingPosition to Lambda event source mappings. Pollers now drop/ack non-matching records, honor SQS batching windows, and keep checkpoints aligned with AWS semantics.

New Features
- fakecloud-lambda::filter::FilterSet adds EventBridge-style JSON pattern matching (exists/prefix/suffix/equals-ignore-case/anything-but/numeric) with SQS body JSON-decode; used by SQS, Kinesis, and DDB Streams. Non-matching records are discarded and progress advances.
- SQS: supports FunctionResponseTypes=["ReportBatchItemFailures"]; parses {"batchItemFailures":[{"itemIdentifier":"<id>"}]} and retries only failed IDs. Honors MaximumBatchingWindowInSeconds for partial batches.
- Kinesis/DDB Streams: honors StartingPosition on first poll — Kinesis (TRIM_HORIZON/LATEST/AT_TIMESTAMP) and DDB Streams (TRIM_HORIZON/LATEST).
- ESM Create/Update/Get round-trip FilterCriteria, FunctionResponseTypes, StartingPosition, StartingPositionTimestamp, MaximumBatchingWindowInSeconds, and ParallelizationFactor.
Bug Fixes
- Filters: exists now checks field presence (null counts as present); numeric requires even, non-empty operator arrays; invalid JSON patterns are rejected on create; non-string Filters[].Pattern is rejected with InvalidParameterValueException; non-object FilterCriteria is rejected at create.
- SQS: batch-failure parsing intersects itemIdentifier values with the actual batch to avoid holding unrelated messages.
- Kinesis/DDB Streams: checkpoints advance only after a successful invoke; if all records are filtered out, checkpoints still advance. Failed invokes leave records for retry (at-least-once).
- Tests: SQS->Lambda ESM e2e now uses a real Python handler, polls for the post-invoke aws:sqs record, and bounds the wait with a 30s timeout to avoid CI hangs.

^{Written for commit 293c0fc. Summary will update on new commits.}

…lure + StartingPosition Closes the long-standing ESM correctness gaps for SQS / Kinesis / DDB Streams pollers. - New `fakecloud-lambda::filter::FilterSet` implements the EventBridge-style JSON pattern subset documented for Lambda ESM FilterCriteria: exact equality, array-of-scalars (or), `exists` / `prefix` / `suffix` / `equals-ignore-case` / `anything-but` / `numeric` operators, and the SQS body-decode special case (patterns rooted at `body` are matched against the JSON-parsed message body). - EventSourceMapping struct gained `filter_patterns`, `function_response_types`, `starting_position`, `starting_position_timestamp`, `parallelization_factor`, and `maximum_batching_window_in_seconds`. Both CreateEventSourceMapping and UpdateEventSourceMapping read/write them, and the JSON serializer reflects them back on Get/List. - SQS poller honors FilterCriteria (drops + acks non-matching), MaximumBatchingWindowInSeconds (holds partial batches), and FunctionResponseTypes=[ReportBatchItemFailures] (parses `{"batchItemFailures":[{"itemIdentifier":...}]}` and only retries the failed messages instead of the entire batch). - Kinesis poller honors FilterCriteria (advances checkpoint past dropped records — AWS treats them as consumed) and StartingPosition on first poll (TRIM_HORIZON / LATEST / AT_TIMESTAMP). - DDB Streams poller honors FilterCriteria + StartingPosition (TRIM_HORIZON / LATEST — AWS doesn't accept AT_TIMESTAMP for DDB). Tests: 7 unit tests on FilterSet (operator coverage), 3 unit tests on `batchItemFailures` parsing, 1 e2e test on SQS FilterCriteria end-to-end (matching message routed, non-matching acked off the queue). Docs/website/README updated to advertise the new ESM behavior.

cubic-dev-ai

7 issues found across 12 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fakecloud-server/src/dynamodb_streams_lambda_poller.rs">

<violation number="1" location="crates/fakecloud-server/src/dynamodb_streams_lambda_poller.rs:215">
P1: Checkpoint is advanced before confirming Lambda invocation success, so failed batches are dropped instead of retried.</violation>
</file>

<file name="crates/fakecloud-lambda/src/extras.rs">

<violation number="1" location="crates/fakecloud-lambda/src/extras.rs:1690">
P2: UpdateEventSourceMapping now mutates new ESM fields but does not return them in the response, so clients cannot round-trip updated `FilterCriteria`/`FunctionResponseTypes`/batching settings from the update call.</violation>
</file>

<file name="crates/fakecloud-server/src/sqs_lambda_poller.rs">

<violation number="1" location="crates/fakecloud-server/src/sqs_lambda_poller.rs:341">
P2: `split_batch_failures` returns failure IDs verbatim from the Lambda response without intersecting them with `batch_ids`. A bogus or stale `itemIdentifier` in the response could match an unrelated message in the queue, incorrectly bumping its `receive_count` and resetting its visibility. Filter the failures to only include IDs present in the current batch.</violation>
</file>

<file name="crates/fakecloud-server/src/kinesis_lambda_poller.rs">

<violation number="1" location="crates/fakecloud-server/src/kinesis_lambda_poller.rs:133">
P1: Checkpoint is advanced before delivery, so failed invocations permanently lose records. The old code only advanced the checkpoint after a successful delivery. Move the unconditional checkpoint advance into the `matched.is_empty()` branch, and keep the post-delivery advance for the success path.</violation>
</file>

<file name="crates/fakecloud-lambda/src/filter.rs">

<violation number="1" location="crates/fakecloud-lambda/src/filter.rs:37">
P1: Invalid JSON filter patterns are silently discarded, which can turn malformed `FilterCriteria` into pass-through delivery for all records.</violation>

<violation number="2" location="crates/fakecloud-lambda/src/filter.rs:110">
P1: `exists` operator is implemented as null-check instead of field-presence check, so present `null` fields are matched incorrectly.</violation>

<violation number="3" location="crates/fakecloud-lambda/src/filter.rs:143">
P2: Malformed `numeric` arrays with odd length currently evaluate to `true`, causing false-positive matches.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

codecov · 2026-04-24T22:56:25Z

Codecov Report

❌ Patch coverage is 32.18884% with 474 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/fakecloud-server/src/sqs_lambda_poller.rs	24.36%	149 Missing ⚠️
...ates/fakecloud-server/src/kinesis_lambda_poller.rs	0.00%	136 Missing ⚠️
...cloud-server/src/dynamodb_streams_lambda_poller.rs	0.00%	68 Missing ⚠️
crates/fakecloud-lambda/src/extras.rs	0.00%	48 Missing ⚠️
crates/fakecloud-lambda/src/filter.rs	77.40%	40 Missing ⚠️
crates/fakecloud-lambda/src/service.rs	54.79%	33 Missing ⚠️

📢 Thoughts on this report? Let us know!

7 findings from Cubic, all addressed: - filter.rs: `exists` operator now tests field *presence* (matching AWS), not value-is-null. `{"foo": null}` matches `{"exists": true}` because the key is present; `{}` matches `{"exists": false}`. Internal evaluator passes `Option<&Value>` so presence and absence are distinguishable end-to-end. - filter.rs: numeric arrays must be even-length and non-empty — malformed `["<", 5, "<"]` now returns no-match instead of true. - filter.rs: invalid JSON patterns are logged at runtime, AND `FilterSet::validate` rejects them at `CreateEventSourceMapping` with `InvalidParameterValueException` (matches AWS behavior). - sqs_lambda_poller.rs: `split_batch_failures` intersects the failure list with the actual batch ids — stale or made-up `itemIdentifier` no longer holds an unrelated message visible. - kinesis_lambda_poller.rs / dynamodb_streams_lambda_poller.rs: checkpoint advances *after* a successful invoke (or after a filter drops every record). A failed invoke leaves the records pending so the next poll retries them — matches AWS at-least-once. - extras.rs: UpdateEventSourceMapping returns `FilterCriteria`, `FunctionResponseTypes`, `MaximumBatchingWindowInSeconds`, and `ParallelizationFactor` in the response so clients can round-trip updated values. 3 new unit tests cover the corrected behavior: - exists treats null-valued field as present, missing field as absent - numeric odd-length arrays do not silently match - validate rejects invalid JSON patterns

cubic-dev-ai

1 issue found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fakecloud-lambda/src/service.rs">

<violation number="1" location="crates/fakecloud-lambda/src/service.rs:1114">
P2: Validation is applied after lossy parsing, so non-string/malformed `FilterCriteria.Filters[].Pattern` values can be silently ignored instead of rejected.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

Cubic flagged that the previous extractor used `as_str` + filter_map, which silently dropped entries where Pattern was a number, object, or boolean. Validation then ran against the cleaned list and accepted it. Strict extraction now rejects any non-string Pattern with InvalidParameterValueException before validation runs.

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fakecloud-lambda/src/service.rs">

<violation number="1" location="crates/fakecloud-lambda/src/service.rs:1108">
P2: `FilterCriteria` type is not validated: non-object values are treated as absent and silently accepted.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

The post-Cubic correctness fix only deletes SQS messages on a successful Lambda invocation; the prior "fake-code" blob caused the runtime to error and the poller to leave the message visible for retry. Use the same runnable Python handler the Kinesis ESM test uses so the assertion that the message is consumed reflects real Lambda success, not poller leniency.

…artup The fixed 2-second sleep before asserting the SQS->Lambda invocation isn't long enough in CI when Docker pulls the Python runtime image: the poller's "aws:sqs" record only lands after invoke completes, and the test was reading the in-flight "aws:lambda:delivery" record from the LambdaDelivery adapter as the last entry. Replace the sleep + last-entry lookup with a polling loop that waits for the "aws:sqs" record explicitly, and use `.iter().rev().find()` to pick it out of the invocation list (matching the kinesis test).

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fakecloud-e2e/tests/cross_service.rs">

<violation number="1" location="crates/fakecloud-e2e/tests/cross_service.rs:553">
P2: Add a timeout to this polling loop so the test fails instead of hanging forever when the invocation never arrives.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

…ping

cubic-dev-ai Bot reviewed Apr 24, 2026

View reviewed changes

Comment thread crates/fakecloud-lambda/src/service.rs

cubic-dev-ai Bot reviewed Apr 24, 2026

View reviewed changes

Comment thread crates/fakecloud-lambda/src/service.rs Outdated

vieiralucas added 2 commits April 24, 2026 20:47

cubic-dev-ai Bot reviewed Apr 25, 2026

View reviewed changes

Comment thread crates/fakecloud-e2e/tests/cross_service.rs Outdated

vieiralucas added 2 commits April 24, 2026 21:16

test(lambda): bound the SQS->Lambda poll loop with a 30s timeout

7a19e2e

fix(lambda): reject non-object FilterCriteria at CreateEventSourceMap…

293c0fc

…ping

vieiralucas merged commit d892a53 into main Apr 25, 2026
48 checks passed

vieiralucas deleted the worktree-integrations-batch3b-lambda-esm branch April 25, 2026 01:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lambda): ESM FilterCriteria + partial-batch failure + StartingPosition#740

feat(lambda): ESM FilterCriteria + partial-batch failure + StartingPosition#740
vieiralucas merged 7 commits intomainfrom
worktree-integrations-batch3b-lambda-esm

vieiralucas commented Apr 24, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

vieiralucas commented Apr 24, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vieiralucas commented Apr 24, 2026 •

edited by cubic-dev-ai Bot

Loading

codecov Bot commented Apr 24, 2026 •

edited

Loading