Skip to content

fix(kinesis): implement SubscribeToShard + fix multi-shard sequence anchors + PutRecords errors#1925

Merged
vieiralucas merged 3 commits into
mainfrom
fix/kinesis-subscribe-seq-putrecords
Jun 25, 2026
Merged

fix(kinesis): implement SubscribeToShard + fix multi-shard sequence anchors + PutRecords errors#1925
vieiralucas merged 3 commits into
mainfrom
fix/kinesis-subscribe-seq-putrecords

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Jun 24, 2026

Copy link
Copy Markdown
Member

Summary

Three Kinesis correctness fixes:

  1. SubscribeToShard (enhanced fan-out) was a constant-error stub (1.9) — returned ResourceNotFoundException unconditionally, even for a consumer Register/Describe had populated, so every fan-out read failed. It now consults state.consumers, resolves the stream/shard/starting-position, and streams the records back as a SubscribeToShardEvent over the application/vnd.amazon.eventstream wire format the SDK expects (new minimal frame encoder — same prelude+headers+payload+CRC format as Lambda response-streaming / S3 Select).
  2. Multi-shard AT_SEQUENCE_NUMBER 400'd (1.19) — DescribeStream advertised StartingSequenceNumber as 0…01 for every shard, but records on shard N>0 carry a per-shard discriminator in the low digits, so GetShardIterator(AT_SEQUENCE_NUMBER, <advertised>) looked up a sequence no record had. The advertised Start/End now use the discriminated format records actually carry.
  3. PutRecords mis-shaped validation errors (1.7) — a malformed record (empty PartitionKey) was surfaced as a per-record InvalidArgumentException. AWS rejects a malformed record with a 400 for the whole request; per-record ErrorCodes are throttling/internal only.

Non-code surface

No new API surface — implements/corrects already-documented operations. No SDK/docs/metadata change applies (checked).

Test plan

  • E2E kinesis_subscribe_to_shard_streams_records (real SDK eventstream consumption), kinesis_at_sequence_number_works_on_non_zero_shard, kinesis_put_records_rejects_malformed_record_whole_request (replaces the old test that asserted the non-AWS per-record behavior).
  • Unit test for the eventstream frame encoder; cargo test -p fakecloud-kinesis --lib (120) passes; clippy clean.

Summary by cubic

Implemented Kinesis SubscribeToShard and fixed multi-shard iterator anchors. Enhanced fan-out now streams records; AT_SEQUENCE_NUMBER works on all shards; PutRecords keeps per-record validation failures (no whole-request 400).

  • Bug Fixes
    • SubscribeToShard: resolves consumer/stream/shard and streams records as SubscribeToShardEvent over eventstream (adds a minimal encoder using crc32fast).
    • Multi-shard anchors: advertised start/end sequence numbers now include the per-shard discriminator, so AT_SEQUENCE_NUMBER iterators work on non-first shards.
    • PutRecords: restore AWS behavior — malformed records are reported per-record (InvalidArgumentException with FailedRecordCount); no HTTP 400 for the whole request.

Written for commit ef6ea15. Summary will update on new commits.

Review in cubic

…chors + PutRecords errors

Three Kinesis correctness fixes:

- SubscribeToShard (enhanced fan-out) returned ResourceNotFoundException
  unconditionally — even for a consumer Register/Describe had populated — so
  every fan-out read failed. It now consults state.consumers, resolves the
  stream/shard/starting-position, and streams the records back as a
  SubscribeToShardEvent over the application/vnd.amazon.eventstream wire format
  the SDK expects (new minimal frame encoder, same format as Lambda/S3 Select).

- DescribeStream advertised StartingSequenceNumber as 0…01 for every shard, but
  records on shard N>0 are minted with a per-shard discriminator packed into
  the low digits. So GetShardIterator(AT_SEQUENCE_NUMBER, <advertised>) on a
  non-first shard looked up a sequence no record had and returned
  InvalidArgumentException. The advertised Start/End now use the discriminated
  format records actually carry.

- PutRecords surfaced a malformed record (empty PartitionKey) as a per-record
  InvalidArgumentException. AWS rejects a malformed record with a 400 for the
  WHOLE request; per-record ErrorCodes are throttling/internal only. Validation
  errors now fail the whole call.

E2E: SubscribeToShard streams a put record to a registered consumer;
AT_SEQUENCE_NUMBER works from a non-first shard's advertised start; a malformed
PutRecords record fails the whole request (updated the old test that asserted
the non-AWS per-record-failure behavior).
…ositive)

The conformance baseline recorded from real AWS (PutRecords checksum 27e5bb6b)
shows a malformed record (empty PartitionKey) is reported as a per-record
InvalidArgumentException with FailedRecordCount incremented — NOT a
whole-request 400. Report finding 1.7 was a false positive; restore the
per-record behavior. SubscribeToShard (1.9) and the multi-shard sequence anchor
fix (1.19) are unaffected and remain.
@vieiralucas vieiralucas merged commit bfd0942 into main Jun 25, 2026
93 checks passed
@vieiralucas vieiralucas deleted the fix/kinesis-subscribe-seq-putrecords branch June 25, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant