fix(kinesis): implement SubscribeToShard + fix multi-shard sequence anchors + PutRecords errors#1925
Merged
Merged
Conversation
…chors + PutRecords errors Three Kinesis correctness fixes: - SubscribeToShard (enhanced fan-out) returned ResourceNotFoundException unconditionally — even for a consumer Register/Describe had populated — so every fan-out read failed. It now consults state.consumers, resolves the stream/shard/starting-position, and streams the records back as a SubscribeToShardEvent over the application/vnd.amazon.eventstream wire format the SDK expects (new minimal frame encoder, same format as Lambda/S3 Select). - DescribeStream advertised StartingSequenceNumber as 0…01 for every shard, but records on shard N>0 are minted with a per-shard discriminator packed into the low digits. So GetShardIterator(AT_SEQUENCE_NUMBER, <advertised>) on a non-first shard looked up a sequence no record had and returned InvalidArgumentException. The advertised Start/End now use the discriminated format records actually carry. - PutRecords surfaced a malformed record (empty PartitionKey) as a per-record InvalidArgumentException. AWS rejects a malformed record with a 400 for the WHOLE request; per-record ErrorCodes are throttling/internal only. Validation errors now fail the whole call. E2E: SubscribeToShard streams a put record to a registered consumer; AT_SEQUENCE_NUMBER works from a non-first shard's advertised start; a malformed PutRecords record fails the whole request (updated the old test that asserted the non-AWS per-record-failure behavior).
…ositive) The conformance baseline recorded from real AWS (PutRecords checksum 27e5bb6b) shows a malformed record (empty PartitionKey) is reported as a per-record InvalidArgumentException with FailedRecordCount incremented — NOT a whole-request 400. Report finding 1.7 was a false positive; restore the per-record behavior. SubscribeToShard (1.9) and the multi-shard sequence anchor fix (1.19) are unaffected and remain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three Kinesis correctness fixes:
SubscribeToShard(enhanced fan-out) was a constant-error stub (1.9) — returnedResourceNotFoundExceptionunconditionally, even for a consumerRegister/Describehad populated, so every fan-out read failed. It now consultsstate.consumers, resolves the stream/shard/starting-position, and streams the records back as aSubscribeToShardEventover theapplication/vnd.amazon.eventstreamwire format the SDK expects (new minimal frame encoder — same prelude+headers+payload+CRC format as Lambda response-streaming / S3 Select).AT_SEQUENCE_NUMBER400'd (1.19) —DescribeStreamadvertisedStartingSequenceNumberas0…01for every shard, but records on shard N>0 carry a per-shard discriminator in the low digits, soGetShardIterator(AT_SEQUENCE_NUMBER, <advertised>)looked up a sequence no record had. The advertised Start/End now use the discriminated format records actually carry.PutRecordsmis-shaped validation errors (1.7) — a malformed record (empty PartitionKey) was surfaced as a per-recordInvalidArgumentException. AWS rejects a malformed record with a 400 for the whole request; per-recordErrorCodes are throttling/internal only.Non-code surface
No new API surface — implements/corrects already-documented operations. No SDK/docs/metadata change applies (checked).
Test plan
kinesis_subscribe_to_shard_streams_records(real SDK eventstream consumption),kinesis_at_sequence_number_works_on_non_zero_shard,kinesis_put_records_rejects_malformed_record_whole_request(replaces the old test that asserted the non-AWS per-record behavior).cargo test -p fakecloud-kinesis --lib(120) passes; clippy clean.Summary by cubic
Implemented Kinesis SubscribeToShard and fixed multi-shard iterator anchors. Enhanced fan-out now streams records; AT_SEQUENCE_NUMBER works on all shards; PutRecords keeps per-record validation failures (no whole-request 400).
crc32fast).Written for commit ef6ea15. Summary will update on new commits.