{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":22246793,"defaultBranch":"master","name":"kinesis-consumer","ownerLogin":"harlow","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2014-07-25T06:03:41.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/739782?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1657121987.8197818","currentOid":""},"activityList":{"items":[{"before":"6720a01733e68abbf0f45585c6c6ded81526c363","after":"553e2392fdf3f9e8e7859481f915d2cfc60e1502","ref":"refs/heads/master","pushedAt":"2024-06-06T15:38:17.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"harlow","name":"hw","path":"/harlow","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/739782?s=80&v=4"},"commit":{"message":"fix nil pointer dereference on AWS errors (#148)\n\n* fix nil pointer dereference on AWS errors\r\n\r\n* return Start errors to Scan consumer\r\n\r\nbefore the previous commit e465b09, client errors panicked the\r\nreader, so consumers would pick up sharditerator errors by virtue of\r\ntheir server crashing and burning.\r\n\r\nNow that client errors are properly returned, the behaviour of\r\nlistShards is problematic because it absorbs any client errors it gets.\r\n\r\nThe result of these two things now is that if you hit an aws error, your server will go into an\r\nendless scan loop you can't detect and can't easily recover from.\r\n\r\nTo avoid that, listShards will now stop if it hits a client error.\r\n\r\n---------\r\n\r\nCo-authored-by: Jarrad Whitaker ","shortMessageHtmlLink":"fix nil pointer dereference on AWS errors (#148)"}},{"before":"188bdff27839535683744a6680ded1b2e058c9fa","after":"6720a01733e68abbf0f45585c6c6ded81526c363","ref":"refs/heads/master","pushedAt":"2024-06-06T15:37:42.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"harlow","name":"hw","path":"/harlow","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/739782?s=80&v=4"},"commit":{"message":"Maintain parent/child shard ordering across shard splits/merges. (#155)\n\nKinesis allows clients to rely on an invariant that, for a given partition key, the order of records added to the stream will be maintained. IE: given an input `pkey=x,val=1 pkey=x,val=2 pkey=x,val=3`, the values `1,2,3` will be seen in that order when processed by clients, so long as clients are careful. It does so by putting all records for a single partition key into a single shard, then maintaining ordering within that shard.\r\n\r\nHowever, shards can be split and merge, to distribute load better and handle per-shard throughput limits. Kinesis does this currently by (one or many times) splitting a single shard into two or by merging two adjacent shards into one. When this occurs, Kinesis still allows for ordering consistency by detailing shard parent/child relationships within its `listShards` outputs. A split shard A will create children B and C, both with `ParentShardId=A`. A merging of shards A and B into C will create a new shard C with `ParentShardId=A,AdjacentParentShardId=B`. So long as clients fully process all records in parents (including adjacent parents) before processing the new shard, ordering will be maintained.\r\n\r\n`kinesis-consumer` currently doesn't do this. Instead, upon the initial (and subsequent) `listShards` call, all visible shards immediately begin processing. Considering this case, where shards split, then merge, and each shard `X` contains a single record `rX`:\r\n\r\n```\r\ntime ->\r\n B\r\n / \\\r\nA D\r\n \\ /\r\n C\r\n```\r\n\r\nrecord `rD` should be processed after both `rB` and `rC` are processed, and both `rB` and `rC` should wait for `rA` to be processed. By starting goroutines immediately, any ordering of `{rA,rB,rC,rD}` might occur within the original code.\r\n\r\nThis PR utilizes the `AllGroup` as a book-keeper of fully processed shards, with the `Consumer` calling `CloseShard` once it has finished a shard. `AllGroup` doesn't release a shard for processing until its parents have fully been processed, and the consumer just processes the shards it receives as it used to.\r\n\r\nThis PR created a new `CloseableGroup` interface rather than append to the existing `Group` interface to maintain backwards compatibility in existing code that may already implement the `Group` interface elsewhere. Different `Group` implementations don't get the ordering described above, but the default `Consumer` does.","shortMessageHtmlLink":"Maintain parent/child shard ordering across shard splits/merges. (#155)"}},{"before":"af2db0d43f140c05b4282144075821e38700afab","after":"188bdff27839535683744a6680ded1b2e058c9fa","ref":"refs/heads/master","pushedAt":"2023-12-15T00:36:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"harlow","name":"hw","path":"/harlow","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/739782?s=80&v=4"},"commit":{"message":"fix: typo (#156)","shortMessageHtmlLink":"fix: typo (#156)"}},{"before":"c2b9f79d7ab65f0c252e366624e96596de2a060b","after":"af2db0d43f140c05b4282144075821e38700afab","ref":"refs/heads/master","pushedAt":"2023-06-16T18:20:07.429Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"harlow","name":"hw","path":"/harlow","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/739782?s=80&v=4"},"commit":{"message":"update go-redis to v9 (#150)","shortMessageHtmlLink":"update go-redis to v9 (#150)"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEXlz8PAA","startCursor":null,"endCursor":null}},"title":"Activity · harlow/kinesis-consumer"}