Repository navigation
What's changed
- A shard produced by
MergeShardsis read only after both of its parents are exhausted. The consumer trackedParentShardIdalone, so it started the merged child as soon as that parent drained, while the adjacent parent could still hold unread records for the same keys. A shard that is itself a parent now also waits for its own parent, which fixes the same ordering break for a split child that is split again (#123). - A
MsgpackProcessorconsumer skips an empty record. Before 3.0.2 a producer could write one, and parsing it raised inside the consumer's fetch: records ahead of it in the batch were re-delivered at every retry, records behind it never arrived, and after 10 consecutive errors the consumer stopped fetching on every shard (#119). - A netstring frame with a negative length raises
ValueErrorinstead of yielding the same item forever, which spun aMsgpackProcessorconsumer's fetch task with no error and no backoff (#120). get_shard_status()reports resharding accurately: a parent drained before the next topology refresh counts as exhausted (#116), andresharding_in_progressreturns toFalseonce a split parent ages out of the shard list, where it used to stayTruefor good after the stream's retention period (#121).shard_detailsnow includesadjacent_parent_shard_id(#123).ExceededPutLimitstates the framed size and the actual byte limit instead of always saying 1MB, and a KPL item too big even without framing raises aValueErrorthat names the record limit (#117, #120). The README now says KPLmax_sizeis in bytes where the other processors take KB.async-kinesis listshows each stream's open shard count. The Shards column read?on every backend, becauseListStreamssummaries carry no shard count (#118).- CI tests DynamoDB checkpointer leases and
AT_TIMESTAMPreads against Floci (#122), and RELEASING.md covers the release environment, the PyPI approval and a tag whose tests fail (#115).
Behaviour changes
- After a
MergeShards, or a split of a shard that was itself a split child, the new shard starts only once its parents are fully read, so its first records arrive later than before. async-kinesis listcallsDescribeStreamSummaryonce per stream, paced at one call per 0.1s (about 10s per 100 streams). A failed lookup shows?and logs a warning.- Code that matched the text of
ExceededPutLimit, or of the KPL "too big"ValueError, sees new messages. The exception types are unchanged. - The
docs/common-patterns.mdanddocs/troubleshooting.mdmonitoring examples warn onresharding_in_progressalone. Copies that also warn onparent_shards > 0will warn for the whole retention period after every split.
Full changelog: 3.0.2...3.0.3