Skip to content

3.0.3

Latest

Choose a tag to compare

@hampsterx hampsterx released this 02 Nov 21:04
b32b19c

What's changed

  • A shard produced by MergeShards is read only after both of its parents are exhausted. The consumer tracked ParentShardId alone, so it started the merged child as soon as that parent drained, while the adjacent parent could still hold unread records for the same keys. A shard that is itself a parent now also waits for its own parent, which fixes the same ordering break for a split child that is split again (#123).
  • A MsgpackProcessor consumer skips an empty record. Before 3.0.2 a producer could write one, and parsing it raised inside the consumer's fetch: records ahead of it in the batch were re-delivered at every retry, records behind it never arrived, and after 10 consecutive errors the consumer stopped fetching on every shard (#119).
  • A netstring frame with a negative length raises ValueError instead of yielding the same item forever, which spun a MsgpackProcessor consumer's fetch task with no error and no backoff (#120).
  • get_shard_status() reports resharding accurately: a parent drained before the next topology refresh counts as exhausted (#116), and resharding_in_progress returns to False once a split parent ages out of the shard list, where it used to stay True for good after the stream's retention period (#121). shard_details now includes adjacent_parent_shard_id (#123).
  • ExceededPutLimit states the framed size and the actual byte limit instead of always saying 1MB, and a KPL item too big even without framing raises a ValueError that names the record limit (#117, #120). The README now says KPL max_size is in bytes where the other processors take KB.
  • async-kinesis list shows each stream's open shard count. The Shards column read ? on every backend, because ListStreams summaries carry no shard count (#118).
  • CI tests DynamoDB checkpointer leases and AT_TIMESTAMP reads against Floci (#122), and RELEASING.md covers the release environment, the PyPI approval and a tag whose tests fail (#115).

Behaviour changes

  • After a MergeShards, or a split of a shard that was itself a split child, the new shard starts only once its parents are fully read, so its first records arrive later than before.
  • async-kinesis list calls DescribeStreamSummary once per stream, paced at one call per 0.1s (about 10s per 100 streams). A failed lookup shows ? and logs a warning.
  • Code that matched the text of ExceededPutLimit, or of the KPL "too big" ValueError, sees new messages. The exception types are unchanged.
  • The docs/common-patterns.md and docs/troubleshooting.md monitoring examples warn on resharding_in_progress alone. Copies that also warn on parent_shards > 0 will warn for the whole retention period after every split.

Full changelog: 3.0.2...3.0.3