Skip to content

v0.0.1

Choose a tag to compare

@github-actions github-actions released this 11 Oct 18:53
· 10 commits to main since this release

🚀 Added

  • BaseKafkaProducerRepository: one AIOKafkaProducer per subclass and process, lazily started, shared by all instances. Events are serialized with msgpack from model_dump(mode='json') and keyed by idempotent_key; a None key stays None so keyless events are spread across partitions. create and bulk_create return once the record is buffered; a record the broker rejects is logged with its topic and key. stop() sends what is buffered and closes the producer, for the application's shutdown hook.
  • BaseKafkaConsumerRepository[Domain]: reads one partition in batches, turning msgpack payloads into the domain type with model_validate. get_batches() polls until the partition runs dry or a poll comes back thinner than batch_size * min_batch_fill_ratio; commit() acknowledges every batch yielded so far. get_batches(timestamp_threshold_minutes=...) reads only messages older than the threshold. A record that cannot be deserialized raises MessageDeserializationError with its topic, partition and offset; skip_undeserializable = True logs and drops it instead.
  • KafkaConsumerRepositoryMaker: wraps a consumer in a distributed lock (anything with acquire, release, extend and timeout, e.g. ddredis.lock.RedisLock), so one partition is processed by one worker at a time. Every batch is followed by a checkpoint (lock heartbeat + commit); an exit caused by an exception does not commit, so the failed batch is read again. stream=True keeps the consumer on its partition after the topic runs dry, with a pace check against the lock timeout.
  • ProducerConfig and ConsumerConfig: frozen pydantic models with a description per setting, forwarded to aiokafka as keyword arguments; unknown aiokafka arguments pass through. acks defaults to 'all', enable_idempotence requires it, enable_auto_commit accepts only False.

Supports Python 3.11–3.14 and aiokafka 0.12–0.14.

Full Changelog: https://github.com/davyddd/ddkafka/commits/v0.0.1