Repository navigation
v0.0.1
🚀 Added
BaseKafkaProducerRepository: oneAIOKafkaProducerper subclass and process, lazily started, shared by all instances. Events are serialized with msgpack frommodel_dump(mode='json')and keyed byidempotent_key; aNonekey staysNoneso keyless events are spread across partitions.createandbulk_createreturn once the record is buffered; a record the broker rejects is logged with its topic and key.stop()sends what is buffered and closes the producer, for the application's shutdown hook.BaseKafkaConsumerRepository[Domain]: reads one partition in batches, turning msgpack payloads into the domain type withmodel_validate.get_batches()polls until the partition runs dry or a poll comes back thinner thanbatch_size * min_batch_fill_ratio;commit()acknowledges every batch yielded so far.get_batches(timestamp_threshold_minutes=...)reads only messages older than the threshold. A record that cannot be deserialized raisesMessageDeserializationErrorwith its topic, partition and offset;skip_undeserializable = Truelogs and drops it instead.KafkaConsumerRepositoryMaker: wraps a consumer in a distributed lock (anything withacquire,release,extendandtimeout, e.g.ddredis.lock.RedisLock), so one partition is processed by one worker at a time. Every batch is followed by a checkpoint (lock heartbeat + commit); an exit caused by an exception does not commit, so the failed batch is read again.stream=Truekeeps the consumer on its partition after the topic runs dry, with a pace check against the lock timeout.ProducerConfigandConsumerConfig: frozen pydantic models with a description per setting, forwarded to aiokafka as keyword arguments; unknown aiokafka arguments pass through.acksdefaults to'all',enable_idempotencerequires it,enable_auto_commitaccepts onlyFalse.
Supports Python 3.11–3.14 and aiokafka 0.12–0.14.
Full Changelog: https://github.com/davyddd/ddkafka/commits/v0.0.1