Repository navigation
Releases
Releases · davyddd/ddkafka
v0.1.3
🔄 Changed
- Python 3.15 is supported (
requires-python >=3.11,<3.16). - Requires
aiokafka >= 0.12, < 0.16, so aiokafka 0.15 can be installed. It is the first aiokafka release with wheels for Python 3.15, and the older ones do not build on it, so on Python 3.15 aiokafka must be 0.15 or newer.
Full Changelog: v0.1.2...v0.1.3
v0.1.2
🔄 Changed
- Requires
ddutils >= 0.4.0, < 0.5.0. The README and the producer docstring wire the producer registry withprocess_loop_scopefrom ddutils 0.4.0 instead ofos.getpid, since anAIOKafkaProduceris bound to the event loop it was started in.
📝 Docs
- Producer: with the default
acks='all'a topic with fewer replicas than the cluster'smin.insync.replicasrejects every record withNotEnoughReplicas.create()does not fail on it, the rejection arrives later and is only visible as theKAFKA_PRODUCER: Delivery failedERROR log entry; a single-broker compose works because itsmin.insync.replicasis 1, a cluster whose topics are not yet configured for it needsProducerConfig(acks=1)until they are. TheProducerConfigdefaults are tabulated,idempotent_key=Nonesends no key and spreads the records round-robin,create_producer,serialize_keyandserialize_valueare named as public. - Consumer: the class attributes with their defaults and the checks at class definition; the offsets of a batch are recorded before it is handed out, so
commit()goes after the batch is processed; records dropped byskip_undeserializableare committed past for good; thetimestamp_threshold_minutesboundary is looked up once perget_batches()and the filter is off when nothing is younger than the threshold;ConsumerConfigis documented,enable_auto_commitaccepts onlyFalseand why. - Consumer maker:
Lockableincludes thetimeoutattribute, the lock key is'{group_id}_{topic}_partition_{partition}'solock_kwargsmust not containkey, an exit caused by an exception does not commit, thestreampacing rule (stream_round_pauseplus poll timeout within half of the lock timeout,ValueErrorotherwise) and why a streaming lock wants tens of seconds, with an example. KafkaDsnsubclasses pydantic's private_BaseMultiHostUrl, the reason forpydantic>=2.10. The layout follows dd-lib-stub, with a Compatibility section.
Full Changelog: v0.1.1...v0.1.2
v0.1.1
🔄 Changed
BaseKafkaProducer.producer_factoryis typedConnectionManagerFactory[Any, AIOKafkaProducer, Any]: a project's factory with its own connection manager type passes the type checker, which treats that parameter as invariant.- The
KafkaConsumerMakerexample wiresRedisLockwithlock_kwargs={'client_factory': redis_client_factory, 'timeout': 300}, as ddredis 0.2.0 takes it; no client has to be opened up front.
Full Changelog: v0.1.0...v0.1.1
v0.1.0
🚀 Added
create_producer(bootstrap_servers, config=None): a startedAIOKafkaProducerwith the library's serializers, thecreate_funcof the registry behind a producer factory.dsn.KafkaDsn: a settings field for the bootstrap servers,kafka://host[:port][,host[:port]], validated as a URL like pydantic'sPostgresDsn;bootstrap_serversgives the['host:port', ...]list the aiokafka clients take, a missing port is 9092.
🔄 Changed
- The producer no longer owns its
AIOKafkaProducer.BaseKafkaProducer.producer_factory, aConnectionManagerFactoryfrom ddutils called withalias, hands the producer out for every send, the way a ddsql adapter runs a query insidefactory(alias=...). The factory owns the producer's scope (aScopedRegistryofcreate_producerper alias, one producer per process and event loop), its lazy start and its stop on shutdown throughclose().
⚠️ Breaking changes
BaseKafkaProducerRepository→BaseKafkaProducer,BaseKafkaConsumerRepository→BaseKafkaConsumer,KafkaConsumerRepositoryMaker→KafkaConsumerMaker(itsconsumer_repoattribute isconsumer). They are owned by a context repository, not repositories themselves.BaseKafkaProducer: thebootstrap_serversandconfigclass attributes, the per-class lock andstop()are replaced byproducer_factory(+ optionalalias); stop the producers withproducer_factory.close()on shutdown.- Requires ddutils 0.3.1 and pydantic 2.10.
Full Changelog: v0.0.2...v0.1.0
v0.0.2
🔄 Changed
- Requires
ddutils >= 0.3.0, < 0.4.0. ddutils 0.3.0 replaced theget_builtin_base_type/convert_to_builtinhelpers of 0.2.4 withget_base_value_type/convert_to_base_value; this library used neither, so nothing changes in its behaviour. The previous< 0.3.0pin kept the new ddutils out of every project depending on this library.
Full Changelog: v0.0.1...v0.0.2
v0.0.1
🚀 Added
BaseKafkaProducerRepository: oneAIOKafkaProducerper subclass and process, lazily started, shared by all instances. Events are serialized with msgpack frommodel_dump(mode='json')and keyed byidempotent_key; aNonekey staysNoneso keyless events are spread across partitions.createandbulk_createreturn once the record is buffered; a record the broker rejects is logged with its topic and key.stop()sends what is buffered and closes the producer, for the application's shutdown hook.BaseKafkaConsumerRepository[Domain]: reads one partition in batches, turning msgpack payloads into the domain type withmodel_validate.get_batches()polls until the partition runs dry or a poll comes back thinner thanbatch_size * min_batch_fill_ratio;commit()acknowledges every batch yielded so far.get_batches(timestamp_threshold_minutes=...)reads only messages older than the threshold. A record that cannot be deserialized raisesMessageDeserializationErrorwith its topic, partition and offset;skip_undeserializable = Truelogs and drops it instead.KafkaConsumerRepositoryMaker: wraps a consumer in a distributed lock (anything withacquire,release,extendandtimeout, e.g.ddredis.lock.RedisLock), so one partition is processed by one worker at a time. Every batch is followed by a checkpoint (lock heartbeat + commit); an exit caused by an exception does not commit, so the failed batch is read again.stream=Truekeeps the consumer on its partition after the topic runs dry, with a pace check against the lock timeout.ProducerConfigandConsumerConfig: frozen pydantic models with a description per setting, forwarded to aiokafka as keyword arguments; unknown aiokafka arguments pass through.acksdefaults to'all',enable_idempotencerequires it,enable_auto_commitaccepts onlyFalse.
Supports Python 3.11–3.14 and aiokafka 0.12–0.14.
Full Changelog: https://github.com/davyddd/ddkafka/commits/v0.0.1