Skip to content

Releases

v0.1.3

Choose a tag to compare

@github-actions github-actions released this 29 Oct 13:58
7da59b7

🔄 Changed

  • Python 3.15 is supported (requires-python >=3.11,<3.16).
  • Requires aiokafka >= 0.12, < 0.16, so aiokafka 0.15 can be installed. It is the first aiokafka release with wheels for Python 3.15, and the older ones do not build on it, so on Python 3.15 aiokafka must be 0.15 or newer.

Full Changelog: v0.1.2...v0.1.3

v0.1.2

Choose a tag to compare

@github-actions github-actions released this 19 Oct 18:14
d12b9e8

🔄 Changed

  • Requires ddutils >= 0.4.0, < 0.5.0. The README and the producer docstring wire the producer registry with process_loop_scope from ddutils 0.4.0 instead of os.getpid, since an AIOKafkaProducer is bound to the event loop it was started in.

📝 Docs

  • Producer: with the default acks='all' a topic with fewer replicas than the cluster's min.insync.replicas rejects every record with NotEnoughReplicas. create() does not fail on it, the rejection arrives later and is only visible as the KAFKA_PRODUCER: Delivery failed ERROR log entry; a single-broker compose works because its min.insync.replicas is 1, a cluster whose topics are not yet configured for it needs ProducerConfig(acks=1) until they are. The ProducerConfig defaults are tabulated, idempotent_key=None sends no key and spreads the records round-robin, create_producer, serialize_key and serialize_value are named as public.
  • Consumer: the class attributes with their defaults and the checks at class definition; the offsets of a batch are recorded before it is handed out, so commit() goes after the batch is processed; records dropped by skip_undeserializable are committed past for good; the timestamp_threshold_minutes boundary is looked up once per get_batches() and the filter is off when nothing is younger than the threshold; ConsumerConfig is documented, enable_auto_commit accepts only False and why.
  • Consumer maker: Lockable includes the timeout attribute, the lock key is '{group_id}_{topic}_partition_{partition}' so lock_kwargs must not contain key, an exit caused by an exception does not commit, the stream pacing rule (stream_round_pause plus poll timeout within half of the lock timeout, ValueError otherwise) and why a streaming lock wants tens of seconds, with an example.
  • KafkaDsn subclasses pydantic's private _BaseMultiHostUrl, the reason for pydantic>=2.10. The layout follows dd-lib-stub, with a Compatibility section.

Full Changelog: v0.1.1...v0.1.2

v0.1.1

Choose a tag to compare

@github-actions github-actions released this 18 Oct 18:54
cbb7e07

🔄 Changed

  • BaseKafkaProducer.producer_factory is typed ConnectionManagerFactory[Any, AIOKafkaProducer, Any]: a project's factory with its own connection manager type passes the type checker, which treats that parameter as invariant.
  • The KafkaConsumerMaker example wires RedisLock with lock_kwargs={'client_factory': redis_client_factory, 'timeout': 300}, as ddredis 0.2.0 takes it; no client has to be opened up front.

Full Changelog: v0.1.0...v0.1.1

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 14 Oct 14:58
e83c42f

🚀 Added

  • create_producer(bootstrap_servers, config=None): a started AIOKafkaProducer with the library's serializers, the create_func of the registry behind a producer factory.
  • dsn.KafkaDsn: a settings field for the bootstrap servers, kafka://host[:port][,host[:port]], validated as a URL like pydantic's PostgresDsn; bootstrap_servers gives the ['host:port', ...] list the aiokafka clients take, a missing port is 9092.

🔄 Changed

  • The producer no longer owns its AIOKafkaProducer. BaseKafkaProducer.producer_factory, a ConnectionManagerFactory from ddutils called with alias, hands the producer out for every send, the way a ddsql adapter runs a query inside factory(alias=...). The factory owns the producer's scope (a ScopedRegistry of create_producer per alias, one producer per process and event loop), its lazy start and its stop on shutdown through close().

⚠️ Breaking changes

  • BaseKafkaProducerRepository → BaseKafkaProducer, BaseKafkaConsumerRepository → BaseKafkaConsumer, KafkaConsumerRepositoryMaker → KafkaConsumerMaker (its consumer_repo attribute is consumer). They are owned by a context repository, not repositories themselves.
  • BaseKafkaProducer: the bootstrap_servers and config class attributes, the per-class lock and stop() are replaced by producer_factory (+ optional alias); stop the producers with producer_factory.close() on shutdown.
  • Requires ddutils 0.3.1 and pydantic 2.10.

Full Changelog: v0.0.2...v0.1.0

v0.0.2

Choose a tag to compare

@github-actions github-actions released this 14 Oct 11:50
16c7c48

🔄 Changed

  • Requires ddutils >= 0.3.0, < 0.4.0. ddutils 0.3.0 replaced the get_builtin_base_type / convert_to_builtin helpers of 0.2.4 with get_base_value_type / convert_to_base_value; this library used neither, so nothing changes in its behaviour. The previous < 0.3.0 pin kept the new ddutils out of every project depending on this library.

Full Changelog: v0.0.1...v0.0.2

v0.0.1

Choose a tag to compare

@github-actions github-actions released this 11 Oct 18:53

🚀 Added

  • BaseKafkaProducerRepository: one AIOKafkaProducer per subclass and process, lazily started, shared by all instances. Events are serialized with msgpack from model_dump(mode='json') and keyed by idempotent_key; a None key stays None so keyless events are spread across partitions. create and bulk_create return once the record is buffered; a record the broker rejects is logged with its topic and key. stop() sends what is buffered and closes the producer, for the application's shutdown hook.
  • BaseKafkaConsumerRepository[Domain]: reads one partition in batches, turning msgpack payloads into the domain type with model_validate. get_batches() polls until the partition runs dry or a poll comes back thinner than batch_size * min_batch_fill_ratio; commit() acknowledges every batch yielded so far. get_batches(timestamp_threshold_minutes=...) reads only messages older than the threshold. A record that cannot be deserialized raises MessageDeserializationError with its topic, partition and offset; skip_undeserializable = True logs and drops it instead.
  • KafkaConsumerRepositoryMaker: wraps a consumer in a distributed lock (anything with acquire, release, extend and timeout, e.g. ddredis.lock.RedisLock), so one partition is processed by one worker at a time. Every batch is followed by a checkpoint (lock heartbeat + commit); an exit caused by an exception does not commit, so the failed batch is read again. stream=True keeps the consumer on its partition after the topic runs dry, with a pace check against the lock timeout.
  • ProducerConfig and ConsumerConfig: frozen pydantic models with a description per setting, forwarded to aiokafka as keyword arguments; unknown aiokafka arguments pass through. acks defaults to 'all', enable_idempotence requires it, enable_auto_commit accepts only False.

Supports Python 3.11–3.14 and aiokafka 0.12–0.14.

Full Changelog: https://github.com/davyddd/ddkafka/commits/v0.0.1