Multi-partition topics with key-based routing. A topic can now own N partitions,
each with its own writer task and log, so producers write in parallel. Records
route by key (hash(key) % partitions) so same-key records keep their order;
keyless records spread round-robin. A single consumer reads all partitions merged
into one stream. Ordering is guaranteed within a partition; cross-partition
order is intentionally undefined — the standard Kafka contract. Default partition
count is 1, so single-partition topics behave as before.
Added
- Resumable consumers via committed offsets (consumer groups, part 1).
Kafko::consumer_for_group(topic, group)returns a consumer that resumes from
the group's durably committed position instead of offset 0 — the
"continue where it stopped after a restart" behaviour.Consumer::commit()
persists the current per-partition read position (atomic temp + fsync + rename
to<topic>/offsets/<group>, CRC-framed);Consumer::committed(partition)and
Consumer::group()introspect. Distinct groups on a topic keep independent
positions (durable pub/sub fan-out). A torn/corrupt offset file degrades to
"start from 0", never an error.consumer_forstays anonymous (reads from 0;
commit()is a no-op). New errorKafkoError::InvalidGroupName.
Single active consumer per group for now; multi-member partition assignment +
rebalancing is a later slice. Kafko::create_topic_with_partitions(name, count)and
create_topic_with_config_and_partitions(name, cfg, count)— create a topic
withcountpartitions.count == 0returnsKafkoError::InvalidPartitionCount.Kafko::partition_count(name) -> Option<u32>.Topic— public type owning a topic's partitions and its routing (FNV-1a key
hash + round-robin). Obtained viaKafko::topic.RecordPosition { partition, offset }— returned by allProducersend
methods.Producer::send_to(partition, key, value)for explicit partition targeting,
andProducer::partition_count().Consumer::next_with_position()(record + itsRecordPosition),
from_topic/from_topic_at,seek_all(offset),seek(partition, offset),
position(partition),partition_count().
Changed (breaking)
Producer::send/send_recordnow returnRecordPositioninstead of a
bareu64offset (an offset is only meaningful within a partition).send_batch
/send_batch_recordsreturnVec<RecordPosition>. Read.offset()/
.partition()on the result.send_batchatomicity is now per-partition. A batch whose records route to
different partitions is atomic within each partition, not across them. For a
single-partition topic this is unchanged (one fully-atomic append).- On-disk layout changed to
<dir>/<topic>/<partition>/<segments>. Data
directories written by kafko <= 0.2 (segments directly under the topic dir) will
not open —Kafko::openreturnsKafkoError::InvalidTopicLayout. There is no
automatic migration. Kafko::topicreturnsArc<Topic>(wasArc<Partition>).Producer::newtakesArc<Topic>;Consumeris built from aTopic
(from_topic/from_topic_at) andConsumer::seeknow takes a partition
index (useseek_allfor the old whole-stream behaviour).