Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
33 lines (20 sloc) 1.16 KB

KafkaSourceOffset

KafkaSourceOffset is a OffsetV2 that…​FIXME

KafkaSourceOffset takes a collection of Kafka TopicPartitions and their offsets when created.

Creating KafkaSourceOffset Instance

Caution
FIXME

Getting Partition Offsets — getPartitionOffsets Method

getPartitionOffsets(offset: Offset): Map[TopicPartition, Long]

getPartitionOffsets takes KafkaSourceOffset.partitionToOffsets from offset.

If offset is KafkaSourceOffset, getPartitionOffsets takes the partitions and offsets straight from it.

If however offset is SerializedOffset, getPartitionOffsets deserializes the offsets from JSON.

getPartitionOffsets reports an IllegalArgumentException when offset is neither KafkaSourceOffset or SerializedOffset.

Invalid conversion from offset of [class] to KafkaSourceOffset
Note
getPartitionOffsets is used exclusively when KafkaSource generates a DataFrame with records from Kafka for a batch.
You can’t perform that action at this time.