[WIP][SS] Python streaming source #44416
Conversation
chaoqin-li1123
changed the title
| self.value = self.from_json(json_str) | ||
|
|
||
| class DataStreamReader(ABC): | ||
| def latest_offset(self) -> DataStreamOffset: |
There was a problem hiding this comment.
By design, all Python API follows camelCase naming rule. If we could make it one word like latestoffset, that's an option too (to make it more Pythonic)
There was a problem hiding this comment.
A rather minor but pertinent point
I see
from pyspark.sql.session import SparkSession
or equally
from pyspark.sql import SparkSession
Although they are both valid, in practice, many developers prefer the second option (from pyspark.sql import SparkSession) as it is concise and aligns with the common convention of grouping related functionality under a common namespace or module. It also reflects the structure of the pyspark.sql module, where SparkSession is a key component.
| def offset_to_json(self, offset: DataStreamOffset) -> str: | ||
| ... | ||
|
|
||
| def json_to_offset(self, json: str) -> DataStreamOffset: |
There was a problem hiding this comment.
I was trying to make this method a member of DataStreamOffset, but the json string doesn't contain any type information and can't be loaded back to original class(we enforce using plaintext json because @HeartSaVioR don't want to store unreadable pickled bytes in offset log). I wonder if @HyukjinKwon have any good suggestion about the interface of deserializing json back to specific DataStreamOffset as a Python expert.
What changes were proposed in this pull request?
POC of streaming python source.
A python runner that embed a streaming data source and handle planInputPartitions and latestOffset call