Are there any plans to support serialization of pyflink's Java and python inter-process communication? #1919
Description
Feature Request
We expect to improve the performance of pyflink through fury serialization
In the python benchmark test, the time-consuming benchmark of each serialization is as follows:
This is the code location where the performance is relatively high when we use pyflink, which is mainly consumed in pickle encoding and decoding.
At present, our company's stock of historical data is 13 million, and each message is between 60kb and 75kb. After discussions with the pyflink community, it is recommended to use pemja for cross-language calls without using beam.
In this way, python's native pickle serialization is very slow
测试方法：
- 以天（一个点位每秒一条数据，一天共86400条）为单位，进行不同的数据量测试
- 分别测试 3 个算子、5 个算子和 10 个算子情况下的性能
- 对比都带有 output_type 和不都带 output_type 参数的性能
-
每个算子参数带有 output_type 参数的测试代码：
-
每个算子都不带 output_type 参数的测试代码：
其中 output_type 定义了传输数据每个字段类型，定义方式如下图：
a) 测试三个算子
| 时长 | 带有 output_type 耗时（秒） | 没有 output_type耗时（秒） | 提升效率 |
|---|---|---|---|
| 1 天 | 9.456 | 9.973 | 5.18% |
| 3 天 | 14.532 | 18.187 | 20.10% |
| 5 天 | 28.911 | 38.786 | 25.46% |
| 7 天 | 34.397 | 51.691 | 33.46% |
b) 测试5个算子
| 时长 | 带有 output_type 耗时（秒） | 没有 output_type耗时（秒） | 提升效率 |
|---|---|---|---|
| 1 天 | 9.971 | 10.401 | 4.13% |
| 3 天 | 20.308 | 25.744 | 21.12% |
| 5 天 | 30.166 | 40.305 | 25.16% |
| 7 天 | 40.340 | 54.405 | 25.85% |
c) 测试10个算子
| 时长 | 带有 output_type 耗时（秒） | 没有 output_type耗时（秒） | 提升效率 |
|---|---|---|---|
| 1 天 | 11.468 | 12.130 | 5.45% |
| 3 天 | 23.697 | 31.121 | 23.85% |
| 5 天 | 38.015 | 49.508 | 23.21% |
| 7 天 | 48.859 | 65.140 | 24.99% |
Judging from the test results, explicitly specifying the output_type parameter in PyFlink DataStream can significantly improve serialization performance, especially when the amount of data is large and there are many operators, the improvement effect is more obvious. Using output_type can reduce the overhead of serialization and deserialization, reduce the calculation of type inference, and thus improve performance.
But now, obviously we hope that fury can improve this situation. Does @chaokunyang have any good suggestions?
Is your feature request related to a problem? Please describe
No response
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response