[FLINK-28009] Optimize data split

[kaori-seasons]

Related to FLINK-28009
Optimizing split data logic for large data volumes using Stream Api

[JingsongLi]

Hi @complone Thanks for the contribution.
Do you mean FileStorePathFactory.createDataFilePathFactory is very slow?

[JingsongLi]

CC @tsreaper

[kaori-seasons]

@JingsongLi Yes, in the case of a large amount of data, there is a lot of data that needs to be sharded. Parallel streams will first allocate the same key to the internal bucket. I think this can improve performance.

[JingsongLi]

@JingsongLi Yes, in the case of a large amount of data, there is a lot of data that needs to be sharded. Parallel streams will first allocate the same key to the internal bucket. I think this can improve performance.

Can you have some test benchmark data? For example, how many times can it be executed in 1 second?

[kaori-seasons]

@JingsongLi According to Oracle's performance test, the parallel stream is affected by the number of cpu cores, I will change it to stream to simplify the code logic
Related to parallel-streams-performance-benchmark

[JingsongLi]

@complone Any update about benchmark?

[kaori-seasons]

@complone关于基准的任何更新？

No. Simplify code with only stream API

[JingsongLi]

Hi @complone
Before submitting a PR, it is best to discuss clearly in JIRA what it is that needs to be done and what the general idea is.
I'll close this one, if we (you and me) make further progress, we'll continue working on the code.