Skip to content

[IOTDB-6209] Pipe: Solving the topological order of the progress index in the historical data collection phase#11478

Merged
SteveYurongSu merged 12 commits intoapache:masterfrom
yschengzi:IOTDB-6209
Nov 9, 2023
Merged

[IOTDB-6209] Pipe: Solving the topological order of the progress index in the historical data collection phase#11478
SteveYurongSu merged 12 commits intoapache:masterfrom
yschengzi:IOTDB-6209

Conversation

@yschengzi
Copy link
Contributor

Problem:
When the pipe performs historical data collection, it currently sends sequential data first and then disorganized data, which is obviously wrong because the progress index of some of the disorganized data may be smaller than that of sequential files, which causes the pipe to record the wrong progress index as the progress information, resulting in some of the disorganized files not being sent.
Solution:
After collecting all the historical data, solve for the topological order according to the progress index contained in the TsFile. Use the topological order of the file as the order in which the historical data is collected.

问题:
在 pipe 进行历史数据收集的时候,目前是先发送顺序数据,再发送乱序数据,这明显是错误的,因为部分乱序数据的 progress index 可能比顺序文件更小,导致 pipe 会记录错误的 progress index 作为进度信息,导致部分乱序文件没有被发送。
解决方案:
在收集完所有历史数据之后,按照 TsFile 包含的 progress index 求解拓扑序。使用文件的拓扑序作为历史数据收集的顺序。

@SteveYurongSu SteveYurongSu self-assigned this Nov 6, 2023
@SteveYurongSu
Copy link
Member

pull and merge master please

Copy link
Member

@SteveYurongSu SteveYurongSu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make Pipe Great Again!!!

@SteveYurongSu SteveYurongSu merged commit d3548ed into apache:master Nov 9, 2023
HTHou pushed a commit that referenced this pull request Dec 20, 2023
…x in the historical data collection phase (#11478)

* Problem:
When the pipe performs historical data collection, it currently sends sequential data first and then disorganized data, which is obviously wrong because the progress index of some of the disorganized data may be smaller than that of sequential files, which causes the pipe to record the wrong progress index as the progress information, resulting in some of the disorganized files not being sent.

* Solution:
After collecting all the historical data, solve for the topological order according to the progress index contained in the TsFile. Use the topological order of the file as the order in which the historical data is collected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants