Optimize overlap checking for external file ingestion #3564
Conversation
|
I have done some tests to show the comparison. During each test, 8 batches of files are ingested to a db. Files in a batch are sorted and not overlapped with each other, while files in different batches are overlapped with each other. Here are the average ingestion time and flame graph.
As you can see, without the optimization, the average ingestion time increases significantly after the 6th batch, when the test starts to ingest a lot of files in L0. |
There was a problem hiding this comment.
@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
If there are a lot of overlapped files in L0, creating a merging iterator for all files in L0 to check overlap can be very slow, because we need to read and seek all files in L0. However, in that case, the ingested file is likely to overlap with some files in L0, so if we check those files one by one, wecan stop once we encounter overlap.
|
@huachaohuang has updated the pull request. |
|
@anand1976 Done, PTAL |
|
lgtm |
There was a problem hiding this comment.
@anand1976 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
If there are a lot of overlapped files in L0, creating a merging iterator for
all files in L0 to check overlap can be very slow because we need to read and
seek all files in L0. However, in that case, the ingested file is likely to
overlap with some files in L0, so if we check those files one by one, we can stop
once we encounter overlap.
Ref: #3540