Refactor: change indexing generation function #92
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 1529261724
💛 - Coveralls |
|
This pull request introduces 2 alerts when merging 28224f6 into 1dbce0e - view on LGTM.com new alerts:
|
Member
Author
|
@shokakucarrier This is a proposal now to support changes in a new coming requirements in MMENG-2319. If you think it is not a correct change, let's hold it. |
ligangty
changed the title
|
This pull request introduces 1 alert when merging 1c65138 into dabc485 - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 8bbabe0 into ceea2f1 - view on LGTM.com new alerts:
|
Member
|
@ligangty using delimiter is a good idea, you we can merge this, do you need me to do further testing and potential fixing on this? |
Member
Author
|
@shokakucarrier Ok. No worry about testing. Next step we will do staging testing with this pr. |
The old way of indexing generation function is using a .index file as cache for each changed directory to avoid full scan of the whole s3 bucket, this brings bunch of complicated steps, like recursive folder scan and reading, and local file reading of .index to do merging. The new way is using a new function to scan and retrieving the contents in a single folder, which is cheap way to get the contents to avoid full scan.
ligangty
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The old way of indexing generation function is using a .index file as
cache for each changed directory to avoid full scan of the whole s3
bucket, this brings bunch of complicated steps, like recursive folder
scan and reading, and local file reading of .index to do merging. The
new way is using a new function to scan and retrieving the contents
in a single folder, which is cheap way to get the contents to avoid
full scan.