-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data usage crawler refactor #9075
Merged
harshavardhana
merged 8 commits into
minio:master
from
klauspost:data-usage-crawler-recfactor
Mar 18, 2020
Merged
Data usage crawler refactor #9075
harshavardhana
merged 8 commits into
minio:master
from
klauspost:data-usage-crawler-recfactor
Mar 18, 2020
Commits on Mar 16, 2020
-
Some fix of chinese docs (minio#9140)
## Description Clarify disk (硬盘) and node (节点). Remove the limit (限制) paragraph since there are no max 16 disks limit now. ## Motivation and Context ## How to test this PR? ## Types of changes - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Checklist: - [ ] Fixes a regression (If yes, please add `commit-id` or `PR #` here) - [ ] Documentation needed - [ ] Unit tests needed - [ ] Functional tests needed (If yes, add [mint](https://github.com/minio/mint) PR # here: )
Configuration menu - View commit details
-
Copy full SHA for 7db902b - Browse repository at this point
Copy the full SHA 7db902bView commit details -
Squashed version of/replaces minio#9075 ## Description Implementation overview: https://gist.github.com/klauspost/1801c858d5e0df391114436fdad6987b Includes staticcheck upgrade and fixes for that. Some quick performance tests of crawls. This is with a cycle size of 16, 174082 XL objects. ~4K folders at prefix level 2, NVME. ``` BEFORE: Crawl time 1m17.3025259s Disk access: 350MB/s Kernel Time = 59.468 = 70% User Time = 21.984 = 25% Process Time = 81.453 = 95% Virtual Memory = 303 MB AFTER: SET MINIO_DISK_USAGE_CRAWL_DELAY=10 (default) Cycle scan time: 45.354097s Disk Access: 15MB/s Kernel Time = 0.562 = 1% User Time = 0.640 = 1% Process Time = 1.203 = 2% Virtual Memory = 304 MB SET MINIO_DISK_USAGE_CRAWL_DELAY=1 Cycle scan time: 3.3367481s Disk Access: 160MB/s Kernel Time = 0.468 = 3% User Time = 1.421 = 12% Process Time = 1.890 = 16% Virtual Memory = 303 MB SET MINIO_DISK_USAGE_CRAWL_DELAY=0 Cycle scan time: 2.1245395s Disk Access: 175MB/s Kernel Time = 1.500 = 14% User Time = 0.765 = 7% Process Time = 2.265 = 22% Virtual Memory = 304 MB ``` ## How to test this PR? For now the server will display extra information on crawling. ## Types of changes - [x] New feature (non-breaking change which adds functionality)
Configuration menu - View commit details
-
Copy full SHA for 003cb18 - Browse repository at this point
Copy the full SHA 003cb18View commit details
Commits on Mar 17, 2020
-
Configuration menu - View commit details
-
Copy full SHA for ef4fa13 - Browse repository at this point
Copy the full SHA ef4fa13View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7459324 - Browse repository at this point
Copy the full SHA 7459324View commit details -
Merge branch 'data-usage-crawler-recfactor' of github.com:klauspost/m…
…inio into data-usage-crawler-recfactor
Configuration menu - View commit details
-
Copy full SHA for 4d3b9ac - Browse repository at this point
Copy the full SHA 4d3b9acView commit details -
Configuration menu - View commit details
-
Copy full SHA for 201be4b - Browse repository at this point
Copy the full SHA 201be4bView commit details -
Configuration menu - View commit details
-
Copy full SHA for d597772 - Browse repository at this point
Copy the full SHA d597772View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b7a814 - Browse repository at this point
Copy the full SHA 8b7a814View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.