Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data usage crawler refactor #9075

Merged
merged 8 commits into from
Mar 18, 2020

Commits on Mar 16, 2020

  1. Some fix of chinese docs (minio#9140)

    ## Description
    Clarify disk (硬盘) and node (节点).
    Remove the limit (限制) paragraph since there are no max 16 disks limit now.
    
    ## Motivation and Context
    
    
    ## How to test this PR?
    
    
    ## Types of changes
    - [x] Bug fix (non-breaking change which fixes an issue)
    - [ ] New feature (non-breaking change which adds functionality)
    - [ ] Breaking change (fix or feature that would cause existing functionality to change)
    
    ## Checklist:
    - [ ] Fixes a regression (If yes, please add `commit-id` or `PR #` here)
    - [ ] Documentation needed
    - [ ] Unit tests needed
    - [ ] Functional tests needed (If yes, add [mint](https://github.com/minio/mint) PR # here: )
    yeungc committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    7db902b View commit details
    Browse the repository at this point in the history
  2. Data usage crawler refactor

    Squashed version of/replaces minio#9075
    
    ## Description
    
    Implementation overview: https://gist.github.com/klauspost/1801c858d5e0df391114436fdad6987b
    
    Includes staticcheck upgrade and fixes for that.
    
    Some quick performance tests of crawls.
    
    This is with a cycle size of 16, 174082 XL objects. ~4K folders at prefix level 2, NVME.
    
    ```
    BEFORE:
    Crawl time 1m17.3025259s
    Disk access: 350MB/s
    Kernel  Time =    59.468 =   70%
    User    Time =    21.984 =   25%
    Process Time =    81.453 =   95%    Virtual  Memory =    303 MB
    
    AFTER:
    SET MINIO_DISK_USAGE_CRAWL_DELAY=10 (default)
    Cycle scan time: 45.354097s
    Disk Access: 15MB/s
    Kernel  Time =     0.562 =    1%
    User    Time =     0.640 =    1%
    Process Time =     1.203 =    2%    Virtual  Memory =    304 MB
    
    SET MINIO_DISK_USAGE_CRAWL_DELAY=1
    Cycle scan time: 3.3367481s
    Disk Access: 160MB/s
    Kernel  Time =     0.468 =    3%
    User    Time =     1.421 =   12%
    Process Time =     1.890 =   16%    Virtual  Memory =    303 MB
    
    SET MINIO_DISK_USAGE_CRAWL_DELAY=0
    Cycle scan time: 2.1245395s
    Disk Access: 175MB/s
    Kernel  Time =     1.500 =   14%
    User    Time =     0.765 =    7%
    Process Time =     2.265 =   22%    Virtual  Memory =    304 MB
    ```
    
    ## How to test this PR?
    
    For now the server will display extra information on crawling.
    
    ## Types of changes
    - [x] New feature (non-breaking change which adds functionality)
    klauspost committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    003cb18 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2020

  1. Configuration menu
    Copy the full SHA
    ef4fa13 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7459324 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'data-usage-crawler-recfactor' of github.com:klauspost/m…

    …inio into data-usage-crawler-recfactor
    klauspost committed Mar 17, 2020
    Configuration menu
    Copy the full SHA
    4d3b9ac View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    201be4b View commit details
    Browse the repository at this point in the history
  5. Update documentation.

    klauspost committed Mar 17, 2020
    Configuration menu
    Copy the full SHA
    d597772 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8b7a814 View commit details
    Browse the repository at this point in the history