Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd: compact: clean partial / marked blocks concurrently #3115

Merged

Commits on Sep 2, 2020

  1. cmd: compact: clean partial / marked blocks concurrently

    Clean partially uploaded and blocks marked for deletion concurrently
    with the whole compaction/downsampling process. One iteration could
    potentially take a few days so it should be nice to periodically clean
    unneeded blocks in the background. Without this, there are huge spikes
    in block storage usage. The spike's size depends on how long it takes to
    complete one iteration.
    
    The implementation of this is simple - factored out the deletion part
    into a separate function. It is called at the end of an iteration +
    concurrently if `--wait` has been specified. Add a mutex to protect from
    concurrent runs. Delete blocks from the deletion mark map so that we
    wouldn't try to delete same blocks twice or more.
    
    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    2591e95 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2020

  1. *: update changelog, e2e tests

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Sep 3, 2020
    Configuration menu
    Copy the full SHA
    09d60b5 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2020

  1. Merge remote-tracking branch 'origin/master' into clean_partial_marke…

    …d_periodically
    
    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    8c14462 View commit details
    Browse the repository at this point in the history
  2. cmd: compact: fix according to comments

    Remove "error" from the `error` and just directly call the function.
    
    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    2e26808 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2020

  1. Merge remote-tracking branch 'origin/master' into clean_partial_marke…

    …d_periodically
    
    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    73c2b49 View commit details
    Browse the repository at this point in the history
  2. CHANGELOG: cleanups

    Forgot to remove this part while solving conflicts.
    
    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    8218d0c View commit details
    Browse the repository at this point in the history
  3. CHANGELOG: update

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    d62aead View commit details
    Browse the repository at this point in the history
  4. CHANGELOG: clean whitespace

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    GiedriusS committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    f97130f View commit details
    Browse the repository at this point in the history