Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add task retry scheduler #1260

Merged
merged 6 commits into from
Dec 6, 2023
Merged

feat: add task retry scheduler #1260

merged 6 commits into from
Dec 6, 2023

Conversation

will-2012
Copy link
Collaborator

@will-2012 will-2012 commented Dec 1, 2023

Description

  1. Add a task retry scheduler, ensuring objects can be sealed or reject-unsealed(the memory task will be aborted when it has been retried several times, and the task retry scheduler can retry this aborted kind of task).
  2. Object can be accessed when the payload is uploaded in the primary sp, and no need to wait for sealed.
  3. Anyone can use the query upload progress interface.

Rationale

The task retry scheduler scans the recent upload context and re-push upload context to the dispatcher. If a task has not been sealed for a long time, it will be reject-unsealed.

Example

[Manager]
# optional, default is false
EnableTaskRetryScheduler = true

# optional, default is 86400
# RejectUnsealThresholdSecond is as the following meanings:
# retry replicate and seal task when the task's create_timestamp + RejectUnsealThresholdSecond > now.time();
# retry reject unseal when the task's create_timestamp + RejectUnsealThresholdSecond <= now.time() && 
# create_timestamp + 2*RejectUnsealThresholdSecond > now.time().
RejectUnsealThresholdSecond = 86400

The above options turn on the task retry scheduler.

Changes

Notable changes:

  • Add task retry scheduler which includes background retry workflow.
  • Update get object's auth workflow.
  • Update query upload progress's auth workflow.

Potential Impacts

  • Database pressure.

@will-2012 will-2012 marked this pull request as draft December 1, 2023 16:09
@will-2012 will-2012 added the wip Working in process label Dec 1, 2023
@will-2012 will-2012 force-pushed the retry-scheduler branch 9 times, most recently from de35deb to 3c69f32 Compare December 3, 2023 09:44
@will-2012 will-2012 force-pushed the retry-scheduler branch 9 times, most recently from f151dc5 to 932b151 Compare December 3, 2023 12:41
@will-2012 will-2012 marked this pull request as ready for review December 4, 2023 09:08
@will-2012 will-2012 added r4r Ready for review and removed wip Working in process labels Dec 4, 2023
@@ -4,6 +4,9 @@ import (
"context"
"encoding/hex"
"errors"
"strings"
"testing"
"time"

sdkmath "cosmossdk.io/math"
"github.com/bnb-chain/greenfield-storage-provider/base/gfspapp"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sort imported pkgs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@@ -21,10 +24,6 @@ import (

"google.golang.org/grpc"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@@ -10,6 +10,7 @@ import (
"time"

"cosmossdk.io/math"
virtualgrouptypes "github.com/bnb-chain/greenfield/x/virtualgroup/types"
sdk "github.com/cosmos/cosmos-sdk/types"
"golang.org/x/exp/slices"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Copy link
Collaborator

@sysvm sysvm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruojunm
Copy link
Collaborator

ruojunm commented Dec 6, 2023

LGTM

@will-2012 will-2012 merged commit 5d32ab0 into develop Dec 6, 2023
11 checks passed
@will-2012 will-2012 deleted the retry-scheduler branch December 6, 2023 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
r4r Ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants