Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance workflowDeletionTaskJitterRange to handle deletes piling up when many workflows have finished at the same time. #5020

Merged
merged 3 commits into from
Nov 3, 2022

Conversation

ZackLK
Copy link
Contributor

@ZackLK ZackLK commented Nov 2, 2022

  • Fix panics when values < 1 are provided
  • Randomize the timestamp on a second granularity so that deletes are
    more spread out than before.
  • Default to 60 minutes.

Tested with current unit tests which change the value

…hen many workflows have finished at the same time.

* Fix panics when values < 1 are provided
* Randomize the timestamp on a second granularity so that deletes are
  more spread out than before.
* Default to 60 minutes.
retentionDuration := (time.Duration(retentionInDays) * time.Hour * 24) + (time.Duration(rand.Intn(workflowDeletionTaskJitterRange)) * time.Minute)
retentionDuration := (time.Duration(retentionInDays) * time.Hour * 24)
if workflowDeletionTaskJitterRange > 1 {
retentionDuration += time.Duration(rand.Intn(workflowDeletionTaskJitterRange*60)) * time.Second
Copy link
Contributor

@Groxx Groxx Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alternatively: random float within that range, then the order of multiplication doesn't matter. or multiply by time.Minute before rand.Intn.

though that's probably a fair bit more casting, so also meh - this works, and it's probably a good enough spread of values 👍.
maybe with generics we'll have a more concise way to do this.

…hen many workflows have finished at the same time.

* Fix panics when values < 1 are provided
* Randomize the timestamp on a second granularity so that deletes are
  more spread out than before.
* Default to 60 minutes.
@coveralls
Copy link

coveralls commented Nov 3, 2022

Pull Request Test Coverage Report for Build 01843ae4-aa5d-49f3-8d54-3ffa2e52e223

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 145 unchanged lines in 20 files lost coverage.
  • Overall coverage decreased (-0.03%) to 57.326%

Files with Coverage Reduction New Missed Lines %
common/types/shared.go 1 41.66%
client/history/client.go 2 38.1%
client/history/metricClient.go 2 45.3%
common/membership/hashring.go 2 83.54%
common/task/weightedRoundRobinTaskScheduler.go 2 89.12%
common/util.go 2 52.44%
service/history/handler.go 2 47.17%
common/persistence/serialization/parser.go 4 63.91%
common/persistence/serialization/thrift_decoder.go 4 57.14%
common/types/mapper/thrift/shared.go 4 63.21%
Totals Coverage Status
Change from base Build 01843abf-475f-4e6c-b729-d96af5563904: -0.03%
Covered Lines: 85245
Relevant Lines: 148702

💛 - Coveralls

@ZackLK ZackLK enabled auto-merge (squash) November 3, 2022 00:29
@ZackLK ZackLK merged commit a668ce1 into uber:master Nov 3, 2022
@ZackLK ZackLK deleted the delete branch November 3, 2022 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants