Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return early when there are no replication tasks #4982

Merged
merged 3 commits into from
Sep 1, 2022

Conversation

vytautas-karpavicius
Copy link
Contributor

What changed?

  • Return early when reading replication tasks if it is clear that no task will be returned. Return empty response immediately.
  • When generating task IDs, ensure that replication tasks will get generated last. This will mean that we can later quickly compare the last shard task ID (with GetTransferMaxReadLevel) with lastReadTaskID coming from remote cluster to decide if there are any potential replication tasks to fetch.

Why?
GetReplicationTasks is one of the most intensive DB operations. It is called periodically when remote clusters fetch replication messages (for each shard). It is entirely possible that a shard did not encounter any activity since last call. We can check that and skip DB call.

How did you test it?

  • Unit test
  • Tested on staging2. It significantly reduces db calls. Although traffic is low on there. On production it may not be that significant.

Potential risks

Release notes

Documentation Changes

@vytautas-karpavicius vytautas-karpavicius marked this pull request as ready for review August 31, 2022 12:53
@coveralls
Copy link

coveralls commented Aug 31, 2022

Pull Request Test Coverage Report for Build 0182f7a2-d4ae-4143-936a-316742c6ba89

  • 10 of 12 (83.33%) changed or added relevant lines in 2 files are covered.
  • 153 unchanged lines in 19 files lost coverage.
  • Overall coverage decreased (-0.05%) to 57.822%

Changes Missing Coverage Covered Lines Changed/Added Lines %
service/history/shard/context.go 5 7 71.43%
Files with Coverage Reduction New Missed Lines %
client/history/client.go 2 38.1%
client/history/metricClient.go 2 45.3%
common/persistence/executionManager.go 2 78.83%
service/history/execution/mutable_state_util.go 2 36.14%
service/history/handler.go 2 47.17%
service/history/task/transfer_active_task_executor.go 2 71.58%
service/history/task/transfer_standby_task_executor.go 2 87.21%
service/matching/taskListManager.go 2 77.11%
service/frontend/workflowHandler.go 4 59.98%
common/persistence/statsComputer.go 5 94.64%
Totals Coverage Status
Change from base Build 0182f5a6-6cfb-4b7b-85a5-434e50ab67c4: -0.05%
Covered Lines: 84485
Relevant Lines: 146113

💛 - Coveralls

@vytautas-karpavicius vytautas-karpavicius requested a review from a team August 31, 2022 12:57
@vytautas-karpavicius vytautas-karpavicius changed the title Return early when there are not replication tasks Return early when there are no replication tasks Aug 31, 2022
@vytautas-karpavicius vytautas-karpavicius merged commit e4b0a83 into master Sep 1, 2022
@vytautas-karpavicius vytautas-karpavicius deleted the replication-early-return branch September 1, 2022 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants