Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Per shard per namespace RPS warning log #4525

Merged
merged 8 commits into from
Jun 27, 2023
Merged

Per shard per namespace RPS warning log #4525

merged 8 commits into from
Jun 27, 2023

Conversation

pdoerner
Copy link
Contributor

What changed?

  • Changed the counting of requests to be per shard+namespace combination.
  • Warning logs are emitted when a namespace RPS on a single shard goes above a certain limit. This limit is configurable under the new system.shardPerNsRPSWarnLimit dynamic config (default 30).
  • Emitting these logs is enabled/disabled by the same flag that controls emitting shard RPS warnings: system.persistenceHealthSignalMetricsEnabled (default true)

Why?
To better detect which namespaces are causing hot shard / noisy neighbor issues

How did you test it?
Existing unit/integration/functional tests

Potential risks

  • Increased noisiness in logs
  • Increased memory usage. Request counts are now tracked as a map of maps instead of a map of counts

Is hotfix candidate?
No

@pdoerner pdoerner requested a review from yycptt June 21, 2023 19:37
@pdoerner pdoerner marked this pull request as ready for review June 21, 2023 19:41
@pdoerner pdoerner requested a review from a team as a code owner June 21, 2023 19:41
yycptt
yycptt approved these changes Jun 26, 2023
View reviewed changes
common/dynamicconfig/constants.go Outdated Show resolved Hide resolved
@pdoerner pdoerner merged commit 11a30b2 into temporalio:master Jun 27, 2023
9 checks passed
@pdoerner pdoerner deleted the per-shard-per-ns-warning branch June 27, 2023 22:18
pdoerner added a commit that referenced this pull request Jul 21, 2023
@pdoerner
Per shard per namespace RPS warning log (#4525)
4edce52 
* add warning log for high per shard per ns rps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yycptt yycptt yycptt approved these changes

Assignees
No one assigned
Labels
release/1.21.3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@pdoerner @yycptt