Add jitter to retries #1274
Merged
Add jitter to retries #1274
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This also removes partition_id from metric labels because of high cardinality
Example from `http localhost:5122/metrics | grep restate_`
### TaskCenter
```
# TYPE restate_task_center_finished_total counter
restate_task_center_finished_total{kind="Disposable",status="completed"} 3
restate_task_center_finished_total{kind="SystemBoot",status="completed"} 1
# TYPE restate_task_center_spawned_total counter
restate_task_center_spawned_total{kind="ConnectionReactor"} 2
restate_task_center_spawned_total{kind="Disposable"} 3
restate_task_center_spawned_total{kind="RpcServer"} 1
restate_task_center_spawned_total{kind="SystemService"} 3
restate_task_center_spawned_total{kind="IngressServer"} 1
restate_task_center_spawned_total{kind="MetadataBackgroundSync"} 1
restate_task_center_spawned_total{kind="Shuffle"} 1024
restate_task_center_spawned_total{kind="PartitionProcessor"} 1024
restate_task_center_spawned_total{kind="RoleRunner"} 1
```
### Network
```
# TYPE restate_network_connection_created_total counter
restate_network_connection_created_total{direction="outgoing"} 1
restate_network_connection_created_total{direction="incoming"} 1
# TYPE restate_network_message_sent_total counter
restate_network_message_sent_total 3
# TYPE restate_network_message_received_total counter
restate_network_message_received_total 3
# TYPE restate_network_message_processing_duration_seconds summary
restate_network_message_processing_duration_seconds{quantile="0"} 0.000020292
restate_network_message_processing_duration_seconds{quantile="0.5"} 0.000027418295730394435
restate_network_message_processing_duration_seconds{quantile="0.9"} 0.000027418295730394435
restate_network_message_processing_duration_seconds{quantile="0.95"} 0.000027418295730394435
restate_network_message_processing_duration_seconds{quantile="0.99"} 0.000027418295730394435
restate_network_message_processing_duration_seconds{quantile="0.999"} 0.000027418295730394435
restate_network_message_processing_duration_seconds{quantile="1"} 0.000082792
restate_network_message_processing_duration_seconds_sum 0.000130501
restate_network_message_processing_duration_seconds_count 3
# TYPE restate_network_connection_send_duration_seconds summary
restate_network_connection_send_duration_seconds{quantile="0"} 0.000012083
restate_network_connection_send_duration_seconds{quantile="0.5"} 0.000015583466555819782
restate_network_connection_send_duration_seconds{quantile="0.9"} 0.000015583466555819782
restate_network_connection_send_duration_seconds{quantile="0.95"} 0.000015583466555819782
restate_network_connection_send_duration_seconds{quantile="0.99"} 0.000015583466555819782
restate_network_connection_send_duration_seconds{quantile="0.999"} 0.000015583466555819782
restate_network_connection_send_duration_seconds{quantile="1"} 0.000047875
restate_network_connection_send_duration_seconds_sum 0.000075541
restate_network_connection_send_duration_seconds_count 3
```
This was referenced Mar 7, 2024
Merged
Test Results 92 files - 2 92 suites - 2 4m 6s ⏱️ +28s Results for commit a0c42fc. ± Comparison against base commit 681cd8f. This pull request removes 5 tests.♻️ This comment has been updated with latest results. |
- IngressDispatcher is not a coroutine anymore - Handles inputs concurrently - Writes to bifrost and receives responses from network
tillrohrmann
approved these changes
Mar 7, 2024
Fixes #912
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add jitter to retries
Fixes #912
Stack created with Sapling. Best reviewed with ReviewStack.