You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To support our US-based users, this release was validated on a US cloud deployment, composed of a DSS instance in the AWS us-east-2 region (3-nodes) and another instance in the GCP us-central1-a region (3-nodes). The latency between those two regions is about 30 ms, as shown by the latency table extracted from the CockroachDB console:
We also tested this release on the normal EU cloud deployment, as in previous releases, to ensure no regression, with the following latency:
Details of the deployment are recorded with the test results.
No functional regression
As evidenced by the attached test results, this release does not introduce any functional regression. It includes the output of prober and USS qualifier runs using monitoring v0.34.0, and covers both CockroachDB and YugabyteDB deployments.
No performance regression with default configuration
In its default configuration (i.e. without the time-based notification index), we identified no performance regression in the DSS. This was validated USS qualifier and prober test suites and evidenced by v0.23.0_EU_10workers_without-time-based-notif-idx.zip, which contains the results for the test setup used in earlier releases (EU deployment).
Observed performance impact of activating time-based notification index
This release ships with an optional flag enable_time_based_notification_index which, when enabled, alters the behaviour of the DSS and its performance. They are improved or worsened depending on the testing scenario, and there is a balance to be found on whether or not the users should enable it in their deployment.
On the EU deployment, while activating the flag, the prober heavy traffic concurrent test failed and had to be adjusted to 8 concurrent workers instead of 10 as a measure of limiting load so that the test succeeds. We believe that the operational regime reached by the DSS under this load is invalid. Similarly, US cloud deployment, where the latency is significantly higher than on the EU cloud deployment, the parallelism had to be set to 2. This was expected since this scenario is generating a similar load as the busy s2 cell scenario 1.
For testing scenarios 2 and 3 (Global Throughput (Multi Site)) there are significant improvements as recorded by locust tests captured below. In scenario 2, there is a 1.5x QPS improvement (in the best case) and in scenario 3, a 2x QPS improvement (in the best case).
Interestingly, the concurrency test of the USS qualifier fails on the US deployment without activating the flag. Enabling the flag improves performance sufficiently to allow those tests to pass.
Configuration tuning
While verifying and improving the performance of the DSS with multiple benchmarks and test runs, few key characteristics have been identified to significantly impact the performance. These include, in order of importance:
Latency
Latency between datastore nodes is one of the main factors limiting performance. Due to how CockroachDB works, it creates a limit on how many queries can be performed.
We recommend limiting latency as much as possible, by keeping datastore nodes as close together as possible (e.g. on west coast datacenters / providers only).
datastore_max_open_conns
The maximum number of open connections maintained with the datastore caps concurrent query execution per client (core-service instance). A value that is too low restricts in-flight requests and amplifies client-side retry bottlenecks since connections are released between retry. The default may be increased if the cluster has available headroom but maximum parallelism shall be balanced against the risk of database exhaustion.
Time-based notification index flag
This release introduces a new flag to enable time-based notification indexes, described in interuss/dss#1541.
Using this flag improves performance in most cases, but we also observed a decrease in performance in some specific scenarios.
The performance degradation was observed in a busy S2 cell scenario (described here), when using benchmarks that try to perform as many queries as possible in a given timeframe, but these tests are known to potentially be on a invalid operational regime.
There is no single best answer on whether to enable this flag, as it depends on many factors, including the number of datastore nodes, latency and traffic patterns.
We recommend testing in an environment similar to your production environment and, based on the results, enabling this flag. We do believe it improves performance in most cases.
Group subscriptions and limit implicit ones
If possible, don't rely on implicit subscriptions when creating flights, but use a dedicated one instead. This has been shown to improve performance, as it reduces the number of steps required, as explained in this comment.
Yugabyte datastore
As of now, there has been little performance work on YugabyteDB, as it was recently introduced and most of the focus has been on CockroachDB, which is still primarily used.
Based on that and on some quick performance benchmarks, we don't recommend using YugabyteDB for performance-sensitive deployments.
Notes on testing
Performance testing is a complex subject, as many variables contribute to the final results and there are different ways to measure performance.
Every case is unique, and it's hard to give a definitive answer for every parameter to get the most performance out of a DSS instance or pool. Some settings that work better in some cases won't in others.
Comparisons between tests must be made carefully, as all known variables have to be identical to allow a fair comparison between two parameters. We also recorded some unavoidable variance, due to how CockroachDB spreads data and uses indexes.
We welcome any feedback on tests performed, as well as detailed information about practical, production deployments, to help with testing and optimization in real-world cases.
Future improvements
We think we're at the limit of what we can get from CockroachDB / YugabyteDB, due to the constraints of working with an SQL interface, both on our side and on theirs. This forces us to use non-optimal solutions for our specific use cases.
To go beyond these limitations, work has started on a Raft implementation, with better performance as synchronization between nodes is limited to a single action. Initial tests and benchmarks show improved performance and full compliance with the test suite. This new implementation is expected to provide more control over the data management and simplify performance characterization.
The latest update on the implementation can be found here: #1633
Appendices
Release validation results
v0.23.0 test results archives (using monitoring v0.34.0):
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Release v0.23.0 - Extended validation
Cloud deployments for this release
To support our US-based users, this release was validated on a US cloud deployment, composed of a DSS instance in the AWS us-east-2 region (3-nodes) and another instance in the GCP us-central1-a region (3-nodes). The latency between those two regions is about 30 ms, as shown by the latency table extracted from the CockroachDB console:
We also tested this release on the normal EU cloud deployment, as in previous releases, to ensure no regression, with the following latency:
Details of the deployment are recorded with the test results.
No functional regression
As evidenced by the attached test results, this release does not introduce any functional regression. It includes the output of prober and USS qualifier runs using monitoring v0.34.0, and covers both CockroachDB and YugabyteDB deployments.
No performance regression with default configuration
In its default configuration (i.e. without the time-based notification index), we identified no performance regression in the DSS. This was validated USS qualifier and prober test suites and evidenced by
v0.23.0_EU_10workers_without-time-based-notif-idx.zip, which contains the results for the test setup used in earlier releases (EU deployment).Observed performance impact of activating time-based notification index
This release ships with an optional flag enable_time_based_notification_index which, when enabled, alters the behaviour of the DSS and its performance. They are improved or worsened depending on the testing scenario, and there is a balance to be found on whether or not the users should enable it in their deployment.
On the EU deployment, while activating the flag, the prober heavy traffic concurrent test failed and had to be adjusted to 8 concurrent workers instead of 10 as a measure of limiting load so that the test succeeds. We believe that the operational regime reached by the DSS under this load is invalid. Similarly, US cloud deployment, where the latency is significantly higher than on the EU cloud deployment, the parallelism had to be set to 2. This was expected since this scenario is generating a similar load as the busy s2 cell scenario 1.
Locust tests have shown that performance is worse under invalid operational regime but perform better under preferred operational regime with limited parallelization, see appendix for results.
For testing scenarios 2 and 3 (Global Throughput (Multi Site)) there are significant improvements as recorded by locust tests captured below. In scenario 2, there is a 1.5x QPS improvement (in the best case) and in scenario 3, a 2x QPS improvement (in the best case).
Interestingly, the concurrency test of the USS qualifier fails on the US deployment without activating the flag. Enabling the flag improves performance sufficiently to allow those tests to pass.
Configuration tuning
While verifying and improving the performance of the DSS with multiple benchmarks and test runs, few key characteristics have been identified to significantly impact the performance. These include, in order of importance:
Latency
Latency between datastore nodes is one of the main factors limiting performance. Due to how CockroachDB works, it creates a limit on how many queries can be performed.
We recommend limiting latency as much as possible, by keeping datastore nodes as close together as possible (e.g. on west coast datacenters / providers only).
datastore_max_open_conns
The maximum number of open connections maintained with the datastore caps concurrent query execution per client (core-service instance). A value that is too low restricts in-flight requests and amplifies client-side retry bottlenecks since connections are released between retry. The default may be increased if the cluster has available headroom but maximum parallelism shall be balanced against the risk of database exhaustion.
Time-based notification index flag
This release introduces a new flag to enable time-based notification indexes, described in interuss/dss#1541.
Using this flag improves performance in most cases, but we also observed a decrease in performance in some specific scenarios.
The performance degradation was observed in a busy S2 cell scenario (described here), when using benchmarks that try to perform as many queries as possible in a given timeframe, but these tests are known to potentially be on a invalid operational regime.
Another set of tests shows improvements in these cases: interuss/dss#1541 (comment).
There is no single best answer on whether to enable this flag, as it depends on many factors, including the number of datastore nodes, latency and traffic patterns.
We recommend testing in an environment similar to your production environment and, based on the results, enabling this flag. We do believe it improves performance in most cases.
Group subscriptions and limit implicit ones
If possible, don't rely on implicit subscriptions when creating flights, but use a dedicated one instead. This has been shown to improve performance, as it reduces the number of steps required, as explained in this comment.
Yugabyte datastore
As of now, there has been little performance work on YugabyteDB, as it was recently introduced and most of the focus has been on CockroachDB, which is still primarily used.
Based on that and on some quick performance benchmarks, we don't recommend using YugabyteDB for performance-sensitive deployments.
Notes on testing
Performance testing is a complex subject, as many variables contribute to the final results and there are different ways to measure performance.
Every case is unique, and it's hard to give a definitive answer for every parameter to get the most performance out of a DSS instance or pool. Some settings that work better in some cases won't in others.
Comparisons between tests must be made carefully, as all known variables have to be identical to allow a fair comparison between two parameters. We also recorded some unavoidable variance, due to how CockroachDB spreads data and uses indexes.
We welcome any feedback on tests performed, as well as detailed information about practical, production deployments, to help with testing and optimization in real-world cases.
Future improvements
We think we're at the limit of what we can get from CockroachDB / YugabyteDB, due to the constraints of working with an SQL interface, both on our side and on theirs. This forces us to use non-optimal solutions for our specific use cases.
To go beyond these limitations, work has started on a Raft implementation, with better performance as synchronization between nodes is limited to a single action. Initial tests and benchmarks show improved performance and full compliance with the test suite. This new implementation is expected to provide more control over the data management and simplify performance characterization.
The latest update on the implementation can be found here: #1633
Appendices
Release validation results
v0.23.0 test results archives (using monitoring v0.34.0):
v0.23.0_US_2workers_without-time-based-notif-idx.zipv0.23.0_US_2workers_with-time-based-notif-idx.zipv0.23.0_EU_10workers_without-time-based-notif-idx.zipv0.23.0_EU_8workers_with-time-based-notif-idx.zipLocust load tests on US cloud deployment for scenarios 1, 2 and 3
All reactions