Skip to content

Commit

Permalink
deprecate master nomenclature in 2.x (#319)
Browse files Browse the repository at this point in the history
Signed-off-by: Kaushal Kumar <kshkmr@amazon.com>
Signed-off-by: Sruti Parthiban <partsrut@amazon.com>
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
Signed-off-by: prudhvigodithi <pgodithi@amazon.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
Co-authored-by: Kaushal Kumar <kshkmr@amazon.com>
Co-authored-by: Sruti Parthiban <partsrut@amazon.com>
Co-authored-by: Kiran Prakash <awskiran@amazon.com>
Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com>
Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
  • Loading branch information
6 people committed Oct 31, 2022
1 parent cda3760 commit 4138436
Show file tree
Hide file tree
Showing 22 changed files with 211 additions and 185 deletions.
13 changes: 7 additions & 6 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ buildscript {
plugins {
id 'java'
id 'nebula.ospackage' version "8.3.0"
id 'com.github.spotbugs' version '5.0.0'
id 'com.github.spotbugs' version '5.0.13'
id 'jacoco'
id 'com.diffplug.spotless' version '5.11.0'
id 'checkstyle'
Expand Down Expand Up @@ -262,6 +262,7 @@ dependencies {
force "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:${jacksonVersion}"
force "com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:${jacksonVersion}"
force "org.apache.commons:commons-lang3:3.12.0"
force "org.slf4j:slf4j-api:2.0.0"
}
}

Expand Down Expand Up @@ -384,7 +385,7 @@ task cloneRcaGitRepo() {
def uri = rcaProjectRepo
def fetch = rcaProjectFetch
def branch = rcaProjectBranch
if(!branch.startsWith(fetch)) { // to avoid checking out origin/origin/branch
if (!branch.startsWith(fetch)) { // to avoid checking out origin/origin/branch
branch = rcaProjectFetch + "/" + rcaProjectBranch
}
doFirst {
Expand Down Expand Up @@ -569,17 +570,17 @@ task integTestRemote(type: RestIntegTestTask) {
systemProperty "password", System.getProperty("password")
}

String bwcVersion = "1.1.0.0"
String bwcVersion = "2.1.0.0"
String baseName = "paBwcCluster"
String bwcFilePath = "src/test/resources/org/opensearch/performanceanalyzer/bwc/"
String bwcRemoteFile = "https://ci.opensearch.org/ci/dbc/bundle-build/1.1.0/20210930/linux/x64/builds/opensearch/plugins/opensearch-performance-analyzer-1.1.0.0.zip"
String bwcOpenSearchPlugin = "opensearch-performance-analyzer-1.1.0.0.zip"
String bwcRemoteFile = "https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip"
String bwcOpenSearchPlugin = "opensearch-performance-analyzer-2.1.0.0.zip"

2.times {i ->
testClusters {
"${baseName}$i" {
testDistribution = "ARCHIVE"
versions = ["1.1.0", opensearch_version]
versions = ["2.1.0", opensearch_version]
numberOfNodes = 3
plugin(provider(new Callable<RegularFile>(){
@Override
Expand Down
4 changes: 2 additions & 2 deletions docs/READER.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The performance analyzer application is written in Java, and this allows us to s

The writer has two different policies for lifecycle management of data in shared memory

* A new file is created in shared memory for every event. These files are then deleted by the writer once their age crosses a threshold. This strategy is used for http, request and master events.
* A new file is created in shared memory for every event. These files are then deleted by the writer once their age crosses a threshold. This strategy is used for http, request and cluster_manager events.
* Samples of statistics are written to a single file, which is overwritten every 5 seconds. This strategy is used for both operating system statistics(cpu, rss) and node level OpenSearch statistics(circuit breaker).

The reader scans the shared memory directory for updates every 2.5 seconds to make sure that no update from the writer is missed. This scheme avoids explicit synchronization between the reader and writer processes.
Expand Down Expand Up @@ -101,7 +101,7 @@ On a large OpenSearch cluster with heavy workload, we often see hundreds of thou

### MetricsEmitter

The metrics emitter queries in memory snapshots of data and then bulk loads them into metricsDB. This helps us process more than 100k updates per second on a single thread. A single emitter can emit multiple metrics. We currently have four emitters - request, http, node and master. Every emitter queries an inmemory snapshot and then populates the results into the corresponding metricsDB tables. The ability to add new metrics and dimensions through configuration instead of code would be a nice enhancement.
The metrics emitter queries in memory snapshots of data and then bulk loads them into metricsDB. This helps us process more than 100k updates per second on a single thread. A single emitter can emit multiple metrics. We currently have four emitters - request, http, node and cluster_manager. Every emitter queries an inmemory snapshot and then populates the results into the corresponding metricsDB tables. The ability to add new metrics and dimensions through configuration instead of code would be a nice enhancement.

### MetricsDB

Expand Down
2 changes: 1 addition & 1 deletion licenses/performanceanalyzer-rca-2.4.0.0-SNAPSHOT.jar.sha1
Original file line number Diff line number Diff line change
@@ -1 +1 @@
8c5cb2ca38982c8d45e3dca9033d44687b9cb798
19fa3423193dc2020fe52092ab0eb66e6273651b
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Supported Elasticsearch version 7.10.0

### Features
* Publish shard state metrics ([#212](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/212))
* Add Master Throttling Collector Metrics ([#227](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/227))
* Add ClusterManager Throttling Collector Metrics ([#227](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/227))
* Publish fault detection metrics ([#218](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/218))
* Adding new GC info collector to the scheduled metrics collector([#225](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/225))

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This is the release of the Open Distro Performance Analyzer that will work with

### Improvements

* Better measurement granularity for Master Metrics [#16](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/16)
* Better measurement granularity for ClusterManager Metrics [#16](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/16)

### Bug fixes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ Version compatible with elasticsearch 6.7.1

This is the release of the Open Distro Performance Analyzer that will work with elasticsearch 6.7.1

* More Master metrics support: Master_Task_Queue_Time [#15](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/15)
* More ClusterManager metrics support: ClusterManager_Task_Queue_Time [#15](https://github.com/opendistro-for-elasticsearch/performance-analyzer/pull/15)
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Compatible with OpenSearch 1.1.0
* Use Collector override disable list for ShardIndexingPressureMetricCollector ([#28](https://github.com/opensearch-project/performance-analyzer/pull/28))
* Adding metric emission + UT for RCA_FRAMEWORK_CRASH ([#36](https://github.com/opensearch-project/performance-analyzer-rca/pull/36))
* Replace String split with Guava Splitter ([#42](https://github.com/opensearch-project/performance-analyzer-rca/pull/42))
* Add master not up writer metric ([#51](https://github.com/opensearch-project/performance-analyzer-rca/pull/51))
* Add cluster_manager not up writer metric ([#51](https://github.com/opensearch-project/performance-analyzer-rca/pull/51))

### Bug fixes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ Compatible with OpenSearch 1.3.0
* Upgrade plugin to 1.3.0 and log4j to 2.17.1 ([#118](https://github.com/opensearch-project/performance-analyzer/pull/118))
* Don't run opensearch-cli in a child process ([#126](https://github.com/opensearch-project/performance-analyzer/pull/126))
* Modify grpc-netty-shaded to grpc-netty ([#129](https://github.com/opensearch-project/performance-analyzer-rca/pull/129))
* Fixes grpc channel leak issue and vertex buffer issue on non active master ([#130](https://github.com/opensearch-project/performance-analyzer-rca/pull/130))
* Fixes RCA crash on active master ([#132](https://github.com/opensearch-project/performance-analyzer-rca/pull/132))
* Fixes grpc channel leak issue and vertex buffer issue on non active cluster_manager ([#130](https://github.com/opensearch-project/performance-analyzer-rca/pull/130))
* Fixes RCA crash on active cluster_manager ([#132](https://github.com/opensearch-project/performance-analyzer-rca/pull/132))

### Maintenance

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## Version 2.3.0 Release Notes

Compatible with OpenSearch 2.3.0


### Maintenance

* Upgrade netty version to 4.1.79 ([#249](https://github.com/opensearch-project/performance-analyzer/pull/249))
* Jackson version bump ([#247](https://github.com/opensearch-project/performance-analyzer/pull/247))
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,14 @@
import org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector;
import org.opensearch.performanceanalyzer.collectors.CircuitBreakerCollector;
import org.opensearch.performanceanalyzer.collectors.ClusterApplierServiceStatsCollector;
import org.opensearch.performanceanalyzer.collectors.ClusterManagerServiceEventMetrics;
import org.opensearch.performanceanalyzer.collectors.ClusterManagerServiceMetrics;
import org.opensearch.performanceanalyzer.collectors.ClusterManagerThrottlingMetricsCollector;
import org.opensearch.performanceanalyzer.collectors.DisksCollector;
import org.opensearch.performanceanalyzer.collectors.ElectionTermCollector;
import org.opensearch.performanceanalyzer.collectors.FaultDetectionMetricsCollector;
import org.opensearch.performanceanalyzer.collectors.GCInfoCollector;
import org.opensearch.performanceanalyzer.collectors.HeapMetricsCollector;
import org.opensearch.performanceanalyzer.collectors.MasterServiceEventMetrics;
import org.opensearch.performanceanalyzer.collectors.MasterServiceMetrics;
import org.opensearch.performanceanalyzer.collectors.MasterThrottlingMetricsCollector;
import org.opensearch.performanceanalyzer.collectors.NetworkInterfaceCollector;
import org.opensearch.performanceanalyzer.collectors.NodeDetailsCollector;
import org.opensearch.performanceanalyzer.collectors.NodeStatsAllShardsMetricsCollector;
Expand Down Expand Up @@ -207,9 +207,10 @@ public PerformanceAnalyzerPlugin(final Settings settings, final java.nio.file.Pa
new NodeStatsAllShardsMetricsCollector(performanceAnalyzerController));
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new NodeStatsFixedShardsMetricsCollector(performanceAnalyzerController));
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(new MasterServiceMetrics());
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new MasterServiceEventMetrics());
new ClusterManagerServiceMetrics());
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new ClusterManagerServiceEventMetrics());
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(new DisksCollector());
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new NetworkInterfaceCollector());
Expand All @@ -221,7 +222,7 @@ public PerformanceAnalyzerPlugin(final Settings settings, final java.nio.file.Pa
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new ShardStateCollector(performanceAnalyzerController, configOverridesWrapper));
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new MasterThrottlingMetricsCollector(
new ClusterManagerThrottlingMetricsCollector(
performanceAnalyzerController, configOverridesWrapper));
scheduledMetricCollectorsExecutor.addScheduledMetricCollector(
new ClusterApplierServiceStatsCollector(
Expand Down
Loading

0 comments on commit 4138436

Please sign in to comment.