Document ccr-stats telemetry device #676

dliappis · 2019-04-09T13:53:28Z

Also use fields prefixed by shard. for the values returned by
_ccr/stats and move "name:ccr-stats" to be at the top level of the
document (used to be "meta.name:ccr-stats").

Also use fields prefixed by `shard.` for the values returned by `_ccr/stats` and move "name:ccr-stats" to be at the top level of the document (used to be "meta.name:ccr-stats").

dliappis · 2019-04-09T13:54:47Z

An easy way to test this (I've used this workflow) is to place these scripts: https://gist.github.com/dliappis/c4c81d10b846d0b023963b8230a34020 in an empty directory and run ./start.sh. Kibana will run on the standard http://localhost:5601 .

danielmitterdorfer

Looks mostly fine but now that this is a documented telemetry device I think it should also show up in esrally list telemetry. Furthermore, I think a small example of how to use it would be great given that it requires multiple clusters and the setup is quite complex?

danielmitterdorfer · 2019-04-10T08:51:42Z

esrally/mechanic/telemetry.py

+                doc = {
+                    "name": "ccr-stats",
+                    "shard": shard_stats
+                }
                shard_metadata = {
                    "cluster": self.cluster_name,
                    "index": name,


This seems to be included in the shard stats already. Does it make sense to duplicate this information?

The shard metrics, as reported by the CCR stats endpoint are associated with each following index and I wanted to retain the structure in this respect.

For example in the following excerpt, we can see this duplication, a key index and then per shard the more specific leader-index / follower_index keys. I feel that for the sake of maintaining this structure it makes sense to keep this duplication.

"indices" : [ { "index" : "geonames-copy", "shards" : [ { "remote_cluster" : "leader", "leader_index" : "geonames", "follower_index" : "geonames-copy", "shard_id" : 0, "leader_global_checkpoint" : -1, "leader_max_seq_no" : -1, "follower_global_checkpoint" : -1, "follower_max_seq_no" : -1, "last_requested_seq_no" : -1, "outstanding_read_requests" : 1, "outstanding_write_requests" : 0, "write_buffer_operation_count" : 0, "write_buffer_size_in_bytes" : 0, "follower_mapping_version" : 1, "follower_settings_version" : 1, "total_read_time_millis" : 0, "total_read_remote_exec_time_millis" : 0, "successful_read_requests" : 0, "failed_read_requests" : 0, "operations_read" : 0, "bytes_read" : 0, "total_write_time_millis" : 0, "successful_write_requests" : 0, "failed_write_requests" : 0, "operations_written" : 0, "read_exceptions" : [ ], "time_since_last_read_millis" : 4004 }, { "remote_cluster" : "leader", "leader_index" : "geonames", "follower_index" : "geonames-copy", "shard_id" : 1, "leader_global_checkpoint" : -1, "leader_max_seq_no" : -1, "follower_global_checkpoint" : -1, "follower_max_seq_no" : -1, "last_requested_seq_no" : -1, "outstanding_read_requests" : 1, "outstanding_write_requests" : 0, "write_buffer_operation_count" : 0, "write_buffer_size_in_bytes" : 0, "follower_mapping_version" : 1, "follower_settings_version" : 1, "total_read_time_millis" : 0, "total_read_remote_exec_time_millis" : 0, "successful_read_requests" : 0, "failed_read_requests" : 0, "operations_read" : 0, "bytes_read" : 0, "total_write_time_millis" : 0, "successful_write_requests" : 0, "failed_write_requests" : 0, "operations_written" : 0, "read_exceptions" : [ ], "time_since_last_read_millis" : 4087 },

Sure, I'm fine with that.

danielmitterdorfer · 2019-04-10T08:51:52Z

esrally/mechanic/telemetry.py

                shard_metadata = {
                    "cluster": self.cluster_name,
                    "index": name,
-                    "shard": shard_stats["shard_id"],
-                    "name": "ccr-stats"
+                    "shard": shard_stats["shard_id"]


This is included in the shard stats already. Does it make sense to duplicate this information?

Whereas here I think it doesn't make sense to duplicate the shard_id; I'll remove.

danielmitterdorfer · 2019-04-11T10:44:07Z

esrally/mechanic/telemetry.py

@@ -31,7 +31,8 @@

 def list_telemetry():
    console.println("Available telemetry devices:\n")
-    devices = [[device.command, device.human_name, device.help] for device in [JitCompiler, Gc, FlightRecorder, PerfStat, NodeStats, RecoveryStats]]
+    devices = [[device.command, device.human_name, device.help] for device in [JitCompiler, Gc, FlightRecorder, PerfStat,


Should we also update the docs where we show an example of the listed telemetry devices?

dliappis · 2019-04-11T17:22:53Z

@danielmitterdorfer Could you please take another pass esp now that I've added the ccr recipe in e592f17 and the corresponding documentation in 5ca587a?

danielmitterdorfer

I left some, mostly minor, suggestions.

danielmitterdorfer · 2019-04-12T07:10:12Z

docs/recipes.rst

+
+.. _recipe_testing_rally_against_ccr_clusters:
+
+Testing Rally against CCR Clusters using a remote metric store


Nit: clusters (lower case)

Fixed in 7202a00

danielmitterdorfer · 2019-04-12T07:13:10Z

docs/recipes.rst

+
+1. Starts a small single node Elasticsearch cluster locally, to serve as a :ref:`metrics store <configuration_options>`. It also starts Kibana attached to the Elasticsearch metric store cluster.
+2. Creates a new configuration file for Rally under ``~/.rally/rally-metricstore.ini`` referencing Elasticsearch from step 1.
+3. Starts two additional small local Elasticsearch clusters with 1 node each, (version ``7.0.0`` by default) called ``leader`` and ``follower`` listening at ports 329201 and 32902 respectively.


329201 -> 39201

Fixed in 7202a00

danielmitterdorfer · 2019-04-12T07:14:03Z

docs/recipes.rst

+
+Rally will push metrics to the metric store configured in 1. and they can be visualized by accessing Kibana at `http://locahost:5601 <http://localhost:5601>`_.
+
+To tear down everything just issue ``./stop.sh``.


Remove "just"

danielmitterdorfer · 2019-04-12T07:17:09Z

docs/recipes.rst

+
+Testing Rally features (such as the ``ccr-stats`` telemetry device) requiring Elasticsearch clusters configured for `cross-cluster replication <https://www.elastic.co/guide/en/elastic-stack-overview/current/ccr-getting-started.html>`_ can be a time consuming process.
+
+For this reason we've created an example under `recipes/ccr in Rally's repository <https://github.com/elastic/rally/tree/master/recipes/ccr>`_ simplifying it.


How about something along these lines?

Use `recipes/ccr in Rally's repository <https://github.com/elastic/rally/tree/master/recipes/ccr>`_ to test a simple but complete example.

(I'd also move this to the previous paragraph)

So no need to provide links to cross-cluster-replication from the Elastic website i.e. get rid of the entire line 158 completely, right?

No. I'd just turn this into a single paragraph. I don't think that we should have two paragraphs here. So line 158 would stay.

Thanks, fixed in 7202a00

danielmitterdorfer · 2019-04-12T07:18:44Z

docs/recipes.rst

+
+Running the ``start.sh`` script requires Docker locally installed and performs the following actions:
+
+1. Starts a small single node Elasticsearch cluster locally, to serve as a :ref:`metrics store <configuration_options>`. It also starts Kibana attached to the Elasticsearch metric store cluster.


Should we maybe state what "small" means? (i.e. 512MB heap for the metrics store and 1 GB heap for the regular nodes?)

Fixed in 7202a00

recipes/ccr/start.sh

danielmitterdorfer

Thanks for iterating! LGTM

Document ccr-stats telemetry device

55ca484

Also use fields prefixed by `shard.` for the values returned by `_ccr/stats` and move "name:ccr-stats" to be at the top level of the document (used to be "meta.name:ccr-stats").

dliappis added :Docs Changes to the documentation :Telemetry Telemetry Devices that gather additional metrics labels Apr 9, 2019

dliappis added this to the 1.1.0 milestone Apr 9, 2019

dliappis requested a review from danielmitterdorfer April 9, 2019 13:53

danielmitterdorfer reviewed Apr 10, 2019

View reviewed changes

Also add ccr-stats in output of esrally list telemetry

fe1ca3e

danielmitterdorfer reviewed Apr 11, 2019

View reviewed changes

dliappis added 4 commits April 11, 2019 18:05

Also update docs

0cd786a

Address PR comment

1c43936

Add CCR recipe in documentation

5ca587a

Commit ccr recipe

e592f17

dliappis requested a review from danielmitterdorfer April 12, 2019 07:03

danielmitterdorfer reviewed Apr 12, 2019

View reviewed changes

Address PR comments

7202a00

danielmitterdorfer approved these changes Apr 12, 2019

View reviewed changes

dliappis merged commit 7d5ac72 into elastic:master Apr 12, 2019

dliappis deleted the document-ccr-and-make-metric-doc-name-consistent branch April 12, 2019 07:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document ccr-stats telemetry device #676

Document ccr-stats telemetry device #676

dliappis commented Apr 9, 2019

dliappis commented Apr 9, 2019

danielmitterdorfer left a comment •

edited

danielmitterdorfer Apr 10, 2019

dliappis Apr 10, 2019

danielmitterdorfer Apr 10, 2019

danielmitterdorfer Apr 10, 2019

dliappis Apr 10, 2019

danielmitterdorfer Apr 11, 2019

dliappis commented Apr 11, 2019

danielmitterdorfer left a comment

danielmitterdorfer Apr 12, 2019

dliappis Apr 12, 2019

danielmitterdorfer Apr 12, 2019

dliappis Apr 12, 2019

danielmitterdorfer Apr 12, 2019

danielmitterdorfer Apr 12, 2019

dliappis Apr 12, 2019

danielmitterdorfer Apr 12, 2019

dliappis Apr 12, 2019

danielmitterdorfer Apr 12, 2019

dliappis Apr 12, 2019

danielmitterdorfer left a comment


		.. _recipe_testing_rally_against_ccr_clusters:

		Testing Rally against CCR Clusters using a remote metric store


		Rally will push metrics to the metric store configured in 1. and they can be visualized by accessing Kibana at `http://locahost:5601 <http://localhost:5601>`_.

		To tear down everything just issue ``./stop.sh``.


		Testing Rally features (such as the ``ccr-stats`` telemetry device) requiring Elasticsearch clusters configured for `cross-cluster replication <https://www.elastic.co/guide/en/elastic-stack-overview/current/ccr-getting-started.html>`_ can be a time consuming process.

		For this reason we've created an example under `recipes/ccr in Rally's repository <https://github.com/elastic/rally/tree/master/recipes/ccr>`_ simplifying it.


		Running the ``start.sh`` script requires Docker locally installed and performs the following actions:

		1. Starts a small single node Elasticsearch cluster locally, to serve as a :ref:`metrics store <configuration_options>`. It also starts Kibana attached to the Elasticsearch metric store cluster.

Document ccr-stats telemetry device #676

Document ccr-stats telemetry device #676

Conversation

dliappis commented Apr 9, 2019

dliappis commented Apr 9, 2019

danielmitterdorfer left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dliappis commented Apr 11, 2019

danielmitterdorfer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielmitterdorfer left a comment

Choose a reason for hiding this comment

danielmitterdorfer left a comment •

edited