Support for cassandra mbean(/org.apache.cassandra.net:type=FailureDetector)

[knight76]

Bug report

If I added mbean(/org.apache.cassandra.net:type=FailureDetector) in telegraf configuration, so it print out, because it only support org.apache.cassandra.metrics mbean.

2017/03/24 17:51:00 I! Unsupported Cassandra metric [/org.apache.cassandra.net:type=FailureDetector], skipping

Then, could you support org.apache.cassandra.net mbean?

Relevant telegraf.conf:

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/cassandra/cassandra.go#L270-L280

System info:

[Include Telegraf version, operating system name, and other relevant details]

Steps to reproduce:

1. ...
2. ...

Expected behavior:

Actual behavior:

Additional info:

[Include gist of relevant config, logs, etc.]

[knight76]

Also, "/org.apache.cassandra.db:type=StorageService" is needed.
The mbean provide LiveNodes and LeavingNodes of Cassandra.

[danielnelson]

In the meantime, you should be able to use the jolokia input.

[mfossen]

Is that metric supported by jolokia? I get the error E! Error in plugin [inputs.jolokia]: Not expected status value in response body (localhost:8080 mbean="org.apache.cassandra.net:type=FailureDetector" attribute="DownEndpointCount"): 404 using the below additions to my telegraf.conf. The Memory entry works as expected and jolokia is running properly and returning responses.

 [[inputs.jolokia]]
  context = "/jolokia/"

  [[inputs.jolokia.servers]]
    name = "test-1"
    host = "localhost"
    port = "8080"

  [[inputs.jolokia.metrics]]
    name = "java_memory"
    mbean = "java.lang:type=Memory"
    attribute = "HeapMemoryUsage"

  [[inputs.jolokia.metrics]]
    name = "down_nodes"
    mbean = "org.apache.cassandra.net:type=FailureDetector"
    attribute = "DownEndpointCount" 

[cabrinha]

So whats the story on this? I'm running into the same issue with org.apache.cassandra.db metrics...

[danielnelson]

I think this input should be deprecated in favor of the jolokia2 input. I would also like to add an example configuration for Cassandra with the jolokia2 input, if someone can contribute one it would be appreciated.

[cabrinha]

Here is the config I'm using, but it's not working:

[[inputs.jolokia2_agent]]
  urls = ["https://localhost:8778/jolokia"]


[[inputs.jolokia2_agent.metric]]
  name       = "jvm_garbage_collector"
  mbean      = "java.lang:name=*,type=GarbageCollector"
  tag_keys   = ["name"]

[[inputs.jolokia2_agent.metric]]
  name       = "jvm_memory"
  mbean      = "java.lang:name=*,type=Memory"
  tag_keys   = ["name"]

[[inputs.jolokia2_agent.metric]]
  name       = "cassandra_threadpool_metrics"
  mbean      = "org.apache.cassandra.metrics:name=*,type=ThreadPools"
  tag_keys   = ["name"]

I'm getting the error:

2018-03-13T02:07:00Z E! Error in plugin [inputs.jolokia2_agent]: took longer to collect than collection interval (1m0s)

[danielnelson]

How long does curl https://localhost:8778/jolokia take to run?

[cabrinha]

Sorry, it doesn't return at all. But changing from https to http does:

$ time curl http://localhost:8778/jolokia
<h1>404 Not Found</h1>No context found for request
real    0m0.026s
user    0m0.004s
sys     0m0.020s

It's worth noting I'm not running a standalone jolokia agent but jolokia is running from within the cassandra JVM itself.

Would this require using jolokia2_proxy? If so, what would the [[inputs.jolokia2_proxy.target]] url be set to?

[cabrinha]

Furthermore, why are only two metric types supported?
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/cassandra/cassandra.go#L275

I tried building the binary by hacking in another else if statement to include "/org.apache.cassandra.db:", and it seems to have worked for the metrics that return ints/floats... Any chance we could just add more?

diff --git a/plugins/inputs/cassandra/cassandra.go b/plugins/inputs/cassandra/cassandra.go
index 7dabe3b1..1a090476 100644
--- a/plugins/inputs/cassandra/cassandra.go
+++ b/plugins/inputs/cassandra/cassandra.go
@@ -73,6 +73,8 @@ func parseJmxMetricRequest(mbean string) map[string]string {
                tokens["class"] = "cassandra"
        } else if classAndPairs[0] == "java.lang" {
                tokens["class"] = "java"
+       } else if classAndPairs[0] == "org.apache.cassandra.db" {
+               tokens["class"] = "cassandra"
        } else {
                return tokens
        }
@@ -271,6 +273,9 @@ func (c *Cassandra) Gather(acc telegraf.Accumulator) error {
                        } else if strings.HasPrefix(metric,
                                "/org.apache.cassandra.metrics:") {
                                m = newCassandraMetric(serverTokens["host"], metric, acc)
+                       } else if strings.HasPrefix(metric,
+                               "/org.apache.cassandra.db:") {
+                               m = newCassandraMetric(serverTokens["host"], metric, acc)
                        } else {
                                // unsupported metric type
                                acc.AddError(fmt.Errorf("E! Unsupported Cassandra metric [%s], skipping",

[danielnelson]

The agent jar and agent jolokia is ran from within the same JVM as Cassandra, so it sounds like you have the right combination. The proxy variation is used when running the jolokia proxy as a separate process.

Once you have the jolokia agent running and that url responds, you will be able to read any of the values with the jolokia2 plugin.

[cabrinha]

When using the jolokia2 input, no errors are showing in the logs and no metrics are written to influxDB.

Running curl http://localhost:8778/jolokia/ does respond successfully, yet the collector doesn't send metrics to influxDB when specifying the URL with or without the trailing slash.

[danielnelson]

If you attach a file with the output of curl http://localhost:8778/jolokia/ and your jolokia2_agent config (if it's different from above) and I can take a look.

[cabrinha]

$ curl http://localhost:8778/jolokia/
{"timestamp":1520923822,"status":200,"request":{"type":"version"},"value":{"protocol":"7.2","config":{"maxDepth":"15","maxCollectionSize":"0","maxObjects":"0","discoveryEnabled":"true","agentContext":"\/jolokia","historyMaxEntries":"10","agentId":"xxx.xx.xx.xx-12555-431d00cf-jvm","agentType":"jvm","debug":"false","debugMaxEntries":"100"},"agent":"1.3.5","info":{}}}

cat /etc/telegraf/telegraf.d/jolokia.conf

[[inputs.jolokia2_agent]]
  urls = ["http://localhost:8778/jolokia"]

        [[inputs.jolokia2_agent.metric]]
          name       = "jvm_garbage_collector"
          mbean      = "java.lang:name=*,type=GarbageCollector"
          paths      = ["CollectionTime", "CollectionCount"]
          tag_keys   = ["name"]

        [[inputs.jolokia2_agent.metric]]
          name       = "jvm_memory"
          mbean      = "java.lang:type=Memory"
          paths      = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"]
          tag_keys = ["name"]

        [[inputs.jolokia2_agent.metric]]
          name       = "cassandra_cache_metrics"
          mbean      = "org.apache.cassandra.metrics:name=*,type=Cache"
          tag_keys   = ["name"]

        [[inputs.jolokia2_agent.metric]]
          name       = "cassandra_client_request_metrics"
          mbean      = "org.apache.cassandra.metrics:name=*,type=ClientRequest"
          tag_keys   = ["name"]

With more testing, I was able to get this working by updating the config to use [inputs.jolokia2_agent.tags] and using the same "source" for tagpass.

[danielnelson]

It looks like we actually need to figure out the POST data for a request to /jolokia/read in order to debug. I will have to figure out the details in the morning, getting late here. In the meantime, try removing tag_keys.

[danielnelson]

@Internaught Everything is working alright then?

[cabrinha]

Yeah all good here, I've dropped the cassandra plugin in favor of jolokia2_agent

[danielnelson]

Okay, when I get a chance I am planning to mark the cassandra plugin deprecated and convert the metrics in the README to an example jolokia2 config.

[danielnelson]

Cassandra input was marked deprecated in 1.7.