kamon/util/GlobPathFilter : Unsupported major.minor version 51.0

[rubinghimire]

Exception in thread "main" java.lang.UnsupportedClassVersionError: kamon/util/GlobPathFilter : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLlassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

Kamon:

io.kamon
kamon-core_2.10
0.3.1


io.kamon
kamon-datadog_2.10
0.3.1

    <SCALA_MAJOR>2.10</SCALA_MAJOR>
    <SCALA_VERSION>2.10.3</SCALA_VERSION>
    <AKKA_VERSION>2.3.2</AKKA_VERSION>

JDK 1.6

[ivantopo]

We are building for Java 1.7 by default, is it possible for you to upgrade?

[rubinghimire]

No, it's not possible. Do you recommend any other setup or tools for monitoring with Datadog?

[ivantopo]

Well, there is no particular reason why we are building for 1.7 other than being the default on our side.. I just published kamon 0.3.2-8bf93c1f26453ef298d965aaa8b9cb932f8b2a64 in our snapshots repository (http://snapshots.kamon.io) in case you want to give it a try, it was built with JDK 1.6 and should work fine for you.. we will make sure that our next releases are 1.6 compatible too, thanks for reporting!

[rubinghimire]

Thanks a lot! I used this info: http://kamon.io/backends/datadog/ and I have a datadog agent running on my machine. Do you have a sample of the datadog.conf settings? I can't find anything with "kamon" in the metric explorer.

[ivantopo]

did you include the Datadog extension in your akka configuration as show bellow?

akka {
  extensions = ["kamon.datadog.Datadog"]
}

[rubinghimire]

Yes, I have. Here's my kamon config:

kamon {
datadog {
# Hostname and port in which your Datadog is running. Remember that Datadog packets are sent using UDP and
# setting unreachable hosts and/or not open ports wont be warned by the Kamon, your data wont go anywhere.
hostname = "127.0.0.1"
port = 8125

# Interval between metrics data flushes to Datadog. It's value must be equal or greater than the
# kamon.metrics.tick-interval setting.
flush-interval = 1 second

# Max packet size for UDP metrics data sent to Datadog.
max-packet-size = 1024 bytes

# Subscription patterns used to select which metrics will be pushed to Datadog. Note that first, metrics
# collection for your desired entities must be activated under the kamon.metrics.filters settings.
includes {
  actor      =  [ "*" ]
  trace      =  [ "*" ]
  dispatcher =  [ "*" ]
}

# Application prefix for all metrics pushed to Datadog. The default namespacing scheme for metrics follows
# this pattern:
#    application.entity-name.metric-name
application-name = "kamon"

}

Default dispatcher for all Kamon components, unless a more specific one is configured.

default-dispatcher = "akka.actor.default-dispatcher"

metrics {
tick-interval = 1 second

# Time interval for recording values on all registered gauges.
gauge-recording-interval = 100 milliseconds


dispatchers {

  # Dispatcher for periodical gauge value recordings.
  gauge-recordings = ${kamon.default-dispatcher}

  # Dispatcher for subscriptions and metrics collection actors.
  metric-subscriptions = ${kamon.default-dispatcher}
}

filters = [
  {
    actor {
      includes = [ "user/batch-statement-generation/*"]
      excludes = []
    }
  },
  {
    trace {
        includes = [ "*" ]
        excludes = []
    }
  },
  {
    dispatcher {
        includes = [ "default-dispatcher" ]
        excludes = []
    }
  }
]

precision {
  actor {
    processing-time {
      highest-trackable-value = 3600000000000
      significant-value-digits = 2
    }
    time-in-mailbox {
      highest-trackable-value = 3600000000000
      significant-value-digits = 2
    }
    mailbox-size {
      highest-trackable-value = 999999999
      significant-value-digits = 2
    }
  }

 trace {
    elapsed-time {
      highest-trackable-value = 3600000000000
      significant-value-digits = 2
    }
    segment {
      highest-trackable-value = 3600000000000
      significant-value-digits = 2
    }
  }

  dispatcher {
    maximum-pool-size {
      highest-trackable-value = 999999999
      significant-value-digits = 2
    }
    running-thread-count {
      highest-trackable-value = 999999999
      significant-value-digits = 2
    }
    queued-task-count {
      highest-trackable-value = 999999999
      significant-value-digits = 2
    }
    pool-size {
      highest-trackable-value = 999999999
      significant-value-digits = 2
    }
  }
}

}
trace {
# If ask-pattern-tracing is enabled, a WARN level log message will be generated if a future generated by the ask
# pattern fails with a AskTimeoutException and the log message will contain a stack trace captured at the moment
# the future was created.
ask-pattern-tracing = off
}

weaver {
# AspectJ options supported by LTW
# showWeaveInfo: show informational messages whenever the weaver touches a class file.
# verbose: show informational messages about the weaving process.
# debug: show a messages for each class passed to the weaver indicating whether it was woven, excluded or ignored.
# showWarn: show warning messages about the weaving process.

    showWeaveInfo = off
    verbose = off
    debug = off
    showWarn = off

}
}

What is the minimum configuration that I need to have for datadog.conf?
I have one host dashboard in Datadog - shows only my system metrics (cpu usage, memory, etc)

[ivantopo]

The minimum would be not providing any custom configuration for kamon-datadog. The default values assume that your datadog agent is running locally on port 8125, so it should work just by adding this to your configuration file:

akka {
  extensions = ["kamon.datadog.Datadog"]
}

I can't find that section on your message, are you really sure it is there? Then, you should modify the filters accordingly to your needs.

[rubinghimire]

Yes, this is my akka config:

Akka settings

akka {
loglevel = "DEBUG"
actor {
debug {
autoreceive = on
lifecycle = off
receive = off
}
}

extensions = ["kamon.datadog.Datadog"]

log-dead-letters-during-shutdown = on

}

I have the following set for datadog.conf
dd_url: https://app.datadoghq.com
api-key:

Do I need this set?
listen_port: 8125

[rubinghimire]

It's an Akka standalone application - I'm assuming it would work.
Here's the datadog.conf for the agent
https://github.com/DataDog/dd-agent/blob/master/datadog.conf.example

[ivantopo]

the port that you care about in the datadog agent configuration is the dogstatsd_port which by default is 8125 and should match the one configured by default in kamon-datadog. During the application startup you should see a "Starting the Kamon(Datadog) extension" message. Are you seeing that?

Also, are you starting your application with the -javaagent:/path-to-aspectj-weaver.jar option?

[rubinghimire]

Yes, I did so and here's the Starting Kamon(Datadog) msg:

[INFO] [07/03/2014 12:29:21.148] [main] [DatadogExtension(akka://batch-statements)] Starting the Kamon(Datadog) extension
Jul 3, 2014 12:29:21 PM kamon.weaver.logging.KamonWeaverMessageHandler showMessage
INFO: weaving 'kamon.datadog.DatadogExtension$$anonfun$buildMetricsListener$1'

All I have set in datadog.conf are:
dd_url: https://app.datadoghq.com
api-key:

Here's what I have in the metric explorer in datadoghq

[ivantopo]

If you do a netstat -an | grep 8125 you see the port 8125 listening for UDP traffic?, what datadog agent version do you have? What's the output of running your-datadog-agent-folder/bin/info?

[rubinghimire]

Yes, it's listed for UDP. The agent version is 4.4.0. The datadog agent doesn't have a bin folder. Here's the installation folder:

[ivantopo]

and, what do the statsd.bad_lines_seen and statsd.packets_received metrics show in your datadog dashboard?

[rubinghimire]

They aren't there anymore with this configuration. I'm assuming they showed up while I was tweaking all kinds of configuration on datadog.conf

Recently reported metrics:

[rubinghimire]

dogstatsd.packet.count graph shows zero.

[ivantopo]

I see that you are running windows, maybe there is some weird thing going on with the windows firewall that might be blocking those packets?

[rubinghimire]

My firewall is off. Should there be a "kamon" app listed here under "Apps" or in metric explorer (even though the packets are not sent)?

Here's the datadog agent:

It's not receiving any metrics?

[ivantopo]

Surely the agent is not receiving any metrics from Kamon... What about sending a metric packet manually to DatadogStatsD and make sure it is working?, if you use this:

echo "sample-latency:190|ms" | nc -u -w0 127.0.0.1 8125

you should be able to see that metric on your dashboard shortly after.

[ivantopo]

Hello there! I was wondering if you were able to manually post the the metrics to your datadog agent?, also I did a more detailed review on the reasons of the bug that initially started this issue and found that we needed to pass a couple more options to the Java compiler for that specific class, that has been solved by c6b8918.

[rubinghimire]

Hi, I was able to see that the packet was received in datadog.dogstatsd.packet.count graph on the dashboard. However, I don't see a metric listed as "sample-latency".

While running the application with the SNAPSHOTS from earlier, I don't see any packets on the graph (count=0).

I tried that build c6b8918 but no packets on the graph. Also, no new metrics.

[ivantopo]

@labrg I'm running out of ideas now, if you are manually posting a metric to the agent and it doesn't show it on your datadog dashboard we have reached a point where there is very little I can do to help, it might be that something is not configured properly in your agent but probably the datadog support guys can do a lot more for you than I.. could you please contact them and make sure that when you report a metric it gets to Datadog and then come back to use to troubleshoot what might be wrong with Kamon?, I'm really sorry for not being able to do more for you at this point.

[rubinghimire]

@ivantopo Thanks for all your feedback. I got in touch with datadog - according to them, a bug affects their windows agent. Here's the change they recommended (adding a pipe at the end of the packet)
sample-latency:190|ms|

This does show the metric on the dashboard.

Is there anything that can be done with Kamon to work with this bug or should I talk to datadog about it?

[ivantopo]

Did they mention if reporting metrics with that format will work properly on Linux/OSX?

[ivantopo]

Also, please try sending a packet like the one bellow and tell me if it is accepting it correctly:

kamon.actor.processing-time:10|ms|@0.5|#actor:user/kamon

[rubinghimire]

I've asked their support if it would work on Linux/OSX. I'll let you know as soon as I hear from them.
Also, that packet was accepted over tag actor:user/kamon:

[ivantopo]

and what about this one:

kamon.actor.time-in-mailbox:20|ms|#actor:user/kamon

[rubinghimire]

Yes, that works as well.

[ivantopo]

well, those are the two ways in which Kamon send metrics to datadog, maybe there is a problem with multiple metrics in the same packet? Tell me if something like this works:

kamon.actor.some-metric:20|ms|#actor:user/kamon\nkamon.actor.some-metric:30|ms|#actor:user/kamon

[rubinghimire]

@ivantopo Yeah, it works with bash. (echo -e ...)

[ivantopo]

@labrg I just opened #52 to track the compatibility issues between kamon-datadog and the Windows Datadog agent, let's track that problem there and close this one which original motivation has been already solved, thanks.