Marathon input plugin #2369

smolse · 2017-02-06T01:01:58Z

Added an input plugin for gathering metrics from Marathon using REST API provided by it.

Required for all PRs:

CHANGELOG.md updated (we recommend not updating this until the PR has been approved by a maintainer)
Sign CLA (if not already signed)
README.md updated (if adding a new plugin)

phemmer · 2017-02-06T13:27:00Z

plugins/inputs/marathon/marathon.go

+		wg.Add(1)
+		go func(c string) {
+			defer wg.Done()
+			errorChannel <- m.gatherMetrics(c, ":8080", acc)


You can simplify the code, and make the errors logs easier to read by doing acc.AddError(...).

Hi @phemmer, thanks for valuable comments. I am not very experienced in Go and Telegraf yet, so tried to implement the plugin as close to existing plugins as possible. Currently used approach seems to be the most prevalent among the other plugins.

acc.AddError() is relatively new. It was added because there was a lot of inconsistency among the error handling within plugins, and many of them had problems.
For example, with joining on newline, the problem is that only the first line will contain the timestamp and plugin name, the other lines will not. This can make the logs harder to read.

I really should go through and clean up all the plugins to fix all the error handling now.

Got it, thanks for the intro. I will update the PR to use acc.AddError() for error handling.

phemmer · 2017-02-06T13:29:28Z

plugins/inputs/marathon/marathon.go

+	return false
+}
+
+func (m *Marathon) filterMetrics(metrics *map[string]interface{}) {


Why is this receiving a pointer to a map? Maps are already a pointer.

True. Fixed.

phemmer · 2017-02-06T13:34:27Z

plugins/inputs/marathon/marathon.go

+		if contains(m.MetricTypes, k) == false {
+			delete(*metrics, k)
+		}
+	}


``` for _, mt := range m.MetricTypes { delete(*metrics, mt) } ``` Edit: Nevermind, I misread the code.

vladshub · 2017-03-14T19:30:59Z

Any updates on this?

smolse · 2017-03-15T08:46:01Z

From my point of view it is ready to be merged.

deric · 2017-05-10T09:51:52Z

Any progress on this?

juggie · 2017-05-15T01:11:47Z

+1 This would be awesome.

danielnelson · 2017-06-09T00:48:31Z

The current metrics, while matching what Marathon provides, don't fit with the existing metrics produced by Telegraf. We need to think about how we can leverage fields and tags to reduce the number of points and better allow the use of functions when we query.

@smolse Do you think you can take a pass at this?

nmische · 2017-07-25T18:55:49Z

+1 We would really like to see this plugin in telegraf.

danielnelson · 2017-07-25T19:00:54Z

Maybe we can do something similar to the graphite input to transform the metrics to be more in line with our data model. If anyone begins work on this please add a comment here.

smolse · 2017-07-27T12:41:23Z

@danielnelson I have updated the pull request to store each metric in a separate measurement, how does it look now?

danielnelson · 2017-07-28T02:15:29Z

This is a step in the right direction, but there is still more to do. You may want to read through the schema and data layout documentation first.

I'll try to give a few examples to illustrate some changes that would be desirable. Here is one of the current points, in line protocol format minus the timestamp:

jvm_memory_total_committed value=42
jvm_memory_total_init value=1
jvm_memory_total_max value=100
jvm_memory_total_used value=20

With this model you can't easily write, for instance, a query to calculate the percentage of used memory. This is because each value is part of a different measurement so you cannot use more than one in an InfluxQL function.

Another issue is that the measurement name won't work well with other inputs. If you are collecting from both Marathon and Jolokia, there would be confusion about which service a JVM measurement is from. Usually an input plugin has no more than a handful of measurement names, and I recommend having them all start with marathon, perhaps for this one it would be marathon_jvm_memory?

So after these two transformations we would have all the values as fields with the same measurement name:

marathon_jvm_memory total_committed=42,total_init=1,total_max=100,total_used=20

That is better but some of the metrics can be improved with good use of tags. These two measurement names should make for a good example:

service.mesosphere.chaos.http.ServiceStatusServlet.init
service.mesosphere.marathon.core.task.update.impl.ThrottlingTaskStatusUpdateProcessor.processing
service.mesosphere.marathon.core.task.update.impl.ThrottlingTaskStatusUpdateProcessor.queued

When a measurement has a deeply nested structure like this, usually it contains multiple independent dimensions. Having them as a single item makes it difficult to group by and slow to select subsets of.

I think here it might make sense to have two tags, one for marathon|chaos, and one for the resource. You can probably decompose and name the tags better than I can since you are familiar with Marathon, so don't necessarily use my exact suggestions:

marathon,service=chaos,resource=http.ServiceStatusServlet count=1i,max=1i
marathon,service=marathon,resource=core.task.update.impl.ThrottlingTaskStatusUpdateProcessor processing=1i,queued=2i

One thing that is tricky about this plugin is the large number of metrics. This necessitates determining the measurement name and tags programmatically, which can be difficult if there are any inconsistencies.

danielnelson · 2017-07-31T23:11:31Z

One thing I should add to my clarify my last few comments. If the format of the marathon stats are inconsistent, it may be impossible to extract tags as I mentioned above automatically. This is where my comment about the graphite parser comes in. It has a template variable which can be used to unpack metrics, perhaps it can be useful. Here is a config file example:

templates = [
    "*.*.* region.region.measurement", # <- all 3-part measurements will match this one.
    "*.*.*.* region.region.host.measurement", # <- all 4-part measurements will match this one.
]

jcmartins · 2017-09-19T17:14:33Z

+1 I really need this plugin in my telegraf.

danielnelson · 2017-10-03T19:37:30Z

Let's use the dropwizard parser once it is complete to deal with the metric tranformations in this plugin.

jcmartins · 2017-11-14T19:13:26Z

Do you have plans to implement Marathon service discovery ?

danielnelson · 2017-11-15T21:27:16Z

No plans specifically with Marathon, though you might want to read through #272. We will be adding a plugin system for configuration and we could potentially write a plugin that creates input plugins based on what is discovered in Marathon.

danielnelson · 2018-01-09T23:07:09Z

@smolse: We have merged @atzoum's dropwizard parser on master, we should update this plugin to use it internally for handling the incoming metrics.

russorat · 2018-04-09T17:53:50Z

@smolse i know this has been open for a while, but are you interested in bringing this across the finish line or should we look for someone to take it over?

smolse · 2018-04-10T10:43:32Z

Unfortunately most likely I will not have time to work on this in the nearest future, please proceed with someone else.

sjwang90 · 2020-11-11T00:40:06Z

If anyone is interested in taking this over from @smolse please go for it!
We can probably get this through even faster as an external plugin that can be used with execd to run seamlessly with Telegraf. Telegraf users would be able to use the plugin sooner by having it as an external plugin since it wouldn't have to go through the typical review process.

Feel free to respond with any questions you may have.

sjwang90 · 2021-02-03T16:48:00Z

Closing this PR due to inactivity.

If you would still like this plugin, please submit it as an external plugin that can be used with execd to run seamlessly with Telegraf.

sparrc added this to the Future Milestone milestone Feb 6, 2017

sparrc added the plugin request label Feb 6, 2017

phemmer reviewed Feb 6, 2017

View reviewed changes

smolse force-pushed the marathon-input-plugin branch from add7784 to d6f5b43 Compare February 6, 2017 18:13

danielnelson modified the milestones: 1.4.0, Future Milestone Jun 1, 2017

Sergei Smolianinov added 4 commits July 27, 2017 15:23

Adds marathon input plugin

8b81809

fix: filterMetrics should receive a map, not a pointer to map

ccfe127

use acc.AddError() for error handling

329af31

Store each Marathon metric in dedicated measurement

850598c

smolse force-pushed the marathon-input-plugin branch from b75f79a to 850598c Compare July 27, 2017 12:23

danielnelson added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin and removed feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin request labels Aug 12, 2017

danielnelson modified the milestones: 1.4.0, 1.5.0 Aug 14, 2017

danielnelson added the new plugin label Sep 18, 2017

This was referenced Oct 3, 2017

init jenkins input plugin #3292

Closed

Add support for dropwizard format #2846

Merged

danielnelson removed this from the 1.5.0 milestone Nov 29, 2017

sjwang90 added the help wanted Request for community participation, code, contribution label Nov 11, 2020

sjwang90 removed the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Nov 18, 2020

sjwang90 added the plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins label Jan 26, 2021

sjwang90 closed this Feb 3, 2021

sjwang90 added closed/external-candidate external plugin Plugins that would be ideal external plugin and expedite being able to use plugin w/ Telegraf labels Feb 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Marathon input plugin #2369

Marathon input plugin #2369

smolse commented Feb 6, 2017 •

edited

Loading

phemmer Feb 6, 2017

smolse Feb 6, 2017

phemmer Feb 6, 2017

smolse Feb 6, 2017

phemmer Feb 6, 2017 •

edited

Loading

smolse Feb 6, 2017

phemmer Feb 6, 2017 •

edited

Loading

vladshub commented Mar 14, 2017

smolse commented Mar 15, 2017

deric commented May 10, 2017

juggie commented May 15, 2017

danielnelson commented Jun 9, 2017

nmische commented Jul 25, 2017

danielnelson commented Jul 25, 2017

smolse commented Jul 27, 2017 •

edited

Loading

danielnelson commented Jul 28, 2017

danielnelson commented Jul 31, 2017

jcmartins commented Sep 19, 2017

danielnelson commented Oct 3, 2017

jcmartins commented Nov 14, 2017

danielnelson commented Nov 15, 2017

danielnelson commented Jan 9, 2018

russorat commented Apr 9, 2018

smolse commented Apr 10, 2018

sjwang90 commented Nov 11, 2020

sjwang90 commented Feb 3, 2021

Marathon input plugin #2369

Marathon input plugin #2369

Conversation

smolse commented Feb 6, 2017 • edited Loading

Required for all PRs:

phemmer Feb 6, 2017

Choose a reason for hiding this comment

smolse Feb 6, 2017

Choose a reason for hiding this comment

phemmer Feb 6, 2017

Choose a reason for hiding this comment

smolse Feb 6, 2017

Choose a reason for hiding this comment

phemmer Feb 6, 2017 • edited Loading

Choose a reason for hiding this comment

smolse Feb 6, 2017

Choose a reason for hiding this comment

phemmer Feb 6, 2017 • edited Loading

Choose a reason for hiding this comment

vladshub commented Mar 14, 2017

smolse commented Mar 15, 2017

deric commented May 10, 2017

juggie commented May 15, 2017

danielnelson commented Jun 9, 2017

nmische commented Jul 25, 2017

danielnelson commented Jul 25, 2017

smolse commented Jul 27, 2017 • edited Loading

danielnelson commented Jul 28, 2017

danielnelson commented Jul 31, 2017

jcmartins commented Sep 19, 2017

danielnelson commented Oct 3, 2017

jcmartins commented Nov 14, 2017

danielnelson commented Nov 15, 2017

danielnelson commented Jan 9, 2018

russorat commented Apr 9, 2018

smolse commented Apr 10, 2018

sjwang90 commented Nov 11, 2020

sjwang90 commented Feb 3, 2021

smolse commented Feb 6, 2017 •

edited

Loading

phemmer Feb 6, 2017 •

edited

Loading

phemmer Feb 6, 2017 •

edited

Loading

smolse commented Jul 27, 2017 •

edited

Loading