Add Prometheus remote write output plugin. by tomwilkie · Pull Request #5003 · influxdata/telegraf

tomwilkie · 2018-11-18T21:15:13Z

Add a output plugin that allows Telegraf to send metrics to systems which support Prometheus remote write format (such as Cortex).

Doesn't handle histograms or summaries right now. Tested and it works.

Signed-off-by: Tom Wilkie tom.wilkie@gmail.com

Required for all PRs:

Signed CLA.
Associated README.md updated.
Has appropriate unit tests.

danielnelson · 2018-11-19T21:34:57Z

Thanks for the pull request, this sounds useful. This would work with any of these integrations that are marked as write?

tomwilkie · 2018-11-19T21:37:30Z

@danielnelson thats right!

tomwilkie · 2018-12-05T16:21:02Z

@danielnelson anything I can do to help this along?

danielnelson · 2018-12-17T21:06:20Z

+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode/100 != 2 {


I would prefer matching the exact status codes or range, to avoid any potential issue where an error can be missed.

The protocol doesn't specify the exact code, and I know some implementation return 200 and 204 (no content).

danielnelson · 2018-12-17T21:52:23Z

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";


This seems like it copied from the prometheus library, is there a package would we need to import to avoid duplicating this file? I would only want to define this again here if it would be a hit on binary size.

The biggest problem is Prometheus imports quite a few projects (for service discovery), including kubernetes, so eventually we end up importing the world and conflicting on versions. I gave it a go and it wasn't as bad as I expected.

danielnelson · 2018-12-17T21:55:57Z

+			copy(labels, commonLabels)
+			labels = append(labels, &Label{
+				Name:  "__name__",
+				Value: metric.Name() + "_" + field.Key,


We may wish to mirror this code to special case a few input formats, this is useful for round tripping data collected by the prometheus input.

telegraf/plugins/outputs/prometheus_client/prometheus_client.go

Lines 472 to 488 in 1334919

switch point.Type() {

case telegraf.Counter:

if fn == "counter" {

mname = sanitize(point.Name())

}

case telegraf.Gauge:

if fn == "gauge" {

mname = sanitize(point.Name())

}

}

if mname == "" {

if fn == "value" {

mname = sanitize(point.Name())

} else {

mname = sanitize(fmt.Sprintf("%s_%s", point.Name(), fn))

}

}

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Required pinning / updating a couple of dependancies, but wasn't as bad as I first throught. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie · 2019-02-01T15:51:47Z

Most of the feedback done; I'd like to refactor and dedupe some of the code with the existing prometheus output format first though, and add support for histograms. WDYT so far?

danielnelson

Looks good, feel free to refactor the prometheus output code so long as the output format isn't changed. You can create a package internal/prometheus if you need a place for shared code.

danielnelson · 2019-02-01T21:52:25Z

  version = "0.2.3"

-[[constraint]]
-  name = "github.com/golang/protobuf"


This is a direct dependency, as is genproto, so they should both stay in the Gopkg.toml. I think perhaps you ran into grpc-ecosystem/grpc-gateway#829, if so we can use the override to workaround the issue:

[[constraint]] name = "github.com/golang/protobuf" version = "1.1.0" version = "1.2.0" [[override]] name = "github.com/golang/protobuf" version = "1.2.0"

danielnelson · 2019-02-04T23:37:55Z

@tomwilkie Just a heads up in case you need this to go out with the next release, we are hoping to do an release candidate for 1.10 on the 12th.

…mp order mismatch 1. Add bearer Token support 2. Sanitize the metrics to follow prometheus format i.e having underscrore. 3. Sorted returns a copy of the metrics in time ascending order. A copy is made to avoid modifying the input metric slice since doing so causing issue of timestamp order mismatch.

Add support of Bearer Token and handle timestamp order mismatch issue

achilles42 · 2019-05-26T16:45:12Z

@tomwilkie @danielnelson What else is pending on this PR? I am happy to work on this if there any work required.

kushwiz · 2019-10-19T00:34:08Z

Please let me know if i can help :)
This is a missing piece to a scalable monitoring solution.

danielnelson · 2019-11-22T01:45:09Z

Continued on #5942

tomwilkie changed the title ~~[WIP] Add Prometheus remote write output plugin.~~ Add Prometheus remote write output plugin. Nov 19, 2018

danielnelson added area/prometheus new plugin labels Nov 19, 2018

glinton self-requested a review December 6, 2018 21:00

danielnelson added this to the 1.10.0 milestone Dec 17, 2018

danielnelson suggested changes Dec 17, 2018

View reviewed changes

tomwilkie added 4 commits February 1, 2019 15:46

Add Prometheus remote write output plugin.

9eacd0c

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Add a test and a basic README.

50664b8

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Add basic auth, test.

4ecd215

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback (part I)

35b28de

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

tomwilkie force-pushed the prometheus-remote-write branch from 4feaa00 to 35b28de Compare February 1, 2019 14:55

tomwilkie added 2 commits February 1, 2019 16:29

Review feedback: use vendored protos from upstream Prometheus.

4e359b2

Required pinning / updating a couple of dependancies, but wasn't as bad as I first throught. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback: add TLS support.

92e359d

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

danielnelson reviewed Feb 1, 2019

View reviewed changes

danielnelson mentioned this pull request Feb 6, 2019

Add prometheus parser and serializer #4414

Closed

danielnelson modified the milestones: 1.10.0, 1.11.0 Feb 25, 2019

Praveen Shukla and others added 2 commits April 25, 2019 21:53

Merge pull request #1 from achilles42/prometheus-remote-write

d610f20

Add support of Bearer Token and handle timestamp order mismatch issue

danielnelson mentioned this pull request May 15, 2019

http_listener_v2 loses duplicated metrics with different timestamps #5862

Closed

danielnelson modified the milestones: 1.11.0, 1.12.0 May 24, 2019

achilles42 mentioned this pull request Jun 1, 2019

Add Prometheus remote write output plugin. #5942

Closed

3 tasks

russorat assigned tomwilkie Jul 26, 2019

danielnelson modified the milestones: 1.12.0, 1.13.0 Aug 19, 2019

sjwang90 modified the milestones: 1.13.0, 1.14.0 Nov 15, 2019

danielnelson closed this Nov 22, 2019

	switch point.Type() {
	case telegraf.Counter:
	if fn == "counter" {
	mname = sanitize(point.Name())
	}
	case telegraf.Gauge:
	if fn == "gauge" {
	mname = sanitize(point.Name())
	}
	}
	if mname == "" {
	if fn == "value" {
	mname = sanitize(point.Name())
	} else {
	mname = sanitize(fmt.Sprintf("%s_%s", point.Name(), fn))
	}
	}

Uh oh!

Conversation

tomwilkie commented Nov 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Required for all PRs:

Uh oh!

danielnelson commented Nov 19, 2018

Uh oh!

tomwilkie commented Nov 19, 2018

Uh oh!

tomwilkie commented Dec 5, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danielnelson Dec 17, 2018

Choose a reason for hiding this comment

Uh oh!

tomwilkie Feb 1, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danielnelson Dec 17, 2018

Choose a reason for hiding this comment

Uh oh!

tomwilkie Feb 1, 2019

Choose a reason for hiding this comment

Uh oh!

danielnelson Dec 17, 2018

Choose a reason for hiding this comment

Uh oh!

tomwilkie commented Feb 1, 2019

Uh oh!

danielnelson left a comment

Choose a reason for hiding this comment

Uh oh!

danielnelson Feb 1, 2019

Choose a reason for hiding this comment

Uh oh!

danielnelson commented Feb 4, 2019

Uh oh!

achilles42 commented May 26, 2019

Uh oh!

kushwiz commented Oct 19, 2019

Uh oh!

danielnelson commented Nov 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tomwilkie commented Nov 18, 2018 •

edited

Loading