feat(new sink): New `sematext_metrics` sink #3501

StephenWakely · 2020-08-19T17:02:04Z

This is very much a work in progress.

The sink is working. I can set it up and it can send metrics into Sematext.

I have an issue with the smoke test. I am attempting to send 10 metrics events into the sink (Here : https://github.com/FungusHumungus/vector/blob/sematext_metrics/src/sinks/sematext_metrics.rs#L343), but no matter what I do I cannot get it to process more than 1 event (as witnessed by the println here - https://github.com/FungusHumungus/vector/blob/sematext_metrics/src/sinks/sematext_metrics.rs#L161.

I'm not sure if this is because I have misunderstood precisely how the sink should work, or if I have something wrong in the code, or indeed something wrong in the test..

Any ideas?

StephenWakely · 2020-08-19T17:05:50Z

Another question. As far as I can tell (I still need to confirm this with Sematext support), the only useful metric types are single values - Counters and Guages.

What would be the best way to handle the situation should the sink receives any other type of metric?

StephenWakely · 2020-08-23T10:41:13Z

I'm not sure if this is because I have misunderstood precisely how the sink should work, or if I have something wrong in the code, or indeed something wrong in the test..

Any ideas?

I have worked it out. It is to do with the batch settings. Without setting the batch.max_events it defaults to 20. So presumably, just sending 10 events to the sink means it is going to batch the events up and wait until it can get to 20. Setting max_events to 1 allows the test to recieve all 10 events.

It all makes sense now!

StephenWakely · 2020-08-25T12:13:21Z

I think this is good to go now.

One thing I'm not sure I have done the best way is in sending the version number of Vector to Sematext. The version string returned by get_version in lib.rs is fairly long and would need to be url-encoded - the version string would then be about half of the entire request . To avoid this I am instead making my own simplified version string that mostly just contains 'vector' with the version number.
(https://github.com/FungusHumungus/vector/blob/sematext_metrics/src/sinks/sematext/metrics.rs#L46)

Not sure if that is the best way. If it is, maybe we would want to move this function into lib.rs so that other code can access it if needed?

Another possible improvement that may be worth considering, currently the sink is sending one request per metric to Sematext. It could be possible to group the events so every metric with the same namespace and timestamp in a batch would be sent in a single request. This could save network traffic if there are a lot of metrics sent through Vector in one go.

bruceg

This looks mostly good, but a couple of cleanups needed and an internal event for the dropped metric warning/error.

src/sinks/influxdb/mod.rs

src/sinks/sematext/mod.rs

src/sinks/sematext/metrics.rs

src/internal_events/sematext_metrics.rs

StephenWakely · 2020-08-30T19:19:58Z

Aaarrg.. darn it. I think I have messed this pull request up. I rebased the branch with master, and it seems to have pulled in a ton of stuff that I don't need.

Really sorry, any idea what I need to do to fix this?

MOZGIII · 2020-08-31T18:06:56Z

The easiest way is to do an interactive rebase (git rebase -i master) and drop the commits that it shows in the list that are coming from master. They don't have to exist in your branch. so rebase shouldn't show them in the final git rebase -i master.

StephenWakely · 2020-08-31T23:17:42Z

@MOZGIII Awesome! That seems to have tidied it up, although it has still kept you on as a reviewer. I presume you can remove yourself if needed?

Relief! Many thanks.

bruceg

Looks good, just a possible pub change and a question for @binarylogic about the docs.

src/sinks/influxdb/mod.rs

.meta/sinks/sematext_metrics.toml.erb

binarylogic · 2020-09-08T19:01:13Z

My biggest question here is the data model, which should align with #3552. That, unfortunately, is something that will likely require an RFC :/.

We could merge this and follow up with any changes, but that'll be a breaking change.

StephenWakely · 2020-09-09T11:59:30Z

Have made the suggested changes.

I'm not sure I fully understand how this is not aligned with the data model from #3552?

binarylogic · 2020-09-09T14:05:24Z

@FungusHumungus that's good to hear. It's more about consensus. In #3552 (comment), @drunkirishcoder proposes a public contract for the schema, which I agree with. We'll try to expedite our discussions there so we don't block this any longer than needed.

binarylogic · 2020-09-09T14:22:56Z

I've opened #3779 for this.

binarylogic · 2020-09-10T03:04:51Z

@FungusHumungus this should be good to go. A few things:

Is the concern in the original PR description still valid?
Did you manually test this to ensure it works?
Need to resolve the conflicts.

Thanks.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely · 2020-10-10T22:32:40Z

@binarylogic Would you be able to have a look over the docs to check I have done them correctly?

In particular, I'm not 100% sure about the is the How it works section.

https://github.com/timberio/vector/pull/3501/files#diff-2c39791f3299d4d31f24a395ece3881f

binarylogic · 2020-10-10T22:33:55Z

docs/reference/components/sinks/sematext_metrics.cue

+
+components: sinks: sematext_metrics: {
+	title:             "Sematext Metrics"
+	short_description: ""


Need a one sentence description here.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

binarylogic

@FungusHumungus docs looks great, nice work!

@JeanMertz

* Initial wip for the sematext_metrics sink Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Moved Sematext modules into a single module Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Added documentation for the Sematext metrics sink Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Send the Vector version with the Sematext request Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixing issues following comments from bruceg Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Moved Sematext modules into a single module Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixing issues following comments from bruceg Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Updated sematext events in line with comments from BruceG. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Changes following a merge with master Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixed input type in documentation Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Added relevant_when fields to docs for host and region fields. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Restricted the visibility of a number of functions Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Added noun to sematext metrics docs Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixed clippy warnings Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixes following a merge with master Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Added a better healthcheck url Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * sink.run no longer takes Result Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Renamed host parameter to endpoint Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Renamed events following similar changes to statsd events Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Changes following a review from @JeanMertz Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Changes following a merge with master Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Add docs cue file Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Further changes following merge with master Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fix cue formatting Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Further attempt to fix cue formatting. Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Use tabs not spaces Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixed docs cue errors Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixed cue doc following merge with master Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Added a short description to the docs Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> * Fixed cue formatting Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com> Co-authored-by: binarylogic <bjohnson@binarylogic.com>

StephenWakely requested a review from lukesteensen as a code owner August 19, 2020 17:02

StephenWakely mentioned this pull request Aug 19, 2020

fix(statsd sink): Handle Absolute kind for values other than Gauge. #3510

Merged

binarylogic requested review from JeanMertz, binarylogic and bruceg and removed request for lukesteensen and JeanMertz August 25, 2020 01:08

bruceg requested changes Aug 25, 2020

View reviewed changes

bruceg reviewed Aug 26, 2020

View reviewed changes

src/internal_events/sematext_metrics.rs Outdated Show resolved Hide resolved

src/internal_events/sematext_metrics.rs Outdated Show resolved Hide resolved

bruceg self-assigned this Aug 26, 2020

bruceg added domain: sinks Anything related to the Vector's sinks provider: sematext Anything `sematext` service provider related type: feature A value-adding code addition that introduce new functionality. labels Aug 26, 2020

StephenWakely requested a review from MOZGIII as a code owner August 30, 2020 19:11

StephenWakely mentioned this pull request Aug 30, 2020

StatsD sink should create an internal event when sent a metric it can't handle. #3635

Closed

StephenWakely force-pushed the sematext_metrics branch from 53de8cd to 0f2d7c4 Compare August 31, 2020 23:11

MOZGIII removed their request for review September 1, 2020 00:08

bruceg approved these changes Sep 8, 2020

View reviewed changes

src/sinks/influxdb/mod.rs Outdated Show resolved Hide resolved

.meta/sinks/sematext_metrics.toml.erb Outdated Show resolved Hide resolved

StephenWakely requested a review from JeanMertz as a code owner September 9, 2020 11:55

StephenWakely added 19 commits October 10, 2020 22:48

Fixed input type in documentation

b60a9e5

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Added relevant_when fields to docs for host and region fields.

5e4dfde

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Restricted the visibility of a number of functions

5434b4a

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Added noun to sematext metrics docs

cffbb6b

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixed clippy warnings

05231e7

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixes following a merge with master

4bb50d1

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Added a better healthcheck url

894f8ab

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

sink.run no longer takes Result

3c03f05

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Renamed host parameter to endpoint

72d33fb

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Renamed events following similar changes to statsd events

e280c08

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Changes following a review from @JeanMertz

d19ac23

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Changes following a merge with master

cb2f8c4

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Add docs cue file

ee072eb

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Further changes following merge with master

3cf8588

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fix cue formatting

1900af1

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Further attempt to fix cue formatting.

8303234

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Use tabs not spaces

274c361

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixed docs cue errors

5f5d5f9

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixed cue doc following merge with master

f6b249c

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

StephenWakely force-pushed the sematext_metrics branch from 8e645df to f6b249c Compare October 10, 2020 21:37

StephenWakely requested a review from binarylogic October 10, 2020 22:30

binarylogic reviewed Oct 10, 2020

View reviewed changes

StephenWakely and others added 3 commits October 11, 2020 00:41

Added a short description to the docs

006911c

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Fixed cue formatting

ce8e717

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

Merge remote-tracking branch 'upstream/master' into sematext_metrics

aba77c9

binarylogic approved these changes Oct 11, 2020

View reviewed changes

StephenWakely merged commit 36dea21 into vectordotdev:master Oct 11, 2020

StephenWakely deleted the sematext_metrics branch October 11, 2020 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new sink): New `sematext_metrics` sink #3501

feat(new sink): New `sematext_metrics` sink #3501

StephenWakely commented Aug 19, 2020 •

edited

Loading

StephenWakely commented Aug 19, 2020

StephenWakely commented Aug 23, 2020

StephenWakely commented Aug 25, 2020

bruceg left a comment

StephenWakely commented Aug 30, 2020

MOZGIII commented Aug 31, 2020 •

edited

Loading

StephenWakely commented Aug 31, 2020

bruceg left a comment

binarylogic commented Sep 8, 2020

StephenWakely commented Sep 9, 2020

binarylogic commented Sep 9, 2020

binarylogic commented Sep 9, 2020

binarylogic commented Sep 10, 2020

StephenWakely commented Oct 10, 2020

binarylogic Oct 10, 2020

binarylogic left a comment

feat(new sink): New sematext_metrics sink #3501

feat(new sink): New sematext_metrics sink #3501

Conversation

StephenWakely commented Aug 19, 2020 • edited Loading

StephenWakely commented Aug 19, 2020

StephenWakely commented Aug 23, 2020

StephenWakely commented Aug 25, 2020

bruceg left a comment

Choose a reason for hiding this comment

StephenWakely commented Aug 30, 2020

MOZGIII commented Aug 31, 2020 • edited Loading

StephenWakely commented Aug 31, 2020

bruceg left a comment

Choose a reason for hiding this comment

binarylogic commented Sep 8, 2020

StephenWakely commented Sep 9, 2020

binarylogic commented Sep 9, 2020

binarylogic commented Sep 9, 2020

binarylogic commented Sep 10, 2020

StephenWakely commented Oct 10, 2020

binarylogic Oct 10, 2020

Choose a reason for hiding this comment

binarylogic left a comment

Choose a reason for hiding this comment

feat(new sink): New `sematext_metrics` sink #3501

feat(new sink): New `sematext_metrics` sink #3501

StephenWakely commented Aug 19, 2020 •

edited

Loading

MOZGIII commented Aug 31, 2020 •

edited

Loading