cmd/tsdbrelay: enable version numbers, fix metadata, add error metrics #1859

Merged
merged 5 commits into from Aug 18, 2016

Conversation

Projects
None yet
2 participants
@gbrayut
Contributor

gbrayut commented Aug 11, 2016

Also:
removes runtime.GOMAXPROCS(runtime.NumCPU()) since that is now the default
tabs to spaces in docker.sh
updates docker image to use to go1.7.linux-amd64.tar.gz
fixes missing call to metadata.init
adds metrics for tracking errors from opentsdb/bosun
adds metrics for tracking relayed/errors from secondary relays.

This also changes the logic so that data points are always relayed to bosun and secondary relays instead of bailing when the primary opentsdb end point returns an error. Will keep thinking on this and open a separate PR.

Fixes #1792

cmd/tsdbrelay/main.go
@@ -14,6 +15,8 @@ import (
"os"
"strings"
+ version "bosun.org/_version"

This comment has been minimized.

@gbrayut

gbrayut Aug 11, 2016

Contributor

@captncraig I'm curious why I had to put an alias here. Locally using go 1.6.3 it can't find the reference to version package and gives an error/removes the import if I try to use a default import or even the _ import. Scollector/Bosun seem to work fine without the alias.

@gbrayut

gbrayut Aug 11, 2016

Contributor

@captncraig I'm curious why I had to put an alias here. Locally using go 1.6.3 it can't find the reference to version package and gives an error/removes the import if I try to use a default import or even the _ import. Scollector/Bosun seem to work fine without the alias.

This comment has been minimized.

@captncraig

captncraig Aug 11, 2016

Contributor

I can't explain why. If it helps to remove the alias, go ahead, but it sounds like something odd may be going on with your go environment.

@captncraig

captncraig Aug 11, 2016

Contributor

I can't explain why. If it helps to remove the alias, go ahead, but it sounds like something odd may be going on with your go environment.

@gbrayut gbrayut changed the title from cmd/tsdbrelay: enable version numbers to cmd/tsdbrelay: enable version numbers, fix metadata, always relay Aug 16, 2016

@gbrayut

This comment has been minimized.

Show comment
Hide comment
@gbrayut

gbrayut Aug 16, 2016

Contributor

I'm going to test this on ny-gbraylx01, which uses tsdbrelay to forward to a local docker instance and an influxdb instance and relay to the primary bosun instance.

@captncraig please review when you have time

Contributor

gbrayut commented Aug 16, 2016

I'm going to test this on ny-gbraylx01, which uses tsdbrelay to forward to a local docker instance and an influxdb instance and relay to the primary bosun instance.

@captncraig please review when you have time

@gbrayut

This comment has been minimized.

Show comment
Hide comment
@gbrayut

gbrayut Aug 16, 2016

Contributor

Hmm... Metadata is working now but it still doesn't always relay. I'll keep working on this.

Contributor

gbrayut commented Aug 16, 2016

Hmm... Metadata is working now but it still doesn't always relay. I'll keep working on this.

@gbrayut

This comment has been minimized.

Show comment
Hide comment
@gbrayut

gbrayut Aug 16, 2016

Contributor

This approach isn't going to work. The reverse proxy generates a 500 response code if there is a connection issue to the primary target and immediately sends that to the client. So even if we still try forwarding the data points to the other endpoints the scollector client will have those same data points queued up to be retried, resulting in duplicate data.

I think there is a simpler way to solve the issue of HA forwarding, so I'm going to bundle up the other commits and push those to master, since they fix a bunch of non forwarding related issues.

Contributor

gbrayut commented Aug 16, 2016

This approach isn't going to work. The reverse proxy generates a 500 response code if there is a connection issue to the primary target and immediately sends that to the client. So even if we still try forwarding the data points to the other endpoints the scollector client will have those same data points queued up to be retried, resulting in duplicate data.

I think there is a simpler way to solve the issue of HA forwarding, so I'm going to bundle up the other commits and push those to master, since they fix a bunch of non forwarding related issues.

@gbrayut gbrayut changed the title from cmd/tsdbrelay: enable version numbers, fix metadata, always relay to cmd/tsdbrelay: enable version numbers, fix metadata, add error metrics Aug 16, 2016

@gbrayut

This comment has been minimized.

Show comment
Hide comment
@gbrayut

gbrayut Aug 16, 2016

Contributor

This branch now has all the non-forwarding related changes. I plan on merging it tomorrow (testing on ny-gbraylx01 overnight)

Contributor

gbrayut commented Aug 16, 2016

This branch now has all the non-forwarding related changes. I plan on merging it tomorrow (testing on ny-gbraylx01 overnight)

cmd/tsdbrelay: include connection failures in additional.error metric…
…. Also continue additional relay urls on failure.

@gbrayut gbrayut merged commit 8eedf8d into master Aug 18, 2016

2 checks passed

bosun All checks Passed!
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@gbrayut gbrayut deleted the tsdbrelay branch Aug 18, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment