Add a cloudwatch surfacer #583

robpickerill · 2021-03-31T18:58:41Z

This PR adds cloudwatch surfacer support for cloudprober.

This is deployed onto AWS ECS and working with the following config:

probe {
  name: "grafana_homepage"
  type: HTTP
  targets {
    host_names: "redacted"
  }
  http_probe: {
    protocol: HTTPS
  }
  interval_msec: 30000
  timeout_msec: 3000

  validator {
    name: "status_code_2xx"
    http_validator {
        success_status_codes: "200-399"
    }
  }

  latency_unit: "ms"
  latency_distribution {
    explicit_buckets: "10,20,40,80,160,320"
  }
}

probe {
  name: "grafana_homepage_https_redirect"
  type: HTTP
  targets {
    host_names: "redacted"
  }
  http_probe: {
    protocol: HTTP
  }
  interval_msec: 30000
  timeout_msec: 3000

  validator {
    name: "status_code_2xx"
    http_validator {
        success_status_codes: "200-399"
    }
  }
}

probe {
  name: "grafana_status"
  type: HTTP
  targets {
    host_names: "redacted"
  }
  http_probe: {
    protocol: HTTPS
    relative_url: "/api/health"
  }
  interval_msec: 30000
  timeout_msec: 3000

  validator {
    name: "status_code_2xx"
    http_validator {
        success_status_codes: "200-399"
    }
  }
}

probe {
  name: "grafana_metrics_dashboard"
  type: HTTP
  targets {
    host_names: "redacted"
  }
  http_probe: {
    protocol: HTTPS
    relative_url: "/d/Iwkba85Mk/grafana-metrics"
  }
  interval_msec: 30000
  timeout_msec: 3000

  validator {
    name: "status_code_2xx"
    http_validator {
        success_status_codes: "200-399"
    }
  }
}

probe {
  name: "grafana_login"
  type: HTTP
  targets {
    host_names: "redacted"
  }
  http_probe: {
    protocol: HTTPS
    relative_url: "/login"
  }
  interval_msec: 30000
  timeout_msec: 3000

  validator {
    name: "status_code_2xx"
    http_validator {
        success_status_codes: "200-399"
    }
  }
}

surfacer {
  type: CLOUDWATCH

  cloudwatch_surfacer {
    namespace: "/cloudprober/grafana"
  }
}

Using this IAM policy attached to the task role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Condition": {
                "StringEqualsIgnoreCase": {
                    "cloudwatch:namespace": "/cloudprober/grafana"
                }
            },
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "PutMetrics"
        }
    ]
}

google-cla · 2021-03-31T18:58:45Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

robpickerill · 2021-03-31T19:11:49Z

@googlebot I signed it!

google-cla · 2021-03-31T19:26:47Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

robpickerill · 2021-03-31T19:27:27Z

@googlebot I fixed it.

Rebased the wrong email out of the commits.

google-cla · 2021-03-31T19:27:31Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

robpickerill · 2021-03-31T19:47:56Z

@googlebot I fixed it.

should be fixed properly now

manugarg · 2021-04-05T19:13:09Z

@robpickerill Thanks a lot for this contribution. I'll try to review it soon -- I am out of office half of this week, so it may get pushed to next week.

manugarg

Hello @robpickerill,

Thanks once again for this. Sorry for the delay in reviewing. I've added some comments. Please take a look.

surfacers/cloudwatch/proto/config.proto

surfacers/cloudwatch/cloudwatch.go

robpickerill · 2021-04-17T13:58:46Z

Hey Manu

Thanks once again for this. Sorry for the delay in reviewing. I've added some comments. Please take a look.

No problems at all. I appreciate the feedback here. Let me work through the comments, and let you know when they are complete.

Thanks again.

robpickerill · 2021-04-17T17:41:20Z

Manu, I've been resolving the comments so I can see which comments are still outstanding - if I've missed anything please feel free to unresolve.

docs/content/surfacers/cloudwatch.md

manugarg

Thank you for addressing the comments. I've added a few more comments, but you don't need to fix them. I am working on importing/exporting this change following the process the described at:
https://github.com/google/cloudprober/blob/master/CONTRIBUTING.md#source-of-truth-sot-and-commit-process

I'll make minor fixes (mentioned in the comments) while doing that.

manugarg · 2021-04-20T06:39:47Z

surfacers/cloudwatch/cloudwatch.go

+
+		switch lu {
+		case time.Second:
+			metricDatum.Value = aws.Float64(value * 1000)
+		case time.Microsecond:
+			metricDatum.Value = aws.Float64(value / 1000)
+		case time.Nanosecond:
+			metricDatum.Value = aws.Float64((value / 1000) / 1000)
+		}
+	}


I think you can do this instead:

metricDatum.Value = aws.Float64(value * float64(latencyUnit) / float64(time.Millisecond))

surfacers/cloudwatch/proto/config.proto

manugarg · 2021-04-21T06:37:35Z

surfacers/cloudwatch/cloudwatch.go

+
+// publish the metrics to cloudwatch, using the namespace provided from configuration
+func (cw *CWSurfacer) publishMetrics(md *cloudwatch.MetricDatum) {
+	if len(cw.cwMetricDatumCache) >= 20 {


You probably want to use cloudwatchMaxMetricDatums here instead of 20.

sorry, that's an obvious miss by me - but thanks for catching

No worries. :)

manugarg · 2021-04-21T06:47:11Z

surfacers/cloudwatch/cloudwatch.go

+	return failure, nil
+}
+
+func (s *CWSurfacer) ignoreMetric(name string) bool {


Receiver "s" needs to consistent.

Again, sorry, thats as obvious miss by me, but thanks again for catching - was copy/paste as I was trying to keep consistent with the stackdriver implementation. Apologies

No worries.

This is import of the original PR at #583. PiperOrigin-RevId: 369784229

manugarg · 2021-04-22T02:39:25Z

@robpickerill Importing this PR through #587.

Documentation changes are not pushed through the import-export method. They can be committed directly. Can you please send another PR, containing just the documentation changes? You can close this PR afterwards.

robpickerill · 2021-04-22T07:26:12Z

Thanks for all your assistance here Manu, very much appreciated.

Can you please send another PR, containing just the documentation changes?

Let me wrap this up today and we can close down this PR, let me also put the items we discussed on regex's into issues for tracking, then this PR is good to close, and I'll work on datadog.

Thanks again!

This is import of the original PR at #583. PiperOrigin-RevId: 369784229

robpickerill · 2021-04-23T09:20:21Z

closing for #587

robpickerill added 2 commits March 31, 2021 20:10

add initial raw implementation of cw surfacer

5c68589

remove generated proto buf for regen

31ed6ae

robpickerill changed the title ~~WIP Add initial implementation of a cw surfacer~~ WIP Add initial implementation of a cloudwatch surfacer Mar 31, 2021

generate pb and align protoc version

4d8d4f0

robpickerill added 2 commits April 1, 2021 21:11

fix failing test

902ddad

add documentation

a3f338f

robpickerill changed the title ~~WIP Add initial implementation of a cloudwatch surfacer~~ Add a cloudwatch surfacer Apr 4, 2021

robpickerill added 2 commits April 4, 2021 20:51

add cloudwatch to overview

2238113

add comment about delta/snapshot data

c03a886

robpickerill and others added 6 commits April 6, 2021 08:38

Fix semantics on m1

39dc2e1

improve readability of unit calculations

8b18f23

Merge branch 'master' of github.com:robpickerill/cloudprober

32ecaf3

remove if guard for not dist, as always true

b00d501

refactor unit types for readability

9cbe89f

retrigger ci

aaeafa2

manugarg reviewed Apr 17, 2021

View reviewed changes

robpickerill added 6 commits April 17, 2021 18:13

pass common options for buffer size

227d723

add copyright notices

fbae5cf

remove doc package explanation

ef17679

remove comment for cwsurfacer struct

47a16f9

rename recordEventMetrics

670e238

typo on distributionBound

4ff2d24

robpickerill added 3 commits April 17, 2021 18:34

fix MapName

9e90aa5

range into mapKey var

6451151

remove unused label

898f4d5

robpickerill commented Apr 17, 2021

View reviewed changes

docs/content/surfacers/cloudwatch.md Show resolved Hide resolved

robpickerill added 4 commits April 18, 2021 16:00

use the provided latency units

6608340

match stackdriver accepted regex

2335d13

due to cw api behaviours, always use milliseconds

ec6dfb7

docs: tidy comment and doc about allowed_metrics_regex

35d82fc

manugarg reviewed Apr 21, 2021

View reviewed changes

manugarg pushed a commit that referenced this pull request Apr 22, 2021

Add AWS cloudwatch surfacer.

7d41a1f

This is import of the original PR at #583. PiperOrigin-RevId: 369784229

manugarg mentioned this pull request Apr 22, 2021

Add AWS cloudwatch surfacer. #587

Merged

robpickerill mentioned this pull request Apr 22, 2021

docs: add documentation for cw surfacer #588

Merged

manugarg pushed a commit that referenced this pull request Apr 22, 2021

Add AWS cloudwatch surfacer.

aa44e00

This is import of the original PR at #583. PiperOrigin-RevId: 369784229

robpickerill mentioned this pull request Apr 23, 2021

Improve regex filtering in surfacers #590

Closed

robpickerill closed this Apr 23, 2021

manugarg added this to the v0.11.3 milestone Sep 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a cloudwatch surfacer #583

Add a cloudwatch surfacer #583

robpickerill commented Mar 31, 2021 •

edited

Loading

google-cla bot commented Mar 31, 2021

robpickerill commented Mar 31, 2021

google-cla bot commented Mar 31, 2021

robpickerill commented Mar 31, 2021

google-cla bot commented Mar 31, 2021

robpickerill commented Mar 31, 2021

manugarg commented Apr 5, 2021

manugarg left a comment

robpickerill commented Apr 17, 2021

robpickerill commented Apr 17, 2021

manugarg left a comment

manugarg Apr 20, 2021

manugarg Apr 21, 2021

robpickerill Apr 21, 2021

manugarg Apr 22, 2021

manugarg Apr 21, 2021

robpickerill Apr 21, 2021

manugarg Apr 22, 2021

manugarg commented Apr 22, 2021 •

edited

Loading

robpickerill commented Apr 22, 2021

robpickerill commented Apr 23, 2021

Add a cloudwatch surfacer #583

Add a cloudwatch surfacer #583

Conversation

robpickerill commented Mar 31, 2021 • edited Loading

google-cla bot commented Mar 31, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

robpickerill commented Mar 31, 2021

google-cla bot commented Mar 31, 2021

robpickerill commented Mar 31, 2021

google-cla bot commented Mar 31, 2021

robpickerill commented Mar 31, 2021

manugarg commented Apr 5, 2021

manugarg left a comment

Choose a reason for hiding this comment

robpickerill commented Apr 17, 2021

robpickerill commented Apr 17, 2021

manugarg left a comment

Choose a reason for hiding this comment

manugarg Apr 20, 2021

Choose a reason for hiding this comment

manugarg Apr 21, 2021

Choose a reason for hiding this comment

robpickerill Apr 21, 2021

Choose a reason for hiding this comment

manugarg Apr 22, 2021

Choose a reason for hiding this comment

manugarg Apr 21, 2021

Choose a reason for hiding this comment

robpickerill Apr 21, 2021

Choose a reason for hiding this comment

manugarg Apr 22, 2021

Choose a reason for hiding this comment

manugarg commented Apr 22, 2021 • edited Loading

robpickerill commented Apr 22, 2021

robpickerill commented Apr 23, 2021

robpickerill commented Mar 31, 2021 •

edited

Loading

manugarg commented Apr 22, 2021 •

edited

Loading