Use DdSite or DdUrl for MetricsForwarder API client #92

mantoine96 · 2020-05-22T08:19:30Z

What does this PR do?

This PR:

Adds a field to the metricsForwarder struct: baseURL
Adds a getBaseURL method that returns the appropriate base URL based on the ddUrl or site fields in the definition of the agent
Sets the base URL after the client has been initialized
Adds the base URL used when error-ing out on invalid credentials.

Motivation

After submitting #90 and testing it out, I realized that the operator was also trying to push metrics, grabbing the credentials from the agent. Unfortunately the metrics forwarder wasn't using the DD_SITE / DD_DD_URL setting and would use the default of the client SDK, which is https://api.datadoghq.com

Additional Notes

I've tested this successfully in a 1.16 EKS cluster, using site: datadoghq.eu

PS: The validation build failure doesn't seem to be related to my changes.

ahmed-mez · 2020-05-22T11:47:53Z

pkg/controller/utils/datadog/metrics_forwarder.go

@@ -653,3 +659,12 @@ func (mf *metricsForwarder) isErrChanFull() bool {
 func (mf *metricsForwarder) isEventChanFull() bool {
 	return len(mf.eventChan) == cap(mf.eventChan)
 }
+
+func getbaseURL(dda *datadoghqv1alpha1.DatadogAgent) string {
+	if dda.Spec.Agent.Config.DDUrl != nil {


this dda.Spec.Agent can be nil

ahmed-mez

Thanks @thehunt33r for the contribution, I added a comment about a potential nil pointer, could you also add a unit test for the changes?

pkg/controller/utils/datadog/metrics_forwarder.go

mantoine96 · 2020-05-22T12:02:30Z

@ahmed-mez Will get to it now :)

pkg/controller/utils/datadog/metrics_forwarder_test.go

ahmed-mez · 2020-05-25T09:00:49Z

pkg/controller/utils/datadog/metrics_forwarder_test.go

+			},
+			want: "https://test.url.com",
+		},
+		// TODO: Add test cases.


nit: can we add a test case where both Site and DDUrl are explicitly set to test the order of precedence?

ahmed-mez · 2020-05-25T09:02:48Z

pkg/controller/datadogagent/datadogagent_controller.go

@@ -103,7 +103,7 @@ func add(mgr manager.Manager, r reconcile.Reconciler, registerFunc func(datadog.
 	// Watch for changes to primary resource DatadogAgent
 	err = c.Watch(&source.Kind{Type: &datadoghqv1alpha1.DatadogAgent{}}, &handler.EnqueueRequestForObject{}, onCreate)
 	if err != nil {
-		return err
+	return err


I'm not sure how that happened. Anyhow, I undid the change. Good spot!

ahmed-mez · 2020-05-25T09:04:02Z

pkg/controller/utils/datadog/metrics_forwarder.go

+	} else if dda.Spec.Site != "" {
+		return fmt.Sprintf("https://api.%s", dda.Spec.Site)
+	}
+	return "https://api.datadoghq.com"


Suggested change

return "https://api.datadoghq.com"

return defaultbaseURL

ahmed-mez · 2020-05-25T10:10:07Z

thanks @thehunt33r
happy to approve and merge this once the CI is all green, the current problem seems caused by a missing } in the metrics forwarder file

mantoine96 · 2020-05-25T10:18:01Z

🤦 whoops. Addressed.

ahmed-mez · 2020-05-25T10:35:11Z

pkg/apis/datadoghq/v1alpha1/test/new.go

@@ -86,6 +87,7 @@ func NewDefaultedDatadogAgent(ns, name string, options *NewDatadogAgentOptions)
 			Log:                datadoghqv1alpha1.LogSpec{},
 			Process:            datadoghqv1alpha1.ProcessSpec{},
 		},
+		Site: options.Site,


this causes a panic when options is nil
it should be moved to the block below when we check if options != nil

🤦 Should have caught it earlier. Thank you!

Make golint happy by renaming baseUrl baseURL Fix site reference Log the base URL that was fetched Co-authored-by: Ahmed Mezghani <38987709+ahmed-mez@users.noreply.github.com> Add unit test for getbaseURL re-fix the potential nil Agent Add test case for testing precedence of DDUrl over Site Use defaultBaseURL const in return Undo change on datadogagent_controller.go Remove Todo forgot a } Fix test

mantoine96 · 2020-05-25T11:02:45Z

pkg/apis/datadoghq/v1alpha1/test/new.go

@@ -104,6 +105,7 @@ func NewDefaultedDatadogAgent(ns, name string, options *NewDatadogAgentOptions)
 		}

 		ad.Spec.Agent.DaemonsetName = options.AgentDaemonsetName
+		ad.Spec.Site = options.Site


@ahmed-mez Addressed here

codecov-commenter · 2020-05-25T11:15:46Z

Codecov Report

Merging #92 into master will increase coverage by 0.00%.
The diff coverage is 50.00%.

@@           Coverage Diff           @@
##           master      #92   +/-   ##
=======================================
  Coverage   57.23%   57.24%           
=======================================
  Files          31       31           
  Lines        4165     4175   +10     
=======================================
+ Hits         2384     2390    +6     
- Misses       1600     1604    +4     
  Partials      181      181

Flag	Coverage Δ
#unittests	`57.24% <50.00%> (+<0.01%)`	⬆️

Impacted Files	Coverage Δ
pkg/controller/utils/datadog/metrics_forwarder.go	`42.30% <50.00%> (+0.53%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 607e3ce...6f9db62. Read the comment docs.

ahmed-mez

💯

mantoine96 force-pushed the Use-Site-for-metrics-forwarder branch from faf3254 to b62612d Compare May 22, 2020 09:30

mantoine96 marked this pull request as ready for review May 22, 2020 11:29

mantoine96 requested a review from a team as a code owner May 22, 2020 11:29

ahmed-mez reviewed May 22, 2020

View reviewed changes

ahmed-mez suggested changes May 22, 2020

View reviewed changes

ahmed-mez reviewed May 22, 2020

View reviewed changes

pkg/controller/utils/datadog/metrics_forwarder.go Outdated Show resolved Hide resolved

mantoine96 commented May 22, 2020

View reviewed changes

pkg/controller/utils/datadog/metrics_forwarder_test.go Show resolved Hide resolved

mantoine96 requested a review from ahmed-mez May 22, 2020 14:03

ahmed-mez reviewed May 25, 2020

View reviewed changes

mantoine96 force-pushed the Use-Site-for-metrics-forwarder branch 2 times, most recently from 9e6136c to 755890c Compare May 25, 2020 09:21

mantoine96 requested a review from ahmed-mez May 25, 2020 09:22

mantoine96 force-pushed the Use-Site-for-metrics-forwarder branch from c515df2 to e1fdfbc Compare May 25, 2020 10:17

ahmed-mez suggested changes May 25, 2020

View reviewed changes

mantoine96 force-pushed the Use-Site-for-metrics-forwarder branch from e1fdfbc to 6f9db62 Compare May 25, 2020 11:02

mantoine96 commented May 25, 2020

View reviewed changes

mantoine96 requested a review from ahmed-mez May 25, 2020 11:08

ahmed-mez approved these changes May 25, 2020

View reviewed changes

ahmed-mez merged commit e44716e into DataDog:master May 25, 2020

clamoriniere added this to the v0.3 milestone Jun 29, 2020

mantoine96 mentioned this pull request Aug 26, 2020

datadoghq.eu supported? #128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use DdSite or DdUrl for MetricsForwarder API client #92

Use DdSite or DdUrl for MetricsForwarder API client #92

mantoine96 commented May 22, 2020 •

edited

ahmed-mez May 22, 2020

ahmed-mez left a comment

mantoine96 commented May 22, 2020

ahmed-mez May 25, 2020

mantoine96 May 25, 2020

ahmed-mez May 25, 2020

mantoine96 May 25, 2020

ahmed-mez May 25, 2020

mantoine96 May 25, 2020

ahmed-mez commented May 25, 2020

mantoine96 commented May 25, 2020

ahmed-mez May 25, 2020

mantoine96 May 25, 2020

mantoine96 May 25, 2020

codecov-commenter commented May 25, 2020

ahmed-mez left a comment

Use DdSite or DdUrl for MetricsForwarder API client #92

Use DdSite or DdUrl for MetricsForwarder API client #92

Conversation

mantoine96 commented May 22, 2020 • edited

What does this PR do?

Motivation

Additional Notes

Choose a reason for hiding this comment

ahmed-mez left a comment

Choose a reason for hiding this comment

mantoine96 commented May 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmed-mez commented May 25, 2020

mantoine96 commented May 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 25, 2020

Codecov Report

ahmed-mez left a comment

Choose a reason for hiding this comment

mantoine96 commented May 22, 2020 •

edited