Datadog monitors - 'metric alert' versus 'query alert'

[kmshultz]

Terraform Version

$ terraform version
Terraform v0.9.2

Affected Resource(s)

• datadog_monitor

Terraform Configuration Files

resource "datadog_monitor" "outliermonitor" {
  name = "test outlier monitor"
  type = "metric alert"
  query = "avg(last_5m):outliers(avg:chef.resources.updated{env:prod} by {host}, 'DBSCAN', 3.0) > 0"
  message = "chef resources outlier alert"
}

Debug Output

https://gist.github.com/kmshultz/23bf92aea315a1596d21920092da1c11

Expected Behavior

Terraform ought not to update the datadog_monitor on every apply.

Actual Behavior

Terraform updated the datadog_monitor because while it passed 'metric alert' to the Datadog API on creation, the API categorized it — because of its more complex query — as 'query alert'.

This is a quirk on Datadog's side, not a bug in Terraform. When a client POSTs or PUTs a Datadog monitor with type 'metric alert', the Datadog API inspects the query and sometimes recategorizes it as 'query alert' (e.g. for APM monitors, Outlier monitors, monitors that do arithmetic on two or more metrics). However, clients are NOT advised to pass 'query alert' in POST/PUTs — it's up to Datadog to decide when a 'metric alert' should be recategorized. 'query alert' is only important for the Datadog monitoring backend to know about, but the type is sort of exposed to clients right now, and this isn't likely to change any time soon.

So, it's hacky, but the datadog_monitor resource should probably look for 'query alert' in its Read function and override it to 'metric alert'. Without this override, terraform apply will always try to PUT a 'query alert' monitor back to 'metric alert', fail to do so (though Datadog will still return 2xx), and keep trying to set it to 'metric alert' every time.

Steps to Reproduce

See Debugging output gist above.

[kmshultz]

Something like this should work:

	clientSideType := m.GetType()
	if clientSideType == "query alert" {
		clientSideType = "metric alert"
	}
	d.Set("type", clientSideType)

[eredi93]

+1

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.