-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add error handling for errors in Datadog API response #103
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What was the failure mode for this? From what it looks like it appears that it didn't return an error at all since err == nil
. If some of the metrics are being accepted returning an error here for 202
responses may cause some metric values not to be submitted if there were a large number of points being submitted. Are we fine with stopping the submission of any remaining metrics when we get a partial success?
sink/datadog.go
Outdated
@@ -393,6 +394,11 @@ func (s *Datadog) sendApi(ddCtx context.Context, dp []datadogV2.MetricSeries) er | |||
return err | |||
} | |||
|
|||
if len(apiResponse.Errors) > 0 { | |||
// datadog can return a 202 Accepted response code but errors in response payload | |||
return fmt.Errorf("error response from Datadog: %s", strings.Join(apiResponse.Errors, ",")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re @Areson comment: if this is partial success, then we should not abort completely.
The series that was being sent all had the same issue, so datadog returned error for the whole batch, but I think it can have partial success. I will verify. But I agree, if it returns partial success, we should log the error and continue. Though I prefer to fail fast, but in this case as It might be a good idea to send a meta metric for partial errors in |
The metrics were not being sent to Datadog and there were no errors in logs. So it was failing silently. |
I thought about meta metrics, but it's one of those areas where probably nothing we do will be right for a general audience. So instead, partial failure should emit a Blip event because (my thinking was) those can be picked up by a custom sink and sent wherever, to do whatever. |
Verified, this can be partial success. Updated the code to log the errors and continue. |
Updated PR to Raise an event as well. |
Datadog can return a
202 Accepted
response code but errors in response payload. This PR adds a check for that.From what it looks like, the endpoint probably also partially accept MetricData which are valid and return errors for invalid ones.
With this, we will at least see in logs what's going on and will be able to fix it quickly. Without this, it was failing silently.