New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Datadog contrib for monitoring purpose #2434
Add Datadog contrib for monitoring purpose #2434
Commits on Dec 13, 2018
-
Add Datadog contrib for monitoring purpose
Datadog is a tool that allows you to send metrics that you create dashboard and add alerting on specific behaviors. Adding this contrib will allow for users of this tool to log their pipeline information to Datadog. Based on the status change of a task, we log that information to Datadog with the parameters that were used to run that specific task. This allow us to easily create dashboard to visualize the health. For example, we can be notified via Datadog if a task has failed, or we can graph the execution time of a specific task over a period of time. The implementation idea was strongly based on the stale PR spotify#2044.
-
Refactor MetricsCollectors in Scheduler
I've also added a few test to ensure that the implementation was working well.
-
Add polish + tests around metrics on task state
This takes care of ensuring that the proper metrics collection calls are being done when they are expected to be happening. We've also removed a few `@RPC_METHOD` that weren't actually being used and that wasn't required.
-
Add tests related to the Datadog contrib
This makes sure that we're properly dispatching API and STATSD call with the proper parameter values to Datadog. This doesn't test all the different possible parameters configuration.
-
Improve configuration documentation with new Datadog contrib
This adds a few extra documentation line for the configuration to allow user to find all the settings they can tweak for each individual contribs instead of having to go through each individual contrib files.
-
Update DataDog dep. to the most recent release
The original implementation was made when 0.16.0 was the latest version. Since there there have been a few improvements and bug fixes made to the library that we should be using. Reading through the release log there shouldn't be any feature-breaking changes so we should be good to update it!
-
Change metrics collection getter to class method
Previously, the getter wasn't a class method and wouldn't work as expected. In order to ensure that the output is what we expect, we've added more tests.
-
Fix spec issues related to new tests
There was multiple problems that needed to be solved in order to get the specs green again. Each individual specs were passing when ran individually, but when ran into tox as a group, some of them would pass and other would fail depending the tox environment. It came to my attention that the time function of this file, was creating an issue with other specs because we were not tearDowning it as expected. Also, using setTime within the setUp group had side effects with unexpected behaviors. Then, the way way that the task_id and task_family was named was also causing problems with the same spec that were failing prior. I'm unsure why this would be the case, but changing either fail, but changing both makes the spec to green. Finally, the last spec would always fail because the setTime was set AFTER the task was actually being run, which would always cause the execution time to be greater than 0. My understanding of all of this is still a bit fuzzy, but hey, now the spec suite passes.
-
Commits on Dec 14, 2018
-
Abstract MetricsCollector class
This will force people to implement this methods of this class when they refer to it.
-
Kwargs on the _send_event call
This allows for less-strict function calls.
-
Commits on Dec 17, 2018
-
-
Change default_tags of DatadogMetricsCollector to a property
The underlying configuration of the Datadog metrics collector is a property, so it makes more sense that it's also a property when used within the class itself.