Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Datadog contrib for monitoring purpose #2434

Merged
merged 14 commits into from Dec 17, 2018

Commits on Dec 13, 2018

  1. Add Datadog contrib for monitoring purpose

    Datadog is a tool that allows you to send metrics that you create
    dashboard and add alerting on specific behaviors.
    
    Adding this contrib will allow for users of this tool to log their pipeline
    information to Datadog.
    
    Based on the status change of a task, we log that information to Datadog
    with the parameters that were used to run that specific task.
    
    This allow us to easily create dashboard to visualize the health. For
    example, we can be notified via Datadog if a task has failed, or we can
    graph the execution time of a specific task over a period of time.
    
    The implementation idea was strongly based on the stale PR
    spotify#2044.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    8ca4cf9 View commit details
    Browse the repository at this point in the history
  2. Refactor MetricsCollectors in Scheduler

    I've also added a few test to ensure that the implementation was working
    well.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    14c1ebb View commit details
    Browse the repository at this point in the history
  3. Add polish + tests around metrics on task state

    This takes care of ensuring that the proper metrics collection calls are
    being done when they are expected to be happening.
    
    We've also removed a few `@RPC_METHOD` that weren't actually being used
    and that wasn't required.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    bacae6e View commit details
    Browse the repository at this point in the history
  4. Add tests related to the Datadog contrib

    This makes sure that we're properly dispatching API and STATSD call with
    the proper parameter values to Datadog.
    
    This doesn't test all the different possible parameters configuration.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    7668cf0 View commit details
    Browse the repository at this point in the history
  5. Improve configuration documentation with new Datadog contrib

    This adds a few extra documentation line for the configuration to allow
    user to find all the settings they can tweak for each individual
    contribs instead of having to go through each individual contrib files.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    6556b3c View commit details
    Browse the repository at this point in the history
  6. Update DataDog dep. to the most recent release

    The original implementation was made when 0.16.0 was the latest version.
    Since there there have been a few improvements and bug fixes made to the
    library that we should be using.
    
    Reading through the release log there shouldn't be any feature-breaking
    changes so we should be good to update it!
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    f4e7965 View commit details
    Browse the repository at this point in the history
  7. Change metrics collection getter to class method

    Previously, the getter wasn't a class method and wouldn't work as
    expected. In order to ensure that the output is what we expect, we've
    added more tests.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    a67f2af View commit details
    Browse the repository at this point in the history
  8. Fix spec issues related to new tests

    There was multiple problems that needed to be solved in order to get the
    specs green again. Each individual specs were passing when ran
    individually, but when ran into tox as a group, some of them would pass
    and other would fail depending the tox environment.
    
    It came to my attention that the time function of this file, was
    creating an issue with other specs because we were not tearDowning it as
    expected. Also, using setTime within the setUp group had side effects
    with unexpected behaviors.
    
    Then, the way way that the task_id and task_family was named was also
    causing problems with the same spec that were failing prior.  I'm unsure
    why this would be the case, but changing either fail, but changing both
    makes the spec to green.
    
    Finally, the last spec would always fail because the setTime was set
    AFTER the task was actually being run, which would always cause the
    execution time to be greater than 0.
    
    My understanding of all of this is still a bit fuzzy, but hey, now the
    spec suite passes.
    thisiscab committed Dec 13, 2018
    Copy the full SHA
    e8fafd5 View commit details
    Browse the repository at this point in the history
  9. Copy the full SHA
    7756b37 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2018

  1. Abstract MetricsCollector class

    This will force people to implement this methods of this class when they
    refer to it.
    thisiscab committed Dec 14, 2018
    Copy the full SHA
    a7f76fc View commit details
    Browse the repository at this point in the history
  2. Kwargs on the _send_event call

    This allows for less-strict function calls.
    thisiscab committed Dec 14, 2018
    Copy the full SHA
    5ba5917 View commit details
    Browse the repository at this point in the history
  3. Fix metrics collector

    thisiscab committed Dec 14, 2018
    Copy the full SHA
    d387b99 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2018

  1. Copy the full SHA
    e876d9c View commit details
    Browse the repository at this point in the history
  2. Change default_tags of DatadogMetricsCollector to a property

    The underlying configuration of the Datadog metrics collector is a
    property, so it makes more sense that it's also a property when used
    within the class itself.
    thisiscab committed Dec 17, 2018
    Copy the full SHA
    d43c478 View commit details
    Browse the repository at this point in the history