Fix multi-instance service discovery

[aerostitch]

Fixes: #3340

As explained in #3340, having multiple instances for the same check in the annotations don't work well.
This brings the few adjustments to make it work.

The doc will need to be updated to say that if you want several instances of the same check in the annotations, you'll need to have an array for then inside the annotations array.

[aerostitch]

@hkaj Do you think we could setup this one for 5.14?
Thanks! :)

[aerostitch]

Note: I've got this running in one of my environments. Works fine with a multi-instance declared via an annotation.

[aerostitch]

Note: this will also require a fix for the page https://docs.datadoghq.com/guides/autodiscovery/
The current examples should be adjusted to be proper arrays.

[aerostitch]

Hi @hkaj do you think we can plan that one for 5.14?
Thanks

[hkaj]

Hi @aerostitch !
Thanks for the PR. It's an interesting feature. The freeze for 5.14 is tomorrow so it's likely going to be pushed to 5.15 though.

I left one comment, let me know what you think. Also merge conflicts need to be fixed. We'll take care of the documentation.

Thanks again!

[hkaj]

I think we should rather apply the same logic as here: https://github.com/DataDog/dd-agent/pull/3341/files#diff-8437fe4d4416c056af154d55fb2993c2R372
You can do something like

if not result_init_config:
    result_init_config = init_config
elif result_init_config != init_config:
    self.log.warning("Different versions of `init_config` found for "
        "check {}. Keeping the first one found.".format('check_name'))

This will avoid unclear errors in case init_configs diverge and values are overriden by later ones. Not sure if it can happen right now but that silent update makes me nervous. Or do you see a legit use case for it?

[aerostitch]

Makes sense @hkaj for the requested change. I rebased & updated the PR waiting for travis to start the build for about 1h now.
I don't really think it's a feature, more a bug no?
If 5.14 is frozen, shouldn't we have that in 5.14.1?

[hkaj]

Thanks, LGTM. Will merge once 5.14.x is branched off. It's more of a feature because it was explicitly designed for a single instance and that was the only configuration that was supported. Moved it to 5.14.1.

[aerostitch]

Thanks a lot! :)