[statsd] Add ability to toggle statsd.disable_buffering state during runtime
#700
Conversation
Since `initialize` uses the module-level DogStatsd instance that is only initialized once, we cannot rely on new instantiation for changes to the `disable_buffering` flag. This change allows the consumers to call `stats.disable_buffering = <bool>` with runtime changes being applied to the module-level instance. By doing this, we can in future changes allow `initialize()` to change this flag if meeded.
Now that the DogStatsd module can handle live toggling of buffering switches, we expose this argument in the `datadog.initialize()`. While not terribly useful currently, it will be very valuable if/when buffering is enabled on the DogStatsd module by default (again).
sgnn7
added
resource/dogstatsd
changelog/Added
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
| @@ -37,6 +37,7 @@ def initialize( | |||
| api_host=None, # type: Optional[str] | |||
| statsd_host=None, # type: Optional[str] | |||
| statsd_port=None, # type: Optional[int] | |||
| statsd_disable_buffering=True, # type: bool | |||
There was a problem hiding this comment.
Is it named disable_buffering because the plan is to have the buffering enabled by default in the future? If that's the case, don't mind the following remark.
Having a parameter "disabling" something (and defaulting to True) is always weird to think about opposed to a parameter "enabling" a feature.
There was a problem hiding this comment.
Is it named disable_buffering because the plan is to have the buffering enabled by default in the future
Correct. The long story is that it's currently a bit of an awkward flag name like you mention because we enabled-then-disabled buffering-by-default already and the flag that was already in the API was disable_buffering. If/when we re-enable buffering-by-default, this flag will be a better indication of the functionality it's just that for now we're a bit stuck with a mid-way change in architectural direction and the flag was already named in the published API.
|
|
||
| self._flush_thread_stop.set() | ||
|
|
||
| self._flush_thread.join() |
There was a problem hiding this comment.
Not really a big deal I guess but if two threads are calling _stop_flush_thread simultaneously, there is an small chance that join() could be called on one of the thread while the _flush_thread has already be terminated by the other.
There was a problem hiding this comment.
True that this method isn't thread-safe in isolation but the comment on the method mentions this as a warning and the only caller is lock-protected. I would have preferred to use the lock directly in this method like you imply but then the toggle to stop would needed 2 lock acquires (1 for toggle method and 1 for stop method; possibly a reentrant one) which seemed like an overkill and over-complication for a helper only called from a single spot.
What does this PR do?
statsd_instance.disable_bufferingflag after initializationinitialize()to be usable for the module-levelstatsdinstanceDescription of the Change
While not likely to be useful currently, if/when we enable buffering on the
DogStatsdclientsby default again we will want to have this feature already in place since the module level
statsdmay not behave as expected in
fork()-based environments. Becausedatadog.initialize()cannotrecreate the instance of the module-level
statsd, we need a runtime setting fordisable_bufferinginstead of recreating the instance, making the implementation somewhat constrained in how it is
done.
As a side-effect, this change allow use of the module-level
statsdinstance with buffering enabledwhich was previously not possible.
Alternate Designs
None that would make sense with the limitations provided.
Possible Drawbacks
Assumptions were made in the logic that
datadog.initialize()would be only called once forthe application. While there are protections around concurrent invocations, there could be edge
cases not covered if multiple concurrent calls to `initialize are made to change this flag or if the
data is being added to the buffer while the buffering is being disabled.
Verification Process
Tests covers the underlying functionality of this feature but it can be tested with:
datadog.initialize()and settingstatsd_disable_bufferingflagDogStatsdAdditional Notes
This feature will be gradually documented as it is expected to be
useful only in very niche high-throughput cases.
Release Notes
statsd.disable_bufferingstate during runtimestatsd_disable_bufferingas an argument todatadog.initialize()Review checklist (to be filled by reviewers)
changelog/label attached. If applicable it should have thebackward-incompatiblelabel attached.do-not-merge/label attached.kind/andseverity/labels attached at least.