Client sampling #8

LotharSee · 2016-06-29T16:04:41Z

Introduce client-sampling (or "pre-sampling").

Explanation of how it will work: https://trello.com/c/xX3p3CyO/390-client-library-sampling

The Sampler keeps (100 * sample_rate)% of the traces.
Any sampled trace should be entirely ignored by the instrumentation and won't be written.
It samples randomly, its main purpose is to reduce the instrumentation footprint.
A field weight is added to the span, allowing us to update stats accordingly.

Update existing integrations not to instrument when a trace is sampled.
We might want to have a nicer interface for that (since most of the time the instrumentation follows the same pattern), but that could wait.
Note that this test on span.sampled is just to improve the performance. It works to create, tag and report a sampled span/trace (but it won't be written).

clutchski · 2016-06-29T16:30:18Z

ddtrace/contrib/elasticsearch/transport.py

            """
            with self._datadog_tracer.trace("elasticsearch.query") as s:
+                # Don't instrument if the trace is sampled
+                if s.sampled:


i don't think this should be mandatory. in fact, people might depend on it for logging, etc. if it were me, sampling would only be for submission.

LeoCavaille · 2016-06-30T10:30:39Z

ddtrace/contrib/flask/middleware.py

+                span.set_tag(http.URL, str(request.base_url or ""))
+                span.set_tag(http.STATUS_CODE, code)
+                span.error = error
            span.finish()


shouldn't this be in the if block as well ?

The span already got created. Even if it is not sampled (and we didn't put the various attributes), we still have to finish it (to keep a valid tree).

LeoCavaille · 2016-06-30T10:39:03Z

Other than that 👍

LotharSee · 2016-06-30T10:57:58Z

Thanks for the review!

The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ```

The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374)

…t 2.12] (#11215) Backport 64b3374 from #11167 to 2.12. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.13] (#11214) Backport 64b3374 from #11167 to 2.13. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.14] (#11213) Backport 64b3374 from #11167 to 2.14. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.15] (#11211) Backport 64b3374 from #11167 to 2.15. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

…t 2.16] (#11210) Backport 64b3374 from #11167 to 2.16. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Nick Ripley <nick.ripley@datadoghq.com> Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

## Description `test_wrapper()` segfaults with following stack backtrace ``` #3 handle_posix_sigaction () at src/collector/crash_handler.rs:108 #4 <signal handler called> #5 0x000075fb182a92b9 in Datadog::Sample::push_release (count=1, lock_time=3896666036706324, this=<optimized out>) at /home/bits/project/ddtrace/internal/datadog/profiling/dd_wrapper/src/sample.cpp:235 #6 Datadog::Sample::push_release (this=<optimized out>, lock_time=3896666036706324, count=1) at /home/bits/project/ddtrace/internal/datadog/profiling/dd_wrapper/src/sample.cpp:231 #7 0x000075fb182bec5c in __pyx_pf_7ddtrace_8internal_7datadog_9profiling_4ddup_5_ddup_12SampleHandle_10push_release ( __pyx_v_count=<optimized out>, __pyx_v_value=0x75fb187c7250, __pyx_v_self=0x75fb187c7630) at /tmp/tmpxso4scef.build-lib/ddtrace.internal.datadog.profiling.ddup._ddup/_ddup.cpp:7956 #8 __pyx_pw_7ddtrace_8internal_7datadog_9profiling_4ddup_5_ddup_12SampleHandle_11push_release ( __pyx_v_self=0x75fb187c7630, __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at /tmp/tmpxso4scef.build-lib/ddtrace.internal.datadog.profiling.ddup._ddup/_ddup.cpp:7907 #9 0x000075fb1c35b495 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x59d0af703300, ``` As it indexes into empty `values` [vector](https://github.com/DataDog/dd-trace-py/blob/d033f37c48994b061a00748a00724b54a110de39/ddtrace/internal/datadog/profiling/dd_wrapper/include/sample.hpp#L81). It is typically initialized by a call to `ddup.config()`. Shuffle the test code into `TestThreadingLockCollector` class to do so.  ## Testing Run `CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) DD_FAST_BUILD=1 DD_TRACE_AGENT_URL=http://localhost:8126 riot -v run 9bb19fe --pass-env -- -s -vv -k test_wrapper`  ## Risks  ## Additional Notes  [PROF-12678](https://datadoghq.atlassian.net/browse/PROF-12678) [PROF-12678]: https://datadoghq.atlassian.net/browse/PROF-12678?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

LotharSee added 10 commits June 28, 2016 19:11

Introduce client-side sampling with sample_rate

39ac2d1

Skip ES instrumentation when sampled

69b5673

Skip Flask instrumentation when sampled

d4f900b

Skip Psycopg instrumentation when sampled

b597b68

Skip Sqlite instrumentation when sampled

113097e

Skip Pylons instrumentation when sampled

585d9fe

Define 'set_sample_rate' in Trace

4ec676c

Simplify 'reporter.py' file

83715d0

Add weight attribute to all spans

2c865f1

Add a test on span.weight value

8af16fd

clutchski reviewed Jun 29, 2016
View reviewed changes

Rename Sampler to RateSampler, improve its interface

ea55443

LeoCavaille reviewed Jun 30, 2016
View reviewed changes

Redefine span.sampled as span being kept

fc0983d

LotharSee merged commit 6cbea11 into master Jun 30, 2016

LotharSee deleted the benjamin/client-sampling branch July 5, 2016 14:33

labbati added a commit that referenced this pull request Sep 13, 2018

[tmp] #8

fbfaeed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Client sampling #8

Client sampling #8

Uh oh!

LotharSee commented Jun 29, 2016

Uh oh!

clutchski Jun 29, 2016 •

edited

Loading

Uh oh!

LeoCavaille Jun 30, 2016

Uh oh!

LotharSee Jun 30, 2016

Uh oh!

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee commented Jun 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Client sampling #8

Client sampling #8

Uh oh!

Conversation

LotharSee commented Jun 29, 2016

Uh oh!

clutchski Jun 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeoCavaille Jun 30, 2016

Choose a reason for hiding this comment

Uh oh!

LotharSee Jun 30, 2016

Choose a reason for hiding this comment

Uh oh!

LeoCavaille commented Jun 30, 2016

Uh oh!

LotharSee commented Jun 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

clutchski Jun 29, 2016 •

edited

Loading