trace/errors: add a way to handle trace errors #2

clutchski · 2016-06-22T21:07:26Z

adding a standard way of collecting exception information. it adds the following tags:

error.msg      # a human readable sentence of the error
error.stack    # a configurable number of lines of stacktrce (right now 20)
error.type     #  a string representing the type of the error (will be usually null on golang for example)

Here's a rough example of a span with errors:

        id 6676202706048782439
  trace_id 4335547038182019601
 parent_id None
   service None
  resource foo
      type None
     start 1466629552.42
       end
  duration None
     error 1
{'error.msg': u'integer division or modulo by zero',
 'error.stack': u'Traceback (most recent call last):\n  File "/home/vagrant/dd-trace-py/ddtrace/test_span.py", line 63, in test_traceback_with_error\n    1/0\nZeroDivisionError: integer division or modulo by zero\n',
 'error.type': u"<type 'exceptions.ZeroDivisionError'>"}

it also adds a few utility functions which you can see in the PR.

clutchski · 2016-06-22T21:22:20Z

from @LotharSee:

a few things:

In your example, we could instead have 'error.type': 'ZeroDivisionError'
should we convert error to a Bool? and then make it clear in our doc that this attribute is meant to say if that’s a “hit” or an “error” in our stats and display
we might want to report a stack even if we don’t have an error. In which case the “stack” field could be higher level. It is also reasonable to prefix it with the language. Like python.stack.

talwai · 2016-06-23T15:52:22Z

Looks good to me. Worth remembering that the agent will truncate huge meta values so for really deep stack traces there's a chance we swallow the most recent calls (including the one that actually raises the exception). i dug through our sentry a bit and haven't found a raw stacktrace that pushes the limit we have though, so seems like a non-issue

…pan, provider, helpers (#2180) * Enabled type hinting/checking for context, monkey, span, helpers * Type checked provider, addressed PR comments * Attempt #2 to remove circular dependency * Attempt to fix circular import #3 * Attempt to remove circular dependency #4 * Attempt to remove circular dependency #5 * Attempt 6 to remove circular dependency * Revert type checking check in provider.py * Reverted type check checking for tracer.py * Changed mistaken int arg type to float in span.duration setter Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Julien Danjou <julien.danjou@datadoghq.com>

…6119) ## Description Based on the failing test below we can see `test_single_trace_too_large` generates over 20MB of trace data but these traces are sent in separate payloads. This PR increases the time internal interval in the flaky test to ensure the buffer fits the 500 span trace chunks into one payload. ## Background This test consistently passes locally but often fails in ci (and mainly on older versions of python). My hypothesis is that the time interval to submit traces to the agent is too short. The test does not have enough time to add the trace chunks to overflow the httpwriters buffer. This PR increases the time interval of submitting a trace from 10s to 100s to address this hypothesis.`tracer.shutdown()` is called and used to submit payloads to the agent. Sample failure: https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/37860/workflows/4da477db-5289-40b1-8375-5d6152f14b8f/jobs/2546383 ## Testing Strategy YOLO. If `test_single_trace_too_large` is still flaky after this PR then I have failed my reviewers and myself. ## Checklist - [x] Change(s) are motivated and described in the PR description. - [x] Testing strategy is described if automated tests are not included in the PR. - [x] Risk is outlined (performance impact, potential for breakage, maintainability, etc). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) are followed. - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)). - [x] Backport labels are set (if [applicable](../docs/contributing.rst#release-branch-maintenance)) ## Reviewer Checklist - [ ] Title is accurate. - [ ] No unnecessary changes are introduced. - [ ] Description motivates each change. - [ ] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Testing strategy adequately addresses listed risk(s). - [ ] Change is maintainable (easy to change, telemetry, documentation). - [ ] Release note makes sense to a user of the library. - [ ] Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment. - [ ] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](../docs/contributing.rst#release-branch-maintenance)

The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ```

The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374)

…t 2.12] (#11215) Backport 64b3374 from #11167 to 2.12. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.13] (#11214) Backport 64b3374 from #11167 to 2.13. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.14] (#11213) Backport 64b3374 from #11167 to 2.14. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t 2.15] (#11211) Backport 64b3374 from #11167 to 2.15. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` (cherry picked from commit 64b3374) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

…t 2.16] (#11210) Backport 64b3374 from #11167 to 2.16. The ThreadSpanLinks singleton holds the active span (if one exists) for a given thread ID. The `get_active_span_from_thread_id` member function returns a pointer to the active span for a thread. The `link_span` member function sets the active span for a thread. `get_active_span_from_thread_id` accesses the map of spans under a mutex, but returns the pointer after releasing the mutex, meaning `link_span` can modify the members of the Span while the caller of `get_active_span_from_thread_id` is reading them. Fix this by returning a copy of the `Span`. Use a `std::optional` to wrap the return value of `get_active_span_from_thread_id`, rather than returning a pointer. We want to tell whether or not there actually was a span associated with the thread, but returning a pointer would require us to heap allocate the copy of the Span. I added a simplistic regression test which fails reliably without this fix when built with the thread sanitizer enabled. Output like: ``` WARNING: ThreadSanitizer: data race (pid=2971510) Read of size 8 at 0x7b2000004080 by thread T2: #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe46e) #4 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)()) <null> (thread_span_links+0xe2fe) #5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe1cf) #6 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> >::operator()() <null> (thread_span_links+0xe0f6) #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)()> > >::_M_run() <null> (thread_span_links+0xdf40) #8 <null> <null> (libstdc++.so.6+0xd6df3) Previous write of size 8 at 0x7b2000004080 by thread T1 (mutexes: write M47): #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:823 (libtsan.so.0+0x42313) #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:815 (libtsan.so.0+0x42313) #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const&) <null> (libstdc++.so.6+0x1432b4) #3 get() <null> (thread_span_links+0xb570) #4 void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) <null> (thread_span_links+0xe525) #5 std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) <null> (thread_span_links+0xe3b5) #6 void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (thread_span_links+0xe242) #7 std::thread::_Invoker<std::tuple<void (*)()> >::operator()() <null> (thread_span_links+0xe158) [ ... etc ... ] ``` ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Nick Ripley <nick.ripley@datadoghq.com> Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

clutchski added 2 commits June 22, 2016 21:02

trace/errors: add a way to handle trace errors

8c10386

fix test.

ebe6900

errors: readable error type

8692a5c

clutchski merged commit 8692a5c into master Jun 23, 2016

clutchski deleted the matt/errors branch June 23, 2016 18:56

labbati added a commit that referenced this pull request Sep 12, 2018

[tmp] #2

031b8b5

Yun-Kim referenced this pull request in Yun-Kim/dd-trace-py Mar 16, 2021

Attempt #2 to remove circular dependency

b508528

mabdinur mentioned this pull request Oct 6, 2022

Inconsistent span parenting using DDWSGIMiddleware #4296

Closed

mabdinur added a commit that referenced this pull request Sep 28, 2023

attempt #2: make the trace buffer overflow

a4d7fbb

wconti27 mentioned this pull request May 31, 2024

support Anthropic chat models #9420

Closed

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

trace/errors: add a way to handle trace errors #2

trace/errors: add a way to handle trace errors #2

Uh oh!

clutchski commented Jun 22, 2016 •

edited

Loading

Uh oh!

clutchski commented Jun 22, 2016

Uh oh!

talwai commented Jun 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trace/errors: add a way to handle trace errors #2

trace/errors: add a way to handle trace errors #2

Uh oh!

Conversation

clutchski commented Jun 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clutchski commented Jun 22, 2016

Uh oh!

talwai commented Jun 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clutchski commented Jun 22, 2016 •

edited

Loading