matt/pylibmc by clutchski · Pull Request #63 · DataDog/dd-trace-py

clutchski · 2016-09-16T18:03:03Z

A pylibmc integration.

particular thoughts

flask cache testing needs a little bit of love. it doesn't distinguish between being able to test memcached or pylibmc.

general thoughts

I want to solidify on using an object proxy instead of monkey patching. I think it's very nice that the user can do this:

c1 = pylibmc.Client(...)
c2 = TracedClient(...)

and have a clean slate to use either the traced client or the untouched original client. It just feels good, makes sure other things can't get stomped, etc.

the only downside of proxies is that it might be difficult to implement in some cases, but when possible we should opt for this route. thoughts?

add a pylibmc integration

makes the scoped envs much faster to install and able to run without dev bindings.

or a bug in the library with particular versions of the libmemcache c library? no biggie.

LotharSee

Since with Redis, this will cover the main kvs situations, it's certainly the moment to unify: span types and metadata (metadata keys, how we format or truncate too long queries, ...).
That's a spec we will need for the UI too.

LotharSee · 2016-09-17T10:24:49Z

@@ -0,0 +1,4 @@
+
+from .client import TracedClient # flake8: noqa


By putting it in __all__, you don't need the flake8 exception.
Also, you can copy the logic used in other integrations to manage the presence of the lib (and not polluting the core logic with that)

i actually don't think i need all or import protection here. since it's a proxy the tracer doesn't actually depend on the library.

LotharSee · 2016-09-17T10:25:19Z

+translate_server_specs = None
+
+try:
+    from pylibmc.client import translate_server_specs


For which reason would this be missing? Specific client version? Might want to leave a comment about that.

just because it' an internal method and not part of the documented API. will comment.

LotharSee · 2016-09-17T10:32:47Z

+    def _trace(self, method_name, *args, **kwargs):
+        """ trace the execution of the method with the given name. """
+        method = getattr(self.__wrapped__, method_name)
+        with self._span(method_name):


What about using args to put in the meta the full request? (like multi_get key1 key2). Not that useful otherwise.

there is no built-in namespace, etc so i don't think we can add anything to the resource.

Sorry I wasn't clear, I meant to put it in a metadata, like what we do with Redis https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/redis/tracers.py#L110 the resource is good as it is.

LotharSee · 2016-09-17T10:37:42Z

-        # but at least we don't raise an exception
-        eq_(span.error, 0)
+
+        # pylibmc raises an exception, memcached doesn't, so don't test that.


"so don't test span.error"

Yeah, while writing the flask_cache integration I was surprised about how memcached driver handles errors/exceptions.

LotharSee · 2016-09-17T10:39:31Z

+
+    def _tag_span(self, span):
+        # FIXME[matt] the host selection is buried in c code. we can't tell what it's actually
+        # using, so fallback to randomly choosing one. can we do better?


We probably can't, and we might even prefer to know which "cluster" we hit instead of which specific host?
That's an issue we will hit with various DBs, we should think about a proper global way to handle these.

yea i'll think about this. not sure yet.

palazzem · 2016-09-19T15:45:45Z

-
+    elif hasattr(client, "addresses"):
+        # pylibmc
+        # FIXME[matt] should we memoize this?


can't we perf test that? If we see an overhead we can cache the value using the built-in in-memory stuff

) ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This fixes a rare segmentation fault that could occur when a profiled application forks. #### Stack trace ``` #0 0x0000ffffaf48f6d0 free #1 0x0000ffffae4367e4 core::ptr::drop_in_place<indexmap::set::IndexSet<libdd_profiling::internal::stack_trace::StackTrace,core::hash::BuildHasherDefault<rustc_hash::FxHasher>>>::hbe422e96ad1ad3f9 #2 0x0000ffffae436478 core::ptr::drop_in_place<libdd_profiling::internal::profile::Profile>::h4a6aee0579496bfb #3 0x0000ffffae43e244 ddog_prof_Profile_drop #4 0x0000ffff9fe5a724 Datadog::Profile::postfork_child #5 0x0000ffffaf4b8804 __libc_fork #6 0x0000ffffaf6bf058 os_fork_impl (/usr/src/python/./Modules/posixmodule.c:7757) #7 0x0000ffffaf6bf058 os_fork (/usr/src/python/./Modules/clinic/posixmodule.c.h:3986) #8 0x0000ffffaf725d2c cfunction_vectorcall_NOARGS (/usr/src/python/Objects/methodobject.c:481:24) #9 0x0000ffffaf72f420 PyCFunction_Call (/usr/src/python/Objects/call.c:387:12) #10 0x0000ffffaf72f420 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #11 0x0000ffffaf750430 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #12 0x0000ffffaf750430 object_vacall (/usr/src/python/Objects/call.c:850:14) #13 0x0000ffffaf7cf1b8 PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:957:14) #14 0x0000ffffae18453c WraptFunctionWrapperBase_call (/project/src/wrapt/_wrappers.c:2455:14) #15 0x0000ffffaf7225c4 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #16 0x0000ffffaf72db54 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #17 0x0000ffffaf725b60 _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:419:16) #18 0x0000ffffaf725b60 _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:133:15) #19 0x0000ffffaf75b928 _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:508:24) #20 0x0000ffffaf75b928 slot_tp_init (/usr/src/python/Objects/typeobject.c:9026:15) #21 0x0000ffffaf722878 type_call (/usr/src/python/Objects/typeobject.c:1679:19) #22 0x0000ffffaf7225c4 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #23 0x0000ffffaf72db54 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #24 0x0000ffffaf775c74 _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:419:16) #25 0x0000ffffaf775c74 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #26 0x0000ffffaf775c74 method_vectorcall (/usr/src/python/Objects/classobject.c:91:18) #27 0x0000ffffaf72f420 PyCFunction_Call (/usr/src/python/Objects/call.c:387:12) #28 0x0000ffffaf72f420 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #29 0x0000ffffaf7745a4 _PyEval_EvalFrame (/usr/src/python/./Include/internal/pycore_ceval.h:89:16) #30 0x0000ffffaf7745a4 gen_send_ex2 (/usr/src/python/Objects/genobject.c:230:14) #31 0x0000ffffaf78cf48 gen_iternext (/usr/src/python/Objects/genobject.c:603:9) #32 0x0000ffffaf78cf48 builtin_next_impl (/usr/src/python/Python/bltinmodule.c:1510) #33 0x0000ffffaf78cf48 builtin_next (/usr/src/python/Python/clinic/bltinmodule.c.h:730) #34 0x0000ffffaf730b30 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2938:20) #35 0x0000ffffaf775d28 _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:419:16) #36 0x0000ffffaf775d28 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #37 0x0000ffffaf775d28 method_vectorcall (/usr/src/python/Objects/classobject.c:61:18) #38 0x0000ffffaf75f614 _PyVectorcall_Call (/usr/src/python/Objects/call.c:283:24) #39 0x0000ffffaf72f420 PyCFunction_Call (/usr/src/python/Objects/call.c:387:12) #40 0x0000ffffaf72f420 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #41 0x0000ffffaf775d28 _PyFunction_Vectorcall (/usr/src/python/Objects/call.c:419:16) #42 0x0000ffffaf775d28 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #43 0x0000ffffaf775d28 method_vectorcall (/usr/src/python/Objects/classobject.c:61:18) #44 0x0000ffffaf75f614 _PyVectorcall_Call (/usr/src/python/Objects/call.c:283:24) #45 0x0000ffffaf72f420 PyCFunction_Call (/usr/src/python/Objects/call.c:387:12) #46 0x0000ffffaf72f420 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3263:26) #47 0x0000ffffaf725bd8 _PyObject_FastCallDictTstate (/usr/src/python/Objects/call.c:144:15) #48 0x0000ffffaf75bc30 _PyObject_Call_Prepend (/usr/src/python/Objects/call.c:508:24) #49 0x0000ffffaf831e40 slot_tp_call (/usr/src/python/Objects/typeobject.c:8782) #50 0x0000ffffaf722678 _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:240:18) #51 0x0000ffffaf72db54 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #52 0x0000ffffaf7cb914 PyEval_EvalCode (/usr/src/python/Python/ceval.c:578:21) #53 0x0000ffffaf7f6af8 builtin_exec_impl (/usr/src/python/Python/bltinmodule.c:1096) #54 0x0000ffffaf7f6af8 builtin_exec (/usr/src/python/Python/clinic/bltinmodule.c.h:586) #55 0x0000ffffaf74844c cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:438:24) #56 0x0000ffffaf747804 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:92:11) #57 0x0000ffffaf747804 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325:12) #58 0x0000ffffaf72db54 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2715:19) #59 0x0000ffffaf811728 pymain_run_module (/usr/src/python/Modules/main.c:300) #60 0x0000ffffaf810a78 pymain_run_python (/usr/src/python/Modules/main.c:627) #61 0x0000ffffaf810a78 Py_RunMain (/usr/src/python/Modules/main.c:713) #62 0x0000ffffaf7b321c Py_BytesMain (/usr/src/python/Modules/main.c:767:12) #63 0x0000ffffaf427818 __libc_start_main #64 0x0000aaaae4960870 _start ``` #### Root cause The root cause of this crash seems to be unfortunate timing on fork. What happened was that, due to ordering of fork handlers, `stack`'s post-fork hook would run _before_ `dd_wrapper`'s. On multi-core systems where we got unlucky, because `stack`'s post-fork hook would restart the Sampling Thread, this meant the Sampling Thread could start before `dd_wrapper`'s post-fork hook had completed (or even before it had started). As a result, `dd_wrapper` would (try to) reset the `Profile` object that the Sampling Thread was writing to through the `Sample` APIs, resulting in a race and all kinds of fun memory issues, such as this crash. Note that the other way around could also happen, where the Sampling Thread could try to write to the `Profile` object that had already been dropped/freed by `dd_wrapper`. #### Are other Profilers impacted? AFAIK, other Profilers shouldn't be impacted as they run in the Main Thread, so I think the risk that they may be writing to the Profile before it's ready post-fork is effectively inexistent. #### Is registering this _at library load time_ OK? As far as I can tell, there is no risk in registering that post-fork hook at library load time, instead of when the Stack Profiler is started. What `stack_atfork_child` is calling `stack_postfork_cleanup`, which we already did at library load time [meaning it's safe to do even when the Profiler doesn't run], then calling `Sampler::restart_after_fork`, which only restarts the Profiler if it was running pre-fork (which in an app that does not use the Profiler is a no-op). Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

clutchski added 3 commits September 16, 2016 17:40

span: add name to pprint

c301afa

pylibmc: initial commit

ef8ca5a

add a pylibmc integration

pylibmc: testing fixes

6844a7d

clutchski assigned palazzem and LotharSee Sep 16, 2016

clutchski added 7 commits September 16, 2016 18:11

pylibmc: fix tox deps

f0719f4

pylibmc: use _eq for testing

28b6f05

memcached: ensure flask-cache can handle pylibmc conns

96bd825

pylibmc: flake8

b67dfeb

tox: only install cass & pg in envs that need it

3d90009

makes the scoped envs much faster to install and able to run without dev bindings.

pylibmc: skip a test that reveals a client bug

caa883f

or a bug in the library with particular versions of the libmemcache c library? no biggie.

tox: be less aggressive about loading libs

2558e93

LotharSee requested changes Sep 17, 2016

View reviewed changes

clutchski added 3 commits September 18, 2016 17:46

pylibmc: add clarifying comments

bf2a622

pylibmc: add docs

4be5cd4

pylibmc: add query metadata

9def4d5

palazzem reviewed Sep 19, 2016

View reviewed changes

clutchski merged commit 9def4d5 into master Sep 19, 2016

clutchski deleted the matt/pylibmc branch September 19, 2016 15:57

		@@ -0,0 +1,4 @@

		from .client import TracedClient # flake8: noqa

Conversation

clutchski commented Sep 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

particular thoughts

general thoughts

Uh oh!

LotharSee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clutchski commented Sep 16, 2016 •

edited

Loading