Matt/patch by clutchski · Pull Request #75 · DataDog/dd-trace-py

clutchski · 2016-10-16T15:30:37Z

Overview

Experiments in how to monkey patch. This is experimental, but is aimed at producing good
patterns we can use in all integrations.

Goals

a one liner to trace everything just works from ddtrace import patch; patch()
easy for devs to distinguish their services that use the same technology (e.g. replica-pg from master-db cluster
pro users have more control:
- can autotrace selected libraries (whitelist, blacklist, etc)
- can trace instances of a connection without tracing everything.
the code is easy to read, understand & test for datadogs and customers alike

Strategies

Proxies

I think we have two options here:

use object proxies around core objects
this is nice because people always have access to the internal connection / whatever
very easy to patch/unpatch
only nice when the patched methods aren't called internally
works nicely on classes and instances'
works well for libs with factory functions psycopg2.connect()
we can add extra attributes (e.g. service name) to cstructs that don't support this (e.g. postgres)

the dbapi / psycopg example is a good example of how this works. all the following is open to api changes, ust more about illustrating the concepts.

patch everything:

from contrib.psycopg2 import patch
patch.patch()  #patches everything

patch one connection:

from contrib.psycopg2 import patch
conn = psycopg2.connect(...)

conn = patch.wrap(conn)  # wraps one instance

set service metadata

from contrib.psycopg2 import patch
from contrib.dbapi import patch

patch.patch()  #patches everything

conn = psycopg2.connect()
dbapi.configure(conn, service="foobar")
# or
setattr(conn, 'datadog_service, 'foobar')

Patching Functions

works well for libs with where people are instantiating classes directly redis.Redis
very good for functions which call functions we need to trace internally
pretty simple
hard to unmonkeypatch

monkey patch everything:

# monkey patch evey
from ddrace.contrib.redis import patch
patch.patch() # monkeypatch everything

monkeypatch an instance:

# monkey patch every
from ddrace.contrib.redis import patch
patch.patch_target(redis.Redis())

Others?

Let me know :)

it can then be easily shared between sqlite and postgres.

LotharSee · 2016-10-28T10:07:02Z

@@ -19,10 +19,10 @@

 POSTGRES_CONFIG = {


These changes should not be here. Just run docker-compose up -d to test locally.

yea i know. i was on the train and had no internet :)

LotharSee · 2016-10-28T11:12:42Z

 from ddtrace.contrib.autopatch import autopatch

+# allow logging
+logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)


Don't hard-code it ; if you want more logging, just use the --logging-* options of nosetests.

yes this is just a temporary testing script. not a nose test.

LotharSee · 2016-10-28T13:00:54Z

+    service = getattr(instance, 'datadog_service', None) or 'redis'
+    tracer = getattr(instance, 'datadog_tracer', None) or ddtrace.tracer
+    with tracer.trace('redis.cmd'):
+        print 'trace'


Various prints, let's not forget to remove them.

yea this is just a hack for now. not tested, not complete.

LotharSee · 2016-10-28T13:05:23Z

+
+import redis
+
+def patch():


This isn't not compatible with our current Redis instrumentation.
The trick is that our Redis instrumentation has to patch instances instead of the global module ; tricker, but that's a pattern which will happen again on other libs.

yes, it's a replacement. not complete, but meant to work on instances, classes, etc.

LotharSee · 2016-10-28T13:31:28Z

 ]


 def autopatch():


For the "autopatch", I suggest to follow the gevent.monkey interfaces

Same naming convention

Optional options to patch_all to control individual libs

But then, the "internals" behind it will be different (out patching is more complex), but should be similar across contribs (like this patch function).

open to whatever naming conventions

talwai · 2016-11-01T21:46:00Z

+
+def _get_module_name(conn):
+    # there must be a better way
+    return str(type(conn)).split("'")[1].split('.')[0]


if this is doing what i think it's trying to do, i like this better:

>>> import psycopg2 >>> conn = psycopg2.connect(host="localhost",port=5432) >>> conn.__class__.__module__ 'psycopg2._psycopg' >>> conn.__class__.__module__.split('.')[0] 'psycopg2'

talwai · 2016-11-01T21:55:25Z

+    service_info.set_on(target)
+
+    # monkeypatch all of the methods.
+    targets = [


target / targets is confusing. call it to_patch

talwai · 2016-11-01T21:57:33Z

+
+    def __init__(self, service, app=None, tracer=None):
+        self.service = service
+        self._app = app


where is this used?

talwai · 2016-11-01T22:02:10Z

+class ServiceInfo(object):
+
+    @classmethod
+    def get_from(self, obj):


can't this just be static?

talwai · 2016-11-01T22:03:36Z

+
+def patch_target(target, service_info=None):
+    if not service_info:
+        service_info = ServiceInfo(name="redis", app="redis")


does this work? ServiceInfo doesn't take a name

default to postgres, sqlite, ec.

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

clutchski added 7 commits October 14, 2016 08:55

autopatch: pyscopg experiments

fd86fa4

autopatch: add dbapi interface patching

6fc31a5

it can then be easily shared between sqlite and postgres.

plug psycopg and sqlite into autopatch

51edc78

experiments with redis autopatching.

defdfec

autopatch: make autopatching safer.

3b6c050

psycopg: add tests for patched version of conn

90083a8

psycopg: fix indentaiton

66facae

LotharSee reviewed Oct 28, 2016

View reviewed changes

clutchski added 5 commits October 31, 2016 13:48

patch/redis: add tests for monkeypatched command

eb53430

patch/redis: add pipeline tracing

3d38b39

patch/redis: implement pipeline execute immediate.

bb5ccb0

patch: lint fixes

cf02884

config: restore original postgres config

99e5802

talwai reviewed Nov 1, 2016

View reviewed changes

patch: remove uneeded code

6dc3c01

talwai reviewed Nov 1, 2016

View reviewed changes

clutchski added 7 commits November 1, 2016 22:32

autopatch: remove cruft

cbe1cba

sqlite: add conn.execute traced method

f24b89d

patch/sqlite: ensure we patch all connect methods

086fbac

redis/patch: fix service info command

083f4be

patch/dbapi: clean up default service names.

1a1c9c2

default to postgres, sqlite, ec.

dbapi: fix module naming

5b74a58

patch/psycopg: share the connection wrapper

a4ccbd6

patch: high level interface & lock

7a5e7be

clutchski mentioned this pull request Nov 4, 2016

Patch #92

Merged

6 tasks

clutchski closed this Nov 4, 2016

clutchski deleted the matt/patch branch January 25, 2017 21:10

Conversation

clutchski commented Oct 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Goals

Strategies

Proxies

Patching Functions

Others?

Uh oh!

LotharSee Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

talwai Nov 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clutchski commented Oct 16, 2016 •

edited

Loading

LotharSee Oct 28, 2016 •

edited

Loading

talwai Nov 1, 2016 •

edited

Loading