trace postgres connect method

[jontonsoup]

Hi friends,

I've been trying to do some heavy profiling of a python program and have been trying to hunt down a 15 millisecond lag at the beginning of my processes.

The tracer shows nothing.

In an act of desperation, I moved over to new relic, which shows the offender is the psycopg2 connect function. This is monkey patched by the tracer, and I suspect the tracer itself is the cause of the slowdown (although I am unsure and removing it now).

1. You should report this -- having mystery untraced time is not good
2. How do we know when the tracer is the actual cause of the problem?

Thanks!

[clutchski]

@jontonsoup we can add connect to the tracing (though often times thye happen outside the scope of a request). I don't see tracing adding 15 ms of setup overhead, but can work through it with you.

can you take a cProfile of your program startup?

[jontonsoup]

@clutchski We already moved off dd-trace because the threading support just wasn't where we need it to be.

That said, the larger issue was that connecting to the DB took 15ms and dd-trace did not report this-- could have been latency or something else (we ended up fixing it by moving connect out of the hot path), but we would have never known if we didn't install new relic.

The issue is that the slowdown was in the datadog patch function itself. Just bringing up that this might be an issue you want to consider in the general case.

[thehesiod]

should be pretty simple, I recently added something similar to my aiohttp client tracing PR. If this gets added make sure to also add to aiopg + asyncpg contrib modules. Also note the complexity of time to connect, vs time to get connection from pool (may have to wait for connection from pool to become available).

[brettlangdon]

The problem I see with tracing the connect method is that it is most likely going to occur outside the context of a normal trace, meaning it'll be hard to correlate the effects of connect on any given trace.

Closing this ticket for now due to it's age, please feel free to re-open if needed.