sql: add pprof labels based on the statement type

[jordanlewis]

pprof profiles will now be enhanced with the type of the statement that
caused the creation of each stack. statement types are things like
'SELECT', 'INSERT', 'ALTER TABLE', etc.

This example output is produced based on a kv95 workload:

(pprof) tags
 stmt: Total 27.7s
       25.4s (91.77%): SELECT
        2.3s ( 8.23%): INSERT

Release note: None

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[jordanlewis]

Here's a sample profile created with the new tags:

pprof.samples.cpu.007.pb.gz

[jordanlewis]

Here's a sample profile created with the new tags:

pprof.samples.cpu.007.pb.gz

[jordanlewis]

https://rakyll.org/profiler-labels/ has more details on how the labels work, btw.

[jordanlewis]

https://rakyll.org/profiler-labels/ has more details on how the labels work, btw.

[tschottdorf]

Just a drive by question, what's the cost of a label? Are they cheap enough to throw in basically as we please?

[tschottdorf]

Just a drive by question, what's the cost of a label? Are they cheap enough to throw in basically as we please?

[jordanlewis]

They seem very cheap, but I don't know about using them more than on the order of once per query. They do a context.Value call and call a runtime function that sets a field on a pointer.

[jordanlewis]

They seem very cheap, but I don't know about using them more than on the order of once per query. They do a context.Value call and call a runtime function that sets a field on a pointer.

[nvanbenschoten]

Same question as Tobi. This is really neat, but I'd like to experimentally confirm that the overhead is completely negligible before making the change.

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/sql/conn_executor_exec.go, line 26 at r1 (raw file):

	"github.com/pkg/errors"

	pprof "runtime/pprof"

Weird import.

[jordanlewis]

Switching to pprof-labels~
+ make bench PKG=./pkg/bench BENCHTIMEOUT=5m 'BENCHES=BenchmarkSelect1/^Cockroach$$' 'TESTFLAGS=-count 10 -benchmem'
Switching to pprof-labels
+ make bench PKG=./pkg/bench BENCHTIMEOUT=5m 'BENCHES=BenchmarkSelect1/^Cockroach$$' 'TESTFLAGS=-count 10 -benchmem'
name                 old time/op    new time/op    delta
Select1/Cockroach-8     126µs ± 1%     140µs ±13%  +10.74%  (p=0.000 n=9+9)

name                 old alloc/op   new alloc/op   delta
Select1/Cockroach-8    18.8kB ± 0%    19.2kB ± 0%   +2.22%  (p=0.000 n=7+9)

name                 old allocs/op  new allocs/op  delta
Select1/Cockroach-8       164 ± 0%       169 ± 0%   +3.05%  (p=0.000 n=10+10)

Pretty disappointing. Not sure this is worth it. Though the Do call is high level and always allocates something - let me see if I can improve the performance by doing things with lower level calls.

[jordanlewis]

Switching to pprof-labels~
+ make bench PKG=./pkg/bench BENCHTIMEOUT=5m 'BENCHES=BenchmarkSelect1/^Cockroach$$' 'TESTFLAGS=-count 10 -benchmem'
Switching to pprof-labels
+ make bench PKG=./pkg/bench BENCHTIMEOUT=5m 'BENCHES=BenchmarkSelect1/^Cockroach$$' 'TESTFLAGS=-count 10 -benchmem'
name                 old time/op    new time/op    delta
Select1/Cockroach-8     126µs ± 1%     140µs ±13%  +10.74%  (p=0.000 n=9+9)

name                 old alloc/op   new alloc/op   delta
Select1/Cockroach-8    18.8kB ± 0%    19.2kB ± 0%   +2.22%  (p=0.000 n=7+9)

name                 old allocs/op  new allocs/op  delta
Select1/Cockroach-8       164 ± 0%       169 ± 0%   +3.05%  (p=0.000 n=10+10)

Pretty disappointing. Not sure this is worth it. Though the Do call is high level and always allocates something - let me see if I can improve the performance by doing things with lower level calls.

[jordanlewis]

Replacing the Do call makes this seem ok again.

name                 old time/op    new time/op    delta
Select1/Cockroach-8     126µs ± 2%     127µs ± 0%    ~     (p=0.156 n=10+9)

name                 old alloc/op   new alloc/op   delta
Select1/Cockroach-8    18.8kB ± 0%    19.2kB ± 0%  +2.26%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Select1/Cockroach-8       164 ± 0%       169 ± 0%  +3.05%  (p=0.000 n=10+10)

[jordanlewis]

Replacing the Do call makes this seem ok again.

name                 old time/op    new time/op    delta
Select1/Cockroach-8     126µs ± 2%     127µs ± 0%    ~     (p=0.156 n=10+9)

name                 old alloc/op   new alloc/op   delta
Select1/Cockroach-8    18.8kB ± 0%    19.2kB ± 0%  +2.26%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Select1/Cockroach-8       164 ± 0%       169 ± 0%  +3.05%  (p=0.000 n=10+10)

[nvanbenschoten]

Can we statically allocate the labels for all statement types to avoid the increase in memory usage?

[nvanbenschoten]

Can we statically allocate the labels for all statement types to avoid the increase in memory usage?

[jordanlewis]

We'll still have to allocate a map in WithLabels, since the labelContextKey isn't exported for us to muck with, but I think you're right that we can avoid making LabelSets over and over. Good suggestion.

[jordanlewis]

We'll still have to allocate a map in WithLabels, since the labelContextKey isn't exported for us to muck with, but I think you're right that we can avoid making LabelSets over and over. Good suggestion.

[jordanlewis]

Nevermind, there's no way we can reduce allocations here. However, I think the increase of allocations is negligible. The query benchmarked here is a worst-case - it's SELECT 1 which does almost no work at all. Normal queries do a lot more. Thoughts?

[jordanlewis]

Nevermind, there's no way we can reduce allocations here. However, I think the increase of allocations is negligible. The query benchmarked here is a worst-case - it's SELECT 1 which does almost no work at all. Normal queries do a lot more. Thoughts?

[nvanbenschoten]

Nevermind, there's no way we can reduce allocations here.

Can't we at least avoid the pprof.Labels allocation?

[nvanbenschoten]

Nevermind, there's no way we can reduce allocations here.

Can't we at least avoid the pprof.Labels allocation?

[jordanlewis]

That doesn't get allocated.

[jordanlewis]

That doesn't get allocated.

[nvanbenschoten]

Really? Aren't there varargs and a slice in pprof.Labels?

[nvanbenschoten]

Really? Aren't there varargs and a slice in pprof.Labels?

[jordanlewis]

I ran this change through pprof and didn't see any allocations on those lines. Maybe I did something wrong?

[jordanlewis]

I ran this change through pprof and didn't see any allocations on those lines. Maybe I did something wrong?

[nvanbenschoten]

Code LGTM, so up to you whether you think the added debuggability from this change is worth the minor increase in allocations.

[jordanlewis]

I'm not really sure. I'm going to let this sit a while longer and revisit when I have more headspace for it. Thanks for the reviews so far.

[jordanlewis]

I'm not really sure. I'm going to let this sit a while longer and revisit when I have more headspace for it. Thanks for the reviews so far.