pkg/debugutil: add 'mutex' profiler (Go 1.8+)

[gyuho]

Sample output

go test -v -run=xxx -benchtime 10s -mutexprofile=mutex.out -bench=BenchmarkWatchableStoreWatchSyncPut
go tool pprof mvcc.test mutex.out

go tool pprof mvcc.test mutex.out
Entering interactive mode (type "help" for commands)
(pprof) top
9.85s of 9.85s total (  100%)
Dropped 4 nodes (cum <= 0.05s)
Showing top 10 nodes out of 14 (cum >= 5.57s)
      flat  flat%   sum%        cum   cum%
     9.85s   100%   100%      9.85s   100%  sync.(*Mutex).Unlock
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc.(*metricsTxnWrite).End
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc.(*storeTxnWrite).End
         0     0%   100%      4.27s 43.36%  github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchers
         0     0%   100%      4.27s 43.36%  github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc.(*watchableStoreTxnWrite).End
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc.(*writeView).Put
         0     0%   100%      5.58s 56.64%  github.com/coreos/etcd/mvcc.BenchmarkWatchableStoreWatchSyncPut
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc/backend.(*batchTx).Unlock
         0     0%   100%      5.57s 56.62%  github.com/coreos/etcd/mvcc/backend.(*batchTxBuffered).Unlock

[heyitsanthony]

This will enable mutex profiling even if it's not exposed via http.

Calling it in PProfHandlers would avoid the unconditional enable. Also, maybe also check if it's already been set so it won't clobber any existing settings:

if runtime.SetMutexProfileFraction(-1) != 0 {
    // 1 out of 5 mutex events are reported, on average
    runtime.SetMutexProfileFraction(5)
}

[gyuho]

@heyitsanthony PTAL.

And did you mean?

if runtime.SetMutexProfileFraction(-1) == 0 {
	// 1 out of 5 mutex events are reported, on average
	runtime.SetMutexProfileFraction(5)
}

if runtime.SetMutexProfileFraction(-1) != 0 means we had previous mutex profile fraction setting and we are overwriting (ref. https://github.com/golang/go/blob/master/src/runtime/mprof.go#L430-L442). If we want to prevent overwriting it, it should be runtime.SetMutexProfileFraction(-1) == 0?

[heyitsanthony]

lgtm. Thanks!

[codecov-io]

Codecov Report

Merging #7765 into master will decrease coverage by 0.17%.
The diff coverage is 40%.

@@            Coverage Diff             @@
##           master    #7765      +/-   ##
==========================================
- Coverage   75.79%   75.61%   -0.18%     
==========================================
  Files         331      331              
  Lines       26050    26053       +3     
==========================================
- Hits        19744    19700      -44     
- Misses       4878     4917      +39     
- Partials     1428     1436       +8

Impacted Files	Coverage Δ
pkg/debugutil/pprof.go	0% <0%> (ø)	⬆️
embed/config.go	70.48% <100%> (ø)	⬆️
pkg/adt/interval_tree.go	83.23% <0%> (-8.54%)	⬇️
etcdserver/api/v3rpc/util.go	52.77% <0%> (-5.56%)	⬇️
mvcc/watcher.go	95.65% <0%> (-4.35%)	⬇️
etcdserver/api/v3rpc/watch.go	88.6% <0%> (-3.63%)	⬇️
etcdserver/api/v3rpc/lease.go	89.28% <0%> (-3.58%)	⬇️
proxy/grpcproxy/lease.go	87.61% <0%> (-1.91%)	⬇️
lease/leasehttp/http.go	61.06% <0%> (-1.53%)	⬇️
rafthttp/peer.go	90.07% <0%> (-1.53%)	⬇️
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dd0d590...5161b74. Read the comment docs.