meta.(*Store).Authenticate uses a lot of CPU

[pilt]

See #3102 for background. It seems like the culprit might be in github.com/influxdb/influxdb/meta.(*Store).Authenticate if I'm not missing something.

I built influxd from source and had it save pprof data (pprof.StartCPUProfile(f)). I let the server run for some time, looked at the log to see that it was processing requests, and saw high CPU usage.

top30:

(pprof) top30
206.07s of 207.25s total (99.43%)
Dropped 48 nodes (cum <= 1.04s)
      flat  flat%   sum%        cum   cum%
   194.29s 93.75% 93.75%    194.29s 93.75%  golang.org/x/crypto/blowfish.encryptBlock
     9.71s  4.69% 98.43%    203.72s 98.30%  golang.org/x/crypto/blowfish.ExpandKey
     2.04s  0.98% 99.42%      2.04s  0.98%  runtime.lostProfileData
     0.03s 0.014% 99.43%    204.12s 98.49%  golang.org/x/crypto/bcrypt.expensiveBlowfishSetup
         0     0% 99.43%    204.16s 98.51%  github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP
         0     0% 99.43%    204.13s 98.49%  github.com/influxdb/influxdb/meta.(*Store).Authenticate
         0     0% 99.43%    204.13s 98.49%  github.com/influxdb/influxdb/meta.(*Store).read
         0     0% 99.43%    204.13s 98.49%  github.com/influxdb/influxdb/meta.func·021
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·001
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·002
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·003
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·004
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·005
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·006
         0     0% 99.43%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·007
         0     0% 99.43%    204.13s 98.49%  golang.org/x/crypto/bcrypt.CompareHashAndPassword
         0     0% 99.43%    204.13s 98.49%  golang.org/x/crypto/bcrypt.bcrypt
         0     0% 99.43%    204.16s 98.51%  net/http.(*conn).serve
         0     0% 99.43%    204.16s 98.51%  net/http.HandlerFunc.ServeHTTP
         0     0% 99.43%    204.16s 98.51%  net/http.serverHandler.ServeHTTP
         0     0% 99.43%    204.16s 98.51%  runtime.goexit

top30 -cum:

(pprof) top30 -cum
206.07s of 207.25s total (99.43%)
Dropped 48 nodes (cum <= 1.04s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%    204.16s 98.51%  github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·001
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·002
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·003
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·004
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·005
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·006
         0     0%     0%    204.16s 98.51%  github.com/influxdb/influxdb/services/httpd.func·007
         0     0%     0%    204.16s 98.51%  net/http.(*conn).serve
         0     0%     0%    204.16s 98.51%  net/http.HandlerFunc.ServeHTTP
         0     0%     0%    204.16s 98.51%  net/http.serverHandler.ServeHTTP
         0     0%     0%    204.16s 98.51%  runtime.goexit
         0     0%     0%    204.13s 98.49%  github.com/influxdb/influxdb/meta.(*Store).Authenticate
         0     0%     0%    204.13s 98.49%  github.com/influxdb/influxdb/meta.(*Store).read
         0     0%     0%    204.13s 98.49%  github.com/influxdb/influxdb/meta.func·021
         0     0%     0%    204.13s 98.49%  golang.org/x/crypto/bcrypt.CompareHashAndPassword
         0     0%     0%    204.13s 98.49%  golang.org/x/crypto/bcrypt.bcrypt
     0.03s 0.014% 0.014%    204.12s 98.49%  golang.org/x/crypto/bcrypt.expensiveBlowfishSetup
     9.71s  4.69%  4.70%    203.72s 98.30%  golang.org/x/crypto/blowfish.ExpandKey
   194.29s 93.75% 98.45%    194.29s 93.75%  golang.org/x/crypto/blowfish.encryptBlock
     2.04s  0.98% 99.43%      2.04s  0.98%  runtime.lostProfileData

gif:

[pauldix]

fixed by #3161 and #3201. In 0.9.1, which we're releasing as I type this.