mlab-ns queries prometheus twice: once without credentials then with them

[stephen-soltesz]

In the NGINX logs for prometheus, we see evidence of mlab-ns running two queries. The first receives a HTTP Status 401 (presumably without credentials). The second includes a user "mlab" and succeeds.

First fails.

107.178.195.145 - [107.178.195.145] - - [13/Jun/2018:19:31:04 +0000] "GET /api/v1/query?query=min+by+%28experiment%2C+machine%29+%28%0A++%28%0A++++%28%0A++++++%28probe_success%7Bservice%3D%22ndt_raw%22%7D%29+%2B%0A++++++ON+%28experiment%2C+machine%29+%28script_suc
cess%7Bservice%3D%22ndt_e2e%22%7D%29+%2B%0A++++++ON+%28experiment%2C+machine%29+%28%28vdlimit_used%7Bexperiment%3D%22ndt.iupui%22%7D+%2F%0A++++++++vdlimit_total%7Bexperiment%3D%22ndt.iupui%22%7D%29+%3C+bool+0.95%29+%2B%0A++++++ON+%28experiment%2C+machine%29%0A++++
++++%28lame_duck_experiment%7Bexperiment%3D%22ndt.iupui%22%7D+%21%3D+bool+1%29%0A++++%29+%3D%3D+bool+4%0A++%29+OR%0A++ON%28experiment%2C+machine%29+probe_success%7Bservice%3D%22ndt_raw%22%7D%0A++++UNLESS+ON%28machine%29+up%7Bservice%3D%22nodeexporter%22%7D+%3D%3D+
1%0A%29%0A HTTP/1.1" 401 195 "-" "Python-urllib/2.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~mlab-ns)" 1032 0.000 [default-prometheus-tls-service-9090] - - - -

Second succeeds:

107.178.195.145 - [107.178.195.145] - mlab [13/Jun/2018:19:31:04 +0000] "GET /api/v1/query?query=min+by+%28experiment%2C+machine%29+%28%0A++%28%0A++++%28%0A++++++%28probe_success%7Bservice%3D%22ndt_raw%22%7D%29+%2B%0A++++++ON+%28experiment%2C+machine%29+%28script_
success%7Bservice%3D%22ndt_e2e%22%7D%29+%2B%0A++++++ON+%28experiment%2C+machine%29+%28%28vdlimit_used%7Bexperiment%3D%22ndt.iupui%22%7D+%2F%0A++++++++vdlimit_total%7Bexperiment%3D%22ndt.iupui%22%7D%29+%3C+bool+0.95%29+%2B%0A++++++ON+%28experiment%2C+machine%29%0A+
+++++++%28lame_duck_experiment%7Bexperiment%3D%22ndt.iupui%22%7D+%21%3D+bool+1%29%0A++++%29+%3D%3D+bool+4%0A++%29+OR%0A++ON%28experiment%2C+machine%29+probe_success%7Bservice%3D%22ndt_raw%22%7D%0A++++UNLESS+ON%28machine%29+up%7Bservice%3D%22nodeexporter%22%7D+%3D%
3D+1%0A%29%0A HTTP/1.1" 200 1621 "-" "Python-urllib/2.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~mlab-ns)" 1108 0.053 [default-prometheus-tls-service-9090] 10.0.1.5:9090 1621 0.053 200

[nkinkade]

It appears that this is default/expected behavior. From the urllib2 documentation:

When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme and a ‘realm’. [...] The client should then retry the request with the appropriate name and password for the realm included as a header in the request. This is ‘basic authentication’.

It would seem that urllib2 first sends an unauthenticated request, which the server responds with a 401 error, also indicating the auth scheme, as well as the auth realm. urllib2 then looks in its password manager to see if it has a mapping for that realm/URL, then resends the appropriate authenticated request.

I suppose one way around this might be to not use any sort of urllib2 auth handler or password manager, but to simply construct a pre-authenticated URL like https://:@.

@stephen-soltesz: What are your thoughts?

[stephen-soltesz]

Well, if that's how it's supposed to work, then I guess it's okay. My opinion is that I think it's awkward. But, we don't want to do any special handling if we don't have to.

[nkinkade]

@stephen-soltesz: Is it okay to close this issue, or should we leave it open to consider alternate ways to access the data (such as constructing the URL ourselves)?

[stephen-soltesz]

I've assigned it to myself. I'll look into the code a little and suggest an approach or close the issue as WAI.