apiserver incorrectly returns 405 when 404 expected during a HEAD request

[satnam6502]

I think this is the root of the reason why the new Kibana to Elasticsearch setup failed to work (even with basic auth). I think a 404 response is incorrectly being turned into a 405 response.

In these example I am running a Kibana service which maps to a pod running a Kibana server.

POD                    IP           CONTAINER(S)     IMAGE(S)                              HOST                                    LABELS                                                   STATUS    CREATED
kibana-logging-88fi0   10.244.0.6   kibana-logging   gcr.io/google_containers/kibana:1.3   kubernetes-minion-02dr/104.197.15.218   kubernetes.io/cluster-service=true,name=kibana-logging   Running   44 minutes
CONTROLLER       CONTAINER(S)     IMAGE(S)                              SELECTOR              REPLICAS
kibana-logging   kibana-logging   gcr.io/google_containers/kibana:1.3   name=kibana-logging   1
NAME             LABELS                                                   SELECTOR              IP            PORT(S)
kibana-logging   kubernetes.io/cluster-service=true,name=kibana-logging   name=kibana-logging   10.0.32.105   5601/TCP

Note that the Kibana service has the IP 10.0.32.105. If we ssh into a minion and curl against this we get the expected response i.e. "a .kibana index does not exist". This is the correct expected behavior.

$ curl -X HEAD -i  10.0.32.105:5601/elasticsearch/.kibana
HTTP/1.1 404 Not Found
X-App-Name: kibana
content-type: text/plain; charset=UTF-8
content-length: 0
Date: Wed, 29 Apr 2015 18:57:21 GMT
Connection: keep-alive

However, the Kibana dashboard reports an error when it tries to connect to Elasticsearch (which it does by going through its own URL with the subpath elasticsearch) saying the method is not allowed:

https://104.197.26.147/api/v1beta3/proxy/namespaces/default/services/kibana-logging/elasticsearch/.kibana Failed to load resource: the server responded with a status of 405 (Method Not Allowed)

If we ssh to the master and perform the same HEAD request against the apiserver we get a 404 Forbidden:

$ curl -X HEAD -i  http://localhost:7080/api/v1beta3/proxy/namespaces/default/services/kibana-logging/.kibana
HTTP/1.1 403 Forbidden
Date: Wed, 29 Apr 2015 19:07:37 GMT
Content-Length: 29
Content-Type: text/plain; charset=utf-8

Adding the basic auth we get a 405 Method Not Allowed:

$ curl -X HEAD -i -k -u admin:REDACTED  https://localhost/api/v1beta3/proxy/namespaces/default/services/kibana-logging/.kibana
HTTP/1.1 405 Method Not Allowed
Content-Type: application/json
Date: Wed, 29 Apr 2015 19:08:36 GMT
Content-Length: 234

However, we should have got the same answer as doing a HEAD request on 10.0.32.105:5601/elasticsearch/.kibana i.e. a 404 Not Found.

I think this is a bug and this is what is preventing Kibana from speaking to Elasticsearch.

@thockin @roberthbailey @lavalamp
Thanks to who @ArtfulCoder who helped diagnose what was going on.

[satnam6502]

If I create an ELB for the Kibana service then everything works perfectly.

[nikhiljindal]

@satnam6502 : In addition to the headers, can you please post the body of response as well. The status object in response body might have more details on why the server thinks that the method is not allowed.

[davidopp]

@satnam6502 Did you mark this v1.0-candidate because Kibana/Elasticsearch aren't working due to this bug, and you think it's important to have them working for 1.0?

[satnam6502]

I marked it as a v1.0-candidate because if this really is a bug then a whole class of applications are likely to fall foul of this bug -- which is a shame -- because it then discourages people to deploy software in a way that suits our model (i.e. running a backend server inside the cluster that can make use of DNS etc. as is the case with the new Kibana 4) vs. running a system entirely as a pile of Javascipt in your browser which has lots of issues for us (and that's what we are forced to do now by using an older Kibana that has to speak to the service proxy at the master to access Elasticsearch).

[satnam6502]

@nikhiljindal : I shall try to do a repo when I am done with some other work -- perhaps later this week.

[lavalamp]

Do we allow HEAD requests at all? I suspect you're getting some of these error codes because of the HEAD verb. Does the client actually attempt HEAD requests? Can you repeat this sequence of actions with GET requests?

[satnam6502]

Kibana makes a HEAD request to Elasticsearch to determine if an index exists and if it does not it creates it. We should adjust our proxy to route HEAD requests.

[lavalamp]

SGTM

[lavalamp]

I think the action item here is to change the spot in api_installer.go where it sets up the go restful stuff to pass all verbs on to the proxy handler.

[satnam6502]

Closed via #7950