Issues during rolling update from 1.3.4 to 1.4.0

[jimmyjones2]

Currently doing a rolling update of my 3 node cluster from 1.3.4 to 1.4.0, but after upgrading the first machine (es-node-1), I noticed Kibana stopped working, digging through the logs, I can reproduce the error:

curl es-node-1:9200/main-2014.39/_aliases

This returns:

{"error":"RemoteTransportException[[es-node-3][inet[/xxx.xxx.xxx.xxx:9300]][indices:admin/get]]; nested: ActionNotFoundTransportException[No handler for action [indices:admin/get]]; ","status":500}

and in the log of es-node-3:

[es-node-3] Message not fully read (request) for [128466] and action [indices:admin/get], resetting

However the same query on the other two nodes works as expected (tried for all of my indicies), eg:

curl es-node-2:9200/main-2014.39/_aliases

Indexing seems to be happening as normal, and the _search endpoint also seems to work as expected.

I've now shut down es-node-3 and everything seems to work as expected, continuing with update...

[jimmyjones2]

Also just noticed in the logs got some further related warnings, which I think was as node-3 came up with 1.4.0 (my rolling update was es-node-1, es-node-3, es-node-2):

[es-node-2] Message not fully read (request) for [0] and action [internal:discovery/zen/unicast_gte_1_4], resetting

[colings86]

Thanks for reporting, we've found the issue. It occurs when the master node has not yet been upgraded and receives a request for GET _aliases from an upgraded node. You can continue with your upgrade and the problem will disappear when you finish the upgrade. We'll work on a fix for it

[colings86]

Also, the messages in your second comment (regarding internal:discovery/zen/unicast_gte_1_4) are expected. The new zen discovery attempts to communicate with the node with the new protocol and then falls back to the old protocol if it is not successful. These messages should also disappear when the upgrade is complete

[jimmyjones2]

Thanks. Now everything is upgraded no problems.