Spread recalculate caches to all nodes. Fixes #1069

[yuce]

Overview

Adds a RecalculateCaches gossip message type. The node which receives /recalculate-caches HTTP request sends RecalculateCaches message to other nodes.

In order to test, deactivate sending RecalculateCaches message by comment out line 2154-2159 of handler.go so it looks like this:

func (h *Handler) handleRecalculateCaches(w http.ResponseWriter, r *http.Request) {
	// err := h.Broadcaster.SendSync(&internal.RecalculateCaches{})
	// if err != nil {
	//	w.WriteHeader(http.StatusInternalServerError)
	//	h.writeQueryResponse(w, r, &QueryResponse{Err: err})
	//	return
	//}
	h.Holder.RecalculateCaches()
	w.WriteHeader(http.StatusNoContent)
}

Run go test -run TestMain_RecalculateHashes github.com/pilosa/pilosa/server. This should fail.

Then, remove those comments and run the test again. This time it should pass.

Fixes #1069

Pull request checklist

• I have read the contributing guide.
• I have agreed to the Contributor License Agreement.
• I have updated the documentation.
• I have resolved any merge conflicts.
• I have included tests that cover my changes.
• All new and existing tests pass.

Code review checklist

This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.

• Ensure that any changes to external docs have been included in this pull request.
• If the changes require that minor/major versions need to be updated, tag the PR appropriately.
• Ensure the new code is properly commented and follows Idiomatic Go.
• Check that tests have been written and that they cover the new functionality.
• Run tests and ensure they pass.
• Build and run the code, performing any applicable integration testing.

[jaffee]

This looks right to me

[yuce]

Great!