Use per-node tokens for agent-server websocket authentication

[SpComb]

The current agent-server websocket RPC mechanism uses a grid-level shared secret token for authentication, together with the agent-provided Docker ID for identifying nodes. This is useful for simplifying autoscaling deployments, allowing any number of grid nodes to be provisioned using a common cloud-config. However, this is bad from a security standpoint, because the compromise of any node will leak the shared grid token, which is also very difficult to replace (#1973), requiring all nodes to be re-provisioned.

The agent provisioning and agent-server websocket connections should be changed to use predetermined node IDs and per-node tokens. Instead of the server dynamically creating new host nodes as agents connect, the host nodes should be created beforehand, using a user-supplied name and a server-generated token.

The agents would then be provisioned with the following configuration:

• KONTENA_URI=wss://...
• KONTENA_NODE_ID=grid/node-name
• KONTENA_NODE_TOKEN=<generated>

Removing a node would then invalidate the node token, preventing the agent from reconnecting. Assuming further support for weave secret rotation, the grid token could then be replaced without affecting the remaining nodes.

With per-node authentication tokens, the agent-server RPC protocol should also be improved to remove the current use of explicit node_id RPC parameters, strictly limiting each agent to only have access to the GridServiceInstance and Container objects associated with the authenticated HostNode. This would mean that grid secrets would only be accessible by the specific nodes that the service is deployed to.

The server must still retain support for the legacy grid token based RPC websocket authentication, both for existing host nodes, as well as simplified autoscaling deployments using a fixed cloud-config. However, if all host nodes are deployed using node tokens, and given further weave secret related improvements, this would serve to avoid the potential #1973 security headaches.

[SpComb]

#2504 implements kontena node create, HostNode.token websocket RPC Kontena-Node-Token authentication.

In addition, this also needs plugin changes for provisioning support.

[SpComb]

#2504 is now merged, and provides agent/server support for per-node tokens, including CLI commands to manage them.

I'm keeping this issue open to track the Kontena plugin changes to use node tokens for provisioning, instead of the grid tokens.

[SpComb]

Switching over to node tokens really also needs a fix for #2502 (separate weave shared secret) first.. currently each node still requires the KONTENA_TOKEN regardless, because it's also used as the weave secret. The agent won't work without each node having the same /etc/kontena-agent.env KONTENA_TOKEN=, and the KONTENA_NODE_TOKEN security benefits require the nodes to NOT have the grid token.

As a (hacky) workaround, you could now provision the nodes with valid KONTENA_NODE_TOKENs + a KONTENA_TOKEN that isn't actually the grid token, but that's not something we want to do in the plugins.

[SpComb]

This isn't happening for 1.4 yet, because it also needs the changes for the separate weave secret. There's a WiP implementation of those in #2583, but it's tied up with the agent refactoring in #2582 which is very much not ready.