Bootstrapping of password-protected clusters

[danielmewes]

_Status quo_

• We want to keep bootstrapping a cluster for development and local deployments simple. Especially in the case where user accounts / password protection is not needed.
• Our current solution is that a new server starts out with a default admin account that doesn't have a password. If you don't need password protection, you can just start your cluster and connect. Once you set a password, it will propagate via a semilattice structure to all nodes that later join the cluster.

_The problem_
The problem arises when you add a new node to a live cluster. In that scenario there is going to be a brief moment where the node starts listening to client connections, but doesn't have the admin password yet. If an attacker gets lucky and connects to the new node at the right time, they could exploit this race condition to run arbitrary queries as admin.

We can delay accepting queries until we have synced the admin password with at least one other node. However this doesn't fully solve the issue. If you start up two new nodes at the same time, they might connect to each other and would still start serving queries without a password.

_Proposal 1_

• A command line argument: --initial-pw that takes one out of two values:
  auto generates a random initial password
  <pw> sets the specified initial password
• The initial password is printed to the console if auto is used.
• If --initial-pw is not given, the password-less admin account will be created just like now.
• If there already is a password, the command line argument is ignored.
• A server without an initial or configured admin password cannot connect to a
  server that has an admin password.
• Initial passwords internally get versions -<current time> (that is minus current timestamp). That way the oldest initial password always wins when joining other nodes, but any manually configured password wins over any initial password.

TODO: Think about also printing an auto-generated initial password to the log file. We might also want to print newer initial passwords if we receive them after connecting to a different node.

Typical scenarios with this proposal:

Scenario 1: Setting up a cluster without password protection
1. Start and join all nodes without a password (just like now)

Scenario 2: Setting up a new cluster in a trustworthy environment with password protection
1. Start and join all nodes without a password
2. Configure the admin password through the users table

Scenario 3: Setting up a new cluster in an untrustworthy environment
1. Reconsider if you actually want to do this. You should probably use TLS instead.
2. Configure servers with an explicit --initial-pw
3. Start and join all nodes
4. Change the admin password through the users table

Scenario 4: Joining a new server into an existing password-protected cluster
1. Start and join the new server with the --initial-pw auto option
(if you forget to specify that option, we'll print a helpful error message both
in the log of the new server, and in the log(s) of the existing server(s))

_Proposal 2 including cluster IDs_

This proposal is the result of a discussion with @larkost and @VeXocide. It also solves #1905 by introducing a cluster ID.

• We provide a script to generate a file containing a "cluster ID" and a salted + hashed admin password (rethinkdb createpw or something like that, though it should also work as a standalone script).
• The administrator distributes the generated cluster ID + password file to all nodes of the cluster. Alternatively, they could pass the contents of the files in as a command-line argument.
• On connection, nodes compare their cluster ID. If it doesn't match, they refuse the connection.
• Passwordless clusters can be set up by not passing in any ID+password file when creating the cluster. In that case RethinkDB assumes a default cluster ID with an empty admin password.
• In the simplest version, the admin password cannot be changed through ReQL (up for debate, there are different possible variations). To change the password, the admin will need to generate a new id+pw file using the existing cluster ID, and copy it onto all servers.

Typical scenarios with this proposal:

Scenario 1: Setting up a cluster without password protection
1. Start and join all nodes without an ID+password file (just like now)

Scenario 2: Setting up a new cluster in a trustworthy environment with password protection
1. Create the ID+password file
2. Distribute the file to all nodes
3. Start and join all nodes

Scenario 3: Setting up a new cluster in an untrustworthy environment
1. Reconsider if you actually want to do this. You should probably use TLS instead.
2. Create the ID+password file
2. Distribute the file to all nodes
3. Start and join all nodes

Scenario 4: Joining a new server into an existing password-protected cluster
1. Copy the ID+password file onto the new node
2. Start and join the new node

[danielmewes]

Pinging @coffeemug @VeXocide @larkost @mlucy @encryptio @Tryneus .

[danielmewes]

This is already an issue with our current auth key by the way.

[larkost]

I am in favor of Option 2 for a few reasons:

1. In addition to synchronizing/setting the admin password it also forms a solid basis for cluster-id
2. That cluster-id pre-share would keep clusters seperated, so (at least for password-set clusters) we would not have the current worries of accidentally merging clusters.
3. Changing the password becomes a much more deliberate process.

There are also other possibilities that seem a more natural extension of it (and less like a reversal to me):

• intercluster TLS could be based on the password/salt making administration of that much easier
• flipping that last one: you could use the a CA to generate certs with the passwords embedded and use those for shared secret, making the protected part much easier to manage

[danielmewes]

The problem I have with the second option is that it

a) Is more work to design and implement, so it would probably delay the 2.3 release.

b) That it comes with the annoyance of adding extra steps to bootstrapping and extending a cluster.
In option 1, all you need to do is add the --initial-pw auto argument to any servers that you add to your cluster after the initial ones. In option 2, you need to first download a script to generate the ID+password file, then start up the cluster servers and copy the generated file onto them. While the second step isn't fundamentally different from having to provide the correct --join parameters which is necessary in either case, I think it does make a difference in people's perception. Copying a file is an additional step, and one that involves the use of an additional tool (like scp or similar). It also slightly complicates automating cluster setup and scaling. It doesn't make it substantially more difficult, but it does add an extra thing that needs to be taken care of.

I further think that the first approach does not interfere with cluster IDs if we want to add them later. I said in the writeup above that the admin password couldn't be changed in proposal 2 except by swapping out the file, but that is not an integral part of a cluster-ID-based approach. Instead we can base a cluster-ID based design on proposal 1, and simply have the ID file replace the initial password if one is provided.

[danielmewes]

I talked to a few people about this offline.

We're going with option 1 and will look into cluster IDs later.

[danielmewes]

Variant 1 is implemented and in code review 3562

[danielmewes]

In next ebbce92