Cluster Setup doesn't set consistent admin user

[wohali]

Similar to #1752, if you use the cluster setup wizard to create admin users on new nodes in the cluster, the created admin user will not use an identical salt on every node, meaning Fauxton fails to obtain a valid _session unless you manually distribute the ini file [admins] user yourself using a config management tool.

Expected Behavior

Setup wizard creates a real server admin user that is usable from Fauxton immediately.

Current Behavior

Setup wizard creates an admin user that cannot be load balanced because the salt is different.

Possible Solution

Pass the salt along as part of the cluster setup API endpoint?

/cc @janl

[jaydoane]

Not sure if this is useful, but I wrote a function to set admin passwords that could maybe be used and/or repurposed? The idea is that it takes a while for the first, locally set password to get hashed, so it waits until the hashing occurs, and only then sets the same Username, and HashedPassword on the other nodes in the cluster:

set_cluster_admin_password(Username, Password) ->
    Section = "admins",
    ok = config:set(Section, Username, Password),
    Condition = fun() ->
        case config:get(Section, Username, Password) of
            Password ->
                wait;
            Hashed ->
                {ok, Hashed}
        end
    end,
    {ok, HashedPassword} = test_util:wait(Condition, 100, 5),
    {Results, []} = rpc:multicall(config, set,
        [Section, Username, HashedPassword]),
    true = lists:all(fun(ok) -> true end, Results),
    {Passwords, []} = rpc:multicall(config, get, [Section, Username]),
    true = lists:all(fun(P) -> P =:= HashedPassword end, Passwords),
    HashedPassword.

This would probably need some better error handling to be viable, but it seems to work under normal conditions:

(node1@127.0.0.1)42> couch_util:set_cluster_admin_password("foo", "pass").
"-pbkdf2-588c7472a6289cef771e31da99e02ca5c48829e8,122b938d618a8062b6c1aebcb74d6059,10"

[jaydoane]

I'm not familiar with fauxton: does it use the _config endpoint to configure admin users? If so, it would seem we'd need to expose a new endpoint for set_cluster_admin_password functionality. Is that correct, or is there an existing endpoint that might be adaptable?

[wohali]

It uses the cluster setup endpoints, which do correctly set admins remotely, not via the config endpoint. See https://github.com/apache/couchdb/tree/master/src/setup

They just don't sync salts.

[jaydoane]

To sync salts, we could modify this code something like this:

diff --git a/src/couch/src/couch_passwords.erl b/src/couch/src/couch_passwords.erl
index baf78f5d5..403b057ba 100644
--- a/src/couch/src/couch_passwords.erl
+++ b/src/couch/src/couch_passwords.erl
@@ -41,18 +41,24 @@ hash_admin_password(ClearPassword) when is_binary(ClearPassword) ->
     hash_admin_password(Scheme, ClearPassword).

 hash_admin_password("simple", ClearPassword) -> % deprecated
-    Salt = couch_uuids:random(),
+    Salt = salt(),
     Hash = crypto:hash(sha, <<ClearPassword/binary, Salt/binary>>),
     ?l2b("-hashed-" ++ couch_util:to_hex(Hash) ++ "," ++ ?b2l(Salt));
 hash_admin_password("pbkdf2", ClearPassword) ->
     Iterations = config:get("couch_httpd_auth", "iterations", "10000"),
-    Salt = couch_uuids:random(),
+    Salt = salt(),
     DerivedKey = couch_passwords:pbkdf2(couch_util:to_binary(ClearPassword),
                                         Salt ,list_to_integer(Iterations)),
     ?l2b("-pbkdf2-" ++ ?b2l(DerivedKey) ++ ","
         ++ ?b2l(Salt) ++ ","
         ++ Iterations).

+salt() ->
+    case config:get("couch_httpd_auth", "salt") of
+        undefined -> couch_uuids:random();
+        Salt -> list_to_binary(Salt)
+    end.
+
 -spec get_unhashed_admins() -> list().
 get_unhashed_admins() ->
     lists:filter(

and then fauton could be modified to obtain a uuid from a node

$ curl -u adm:pass http://127.0.0.1:15984/_uuids
{"uuids":["32528dddde8aac395b7dc6848b000a78"]}

and then set the config option on each node separately via

$ curl -u adm:pass http://127.0.0.1:15984/_node/node1@127.0.0.1/_config/couch_httpd_auth/salt -X PUT -d '"32528dddde8aac395b7dc6848b000a78"'

after which the way it's currently setting the admin passwords should start yielding the same hashes. I might be worried that having the salt in the config file is a security concern, but then I notice that we already store the salt with each admin user already

(node1@127.0.0.1)17> config:get("admins", "admin", "pass").
"-pbkdf2-54d94fc58cabe34b3cf6391c71290dd0198e2996,32528dddde8aac395b7dc6848b000a78,10"

[wohali]

@jaydoane But that breaks the entire cluster setup wizard approach, doesn't it?

[jaydoane]

@wohali Hmm, I thought it kept the current approach mostly the same, and just added a couple additional requests beforehand to sync the salt on all nodes. In what way do you think it breaks the setup wizard approach? I suppose salt could be added to the setup request, but then it would mean we'd need to be careful to apply that before setting any admin users get set. And fauton would still need to either request or generate its own salt so it's the same on all nodes.

[wohali]

@jaydoane setup wizard can be fully run from the CLI. does your workflow work WITHOUT Fauxton and WITHOUT additional API calls? that's the point of the setup wizard.

See http://docs.couchdb.org/en/latest/setup/cluster.html#the-cluster-setup-api

I know for a fact people use this in Chef recipes et. al. and expect it to do all the setup for them.

This ticket was specifically to fix that API endpoint so it did all the things...

[jaydoane]

@wohali ok, got it -- thanks for patiently explaining the requirements. It seems like your initial suggestion of passing the salt in with the enable_cluster action, in combination with making the salt configurable in couch_passwords could work. Would it be an error if a user passes in both password_hash (a current option) and password_salt, or do we just give one precedence over the other? Would we require a minimum length or randomness for the salt or just use whatever gets passed in? Can you think of other subtleties?

[wohali]

Ideally I'd like it to set the admin password on the "coordinator" node, then re-use that salt on subsequent nodes added to the cluster. This way you'd never have to pass in the salt.

[jaydoane]

It seems that we can sync the salt from the coordinator node at the end of setup, because it's only then, after the other nodes have been added, that all nodes will become connected. Similarly, we can sync the admin users from the coordinating node at the same time. It turns out these operations are pretty much independent, and syncing the salt is only really necessary for subsequently added admin users to have the same salt and hashed values (which still seems valuable).

In any case, I have a working proof of concept which still needs some error handling before it's worthy of review.

[janl]

I had this fixed a while ago in this commit: apache/couchdb-setup@54623ce

The scenario is first-time setup: the admin us created on the coordinating note, after that, only the password hash which includes the salt, is sent to the other nodes.

I’m not sure how this plays with adding new nodes to existing clusters, but that’s so far been out of scope for the setup wizard.

[wohali]

@janl I'm afraid this is still a bug.

In this example, couchdb1/ couchdb2/ and couchdb3/ are all full installs of CouchDB. Each has been modified to run at ports {1|2|3}5984/{1|2|3}5986. Each is running and unconfigured at the start of the test:

$ curl localhost:15984/_node/_local/_config/admins
{}
$ curl localhost:25984/_node/_local/_config/admins
{}
$ curl localhost:35984/_node/_local/_config/admins
{}

Our instructions then tell users to set an initial admin username and password on each node independently. In this example, I don't set a password on node 2, but I do on nodes 1 and 3:

$ curl -X POST -H "Content-Type: application/json" http://localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}'
{"ok":true}
$ grep "admin =" couchdb1/etc/local.ini
admin = -pbkdf2-818168f1162fb7312621227520bae09a41ebe8d6,42c6f5d002106cbd8768419a34f6c1ef,10
$ curl -X POST -H "Content-Type: application/json" localhost:35984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "port":"35984", "username": "admin", "password":"password", "node_count":"3"}'
{"ok":true}
$ grep "^admin =" couchdb3/etc/local.ini
admin = -pbkdf2-0fe0bc68ab3fdb12e790203fa21e668e31035d26,481f201c37df287ea3221bc6ccd44583,10

Note the different salts - at this point, this is expected.

Now I proceed to join the nodes into the cluster. Node 2 goes as expected:

$ curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 25984, "node_count": "3", "remote_node": "127.0.0.1"}'
{"ok":true}
$ curl localhost:25984/_node/_local/_config/admins
{"error":"unauthorized","reason":"You are not a server admin."}
$ grep "^admin" couchdb2/etc/local.ini
admin = -pbkdf2-818168f1162fb7312621227520bae09a41ebe8d6,42c6f5d002106cbd8768419a34f6c1ef,10

But node 3 doesn't work:

$ curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 35984, "node_count": "3", "remote_node": "127.0.0.1"}'
{"ok":true}
$ curl localhost:35984/_node/_local/_config/admins
{"error":"unauthorized","reason":"You are not a server admin."}
$ grep "^admin" couchdb3/etc/local.ini
admin = -pbkdf2-0fe0bc68ab3fdb12e790203fa21e668e31035d26,481f201c37df287ea3221bc6ccd44583,10

Oops, I thought, I forgot to include the remote_current_user/password and it's incorrectly telling me that everything was fine. I mean, how could it possibly change anything without a valid admin/password for node 3?

Even specifying the value doesn't change the result:

$ curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 35984, "node_count": "3", "remote_node": "127.0.0.1", "remote_current_user": "admin", "remote_current_password": "password"}'
{"ok":true}
$ curl localhost:35984/_node/_local/_config/admins
{"error":"unauthorized","reason":"You are not a server admin."}
$ grep "^admin" couchdb3/etc/local.ini
admin = -pbkdf2-0fe0bc68ab3fdb12e790203fa21e668e31035d26,481f201c37df287ea3221bc6ccd44583,10

---

OK, I thought, maybe I should start with random logins per node and then create a new, different admin user using the setup wizard.

Same initial config setup: 3 nodes, unconfigured.

$ curl localhost:15984/_node/_local/_config/admins
{}
$ curl localhost:25984/_node/_local/_config/admins
{}
$ curl localhost:35984/_node/_local/_config/admins
{}

Set up a unique admin user per node with the cluster setup API:

$ curl -X POST -H "Content-Type: application/json" http://localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin1", "password":"password1", "node_count":"3"}'
{"ok":true}
$ curl -X POST -H "Content-Type: application/json" http://localhost:25984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin2", "password":"password2", "node_count":"3"}'
{"ok":true}
$ curl -X POST -H "Content-Type: application/json" http://localhost:35984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin3", "password":"password3", "node_count":"3"}'
{"ok":true}
$ grep ^admin couchdb1/etc/local.ini
admin1 = -pbkdf2-68f22a1cdaee4128066489da65ddf0bd5d3b7cf6,f398ebdfb8b31c3765dbe6e3e7b1c088,10
$ grep ^admin couchdb2/etc/local.ini
admin2 = -pbkdf2-fe9036152614bf8b68fc87ccf87742b073924bdf,048ef75cf6cf1c69c8eb3ed6f6a43d47,10
$ grep ^admin couchdb3/etc/local.ini
admin3 = -pbkdf2-a558c85a5a1b4597eb2017bfc3ef9aa848e706dc,5f9a033e7d85e8447eb5e37db8e87098,10
$

Now let's try and create a unified admin1 user across all nodes:

$ curl -X POST -H "Content-Type: application/json" http://admin1:password1@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin1", "password":"password1", "port": 25984, "node_count": "3", "remote_node": "127.0.0.1", "remote_current_user": "admin2", "remote_current_password": "password2" }'
{"ok":true}
$ curl -X POST -H "Content-Type: application/json" http://admin1:password1@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin1", "password":"password1", "port": 35984, "node_count": "3", "remote_node": "127.0.0.1", "remote_current_user": "admin3", "remote_current_password": "password3" }'
{"ok":true}
$
$ grep ^admin couchdb1/etc/local.ini
admin1 = -pbkdf2-68f22a1cdaee4128066489da65ddf0bd5d3b7cf6,f398ebdfb8b31c3765dbe6e3e7b1c088,10
$ grep ^admin couchdb2/etc/local.ini
admin2 = -pbkdf2-fe9036152614bf8b68fc87ccf87742b073924bdf,048ef75cf6cf1c69c8eb3ed6f6a43d47,10
$ grep ^admin couchdb3/etc/local.ini
admin3 = -pbkdf2-a558c85a5a1b4597eb2017bfc3ef9aa848e706dc,5f9a033e7d85e8447eb5e37db8e87098,10

Something's broken - either our documentation is wrong, the code is wrong, or both.

[janl]

Docs then, the wizard expects to init the admin user on all nodes.

[wohali]

@janl I want to make sure I understand this correctly. The correct workflow is:

1. User pre-seeds the exact same admin user on all nodes by hand first with the same salt
2. ???

My example shows that even setting the same admin user on all nodes first with different salts doesn't allow the cluster setup wizard to forcibly fix the salt on all nodes.

[jaydoane]

It sounds like there's still some uncertainty about how admin passwords should be set, but I can confirm what @wohali observes: the way setup is currently being used (in e.g. dev/run) does not result in admin passwords with the same hash, nor is it possible to have the same salt on all nodes since it is a new random number each time a password gets hashed. I opened #1803 in an attempt to solve both of these issues without changing the setup API.

[jaydoane]

Commenting on @wohali's examples:

Our instructions then tell users to set an initial admin username and password on each node independently. In this example, I don't set a password on node 2, but I do on nodes 1 and 3:

The docs specify that the enable_cluster action, using the same admin username and password, must happen on every node in the cluster, so I don't understand why you skipped node 2 here. I definitely would not expect that to work in general.

Now I proceed to join the nodes into the cluster. Node 2 goes as expected:

But the command you show is

curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 25984, "node_count": "3", "remote_node": "127.0.0.1"}'

which is the same "enable_cluster" action as used previously on nodes couchdb1 and couchdb3, and is effectively setting the password on that node. When you write "join the nodes", I was expecting the "add_node" action, which is responsible for making sure the cookies are the same and starting Erlang distribution, etc.

For the second example using different admin users, the docs seem fairly clear to me that it should actually be the same admin user for each node, so I wouldn't expect that to work.

---

So, here's my simple minded summary of how we expect users to set up a cluster:

1. Each node in turn is sent a request with the enable_cluster action using identical admin usernames & passwords:

curl -X POST -H "Content-Type: application/json" http://localhost:15984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}'
curl -X POST -H "Content-Type: application/json" http://localhost:25984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}'
curl -X POST -H "Content-Type: application/json" http://localhost:35984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}'

At this point, we expect the salts to differer on all nodes. One thing that's confusing about the docs is that they specify using a username:password in those commands, but that would not work if no admin passwords have been configured in e.g. local.ini. Maybe we should remove those creds from the docs if we're assuming no previously configured admins?

2. The "coordinator" node is sent requests to join with the other nodes in the cluster. In this example, the node listening on 15984 is coordinating:

curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "add_node", "host":"127.0.0.1", "port": 25984, "username": "admin", "password":"password"}'
curl -X POST -H "Content-Type: application/json" http://admin:password@localhost:15984/_cluster_setup -d '{"action": "add_node", "host":"127.0.0.1", "port": 35984, "username": "admin", "password":"password"}'

Note that we're using the admin creds set in previous "enable_cluster" actions

3. The coordinator node is sent a final request/action to "finish_cluster", which basically ensures the system dbs are created.

curl -X POST -H "Content-Type: application/json" http://admin:password@lcoalhost:15984/_cluster_setup -d '{"action": "finish_cluster"}'

By extending the functionality of this request we could sync the admin passwords configured in the "enable_cluster" requests. However, if we don't also configure a common salt among nodes, subsequent admin users created with e.g. rpc:multicall(config, set, ["admins", "my-admin", "pass"]). will not have the same salt/hashes on all nodes.

[wohali]

So there's 2 things you've left unaddressed here that I think need to be explained before I can discuss your response.

1. Why are we bothering to pass in both a username/password and remote_current_user/password pair if we require everything to be identical on all nodes? I would have thought that the username/password pair is the admin user we are trying to create everywhere, and the remote_current_user/password pair is the pre-existing admin user on that remote node we need to use to affect change on that node.

   1. In an ideal world, assuming these two pairs of data are allowed different, why couldn't the remote version be some unique auto-generated set of creds on every node? I'm looking towards CouchDB 3.0 where we intend to completely remove admin party. In this situation, nodes will stand up with an arbitrarily generated admin/password pair, and there is no way (short of pre-seeding your own local.ini file) that Couch could possibly know a priori what the eventual cluster's admin/password pair would be. But, if it printed out its randomly generated u/p on startup (similar to how dev/run does it), that info could be gathered by a script or a human and plugged into the cluster wizard on setup. These would be the remote credentials, and the primary username/password pair passed in via JSON would be the newly created admin user, with the unified salt. (The wizard could then also remove the autogenerated u/p credentials, as a bonus feature to be added later.)

   2. Assuming for whatever reason we don't want to do 1, why are we even bothering to pass in 2 sets of credentials, if the only way this works is if we forcibly manually set the creds on every node to be identical at the start? I notice you've removed the second set of credentials in your example, but that still has confusion with the difference between the two sets of creds - if we're forcing it via the first step, why bother even supporting passing them in?

2. The situation where a different admin password is set on a node and I forgot to pass in the correct remote_current_user/password should have been an error, no? It shouldn't have been able to make any change to that remote node. And yet we got back an {"ok":true}. That's got to be wrong.

[jaydoane]

Taking a step back for a sec, the docs definitely seem confusing on a couple points:

First, they say "On each node we need to run the following command to set up the node:"

curl -X POST -H "Content-Type: application/json" http://admin:password@127.0.0.1:5984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}'

but then a few paragraphs later it says: "To join a node to the cluster, run these commands for each node you want to add:"

curl -X POST -H "Content-Type: application/json" http://admin:password@<setup-coordination-node>:5984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 5984, "node_count": "3", "remote_node": "<remote-node-ip>", "remote_current_user": "<remote-node-username>", "remote_current_password": "<remote-node-password>" }'
curl -X POST -H "Content-Type: application/json" http://admin:password@<setup-coordination-node>:5984/_cluster_setup -d '{"action": "add_node", "host":"<remote-node-ip>", "port": <remote-node-port>, "username": "admin", "password":"password"}'

So, the first paragraph implies each node separately needs "enable_cluster" run on it (and also that is how dev/run currently does it), while the second implies you can actually make all setup requests via the coordinator node. However, perhaps the most significant difference between "direct" and "remote" cluster enabling, is that the latter uses the password hash, which I finally understand is how @janl claims that synced password hashes are already fixed. But since it only works for "remote" cluster enabling, is not well documented, and doesn't sync hashes for admin users added subsequent to cluster setup, it could perhaps still use some improvement.

Anyway, to try to address @wohali's questions:

Why are we bothering to pass in both a username/password and remote_current_user/password pair if we require everything to be identical on all nodes?

It appears that we're trying to handle a situation where known admin creds may already exist on the nodes. In that case, we'd need to use those creds either in a basic auth header if we're making direct requests, or setting them as remote_current_user/password in the case when the coordinator is making the requests.

I would have thought that the username/password pair is the admin user we are trying to create everywhere, and the remote_current_user/password pair is the pre-existing admin user on that remote node we need to use to affect change on that node.

That is my understanding as well.

In an ideal world, assuming these two pairs of data are allowed different, why couldn't the remote version be some unique auto-generated set of creds on every node?

I think they could.

Assuming for whatever reason we don't want to do 1, why are we even bothering to pass in 2 sets of credentials, if the only way this works is if we forcibly manually set the creds on every node to be identical at the start?

I assume the idea is that the remote creds (and basic auth header for coordinator) are to be used for existing admin user(s), if any, but the "username" and "password" fields are for a single, new admin user common to all nodes. I think this interpretation is supported by how the "remote" enable_cluster uses the existing admin user's hash.

I notice you've removed the second set of credentials in your example, but that still has confusion with the difference between the two sets of creds - if we're forcing it via the first step, why bother even supporting passing them in?

I think I answered that already. Please LMK though if it's not clear.

The situation where a different admin password is set on a node and I forgot to pass in the correct remote_current_user/password should have been an error, no? It shouldn't have been able to make any change to that remote node. And yet we got back an {"ok":true}. That's got to be wrong.

According to this we should treat anything other than a 200 as an error. However, you will note that we only handle one kind of error here cluster_enabled, so I think you're right that this needs better error handling for the situation you describe.

[wohali]

So we have a few bugs here:

1. Docs are unclear on what actually works now (and, unfortunately, so am I still)
2. Code isn't handling errors correctly when bad data is passed in
3. There's still at least one situation described above where we're not doing the right thing

[janl]

the _cluster_setup endpoint expects that nodes have no admin users set at all.

[jaydoane]

Hmm, but the docs say "The settings we recommend you set now, before joining the nodes into a cluster, are: ... At least one server administrator user (and password)"

Also, these docs show admin creds being used e.g. http://admin:password@127.0.0.1:5984/_cluster_setup

[chladic]

Im very confused with docs, tried to set up cluster few times and it does not work as its described. When I dont set admin user in local.ini, my couchDB server does not even start, so I cant start configuring it via API.

I followed @jaydoane manual for setting 3 node couchdb1/2/3 and when I do at the end:

curl -X POST -H "Content-Type: application/json" http://couchdb1:5984/_cluster_setup -d '{"action": "finish_cluster" }'
{"ok":true}
curl http://couchdb1:5984/_cluster_setup
{"state":"single_node_enabled"}

Cluster looks like its working, but _cluster_setup endpoint is telling me its single_node, not cluster. Is there some magic trick for getting result: "cluster_finished"

[chladic]

I finally found the reason.
In Debian default installation there is a file:

cat etc/default.d/5-single-node.ini

# Package-introduced single node file
[cluster]
n = 1

After removing this file, I get _cluster_setup correctly to cluster_finished