Start shards with --keyFile

[ajdavis]

MongoDB 4.1.1 requires shards to be replica sets. Orchestration appears to know that; it starts shard servers as replicas even if there is only one server per shard.

However, with auth enabled and multiple servers per shard, initialization fails. Shard servers log:

2018-08-05T10:42:47.937+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:59888 #62 (3 connections now open)
2018-08-05T10:42:47.941+0000 I NETWORK  [conn62] received client metadata from 127.0.0.1:59888 conn62: { driver: { name: "MongoDB Internal Client", version: "4.1.1-224-gecb0b6c" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "14.04" } }
2018-08-05T10:42:47.942+0000 I ACCESS   [conn62] SASL SCRAM-SHA-1 authentication failed for __system on local from client 127.0.0.1:59888 ; AuthenticationFailed: It is not possible to authenticate as the __system user on servers started without a --keyFile parameter
2018-08-05T10:42:47.942+0000 I NETWORK  [conn62] end connection 127.0.0.1:59888 (2 connections now open)

An example config file that triggers this failure is the C Driver's sharded_clusters/auth-ssl.json at version 2f387895. This config file starts two servers per shard; the standard drivers-evergreen-tools config file uses one server per shard and that still works with the latest MongoDB.

I think this is due to a recent server change, since Orchestration can still use this config file, with two members per shard, with MongoDB 4 and earlier, and the failure began with the latest MongoDB about a week ago. Furthermore the failure might be intermittent for some reason.

First tracked in https://jira.mongodb.org/browse/CDRIVER-2783

[ShaneHarvey]

I think the issue in CDRIVER-2783 is unrelated to the auth failures and is probably caused by the "unsupported certificate purpose" issue.

The "authentication failed" log lines are normal because of the way MO starts clusters with auth. It first starts the cluster without auth, then adds the initial admin user, then (sequentially) restarts each server with auth. "authentication failed" is logged when a server is restarted with auth while other servers have auth disabled.

This is probably not the most efficient process to start modern MongoDB clusters but it is the one that works with 2.4+.

[ajdavis]

Huh. I tried updating only the C Driver’s server.pem, which got rid of the “purpose unsupported “ error, while keeping the C Driver’s auth-ssl.json, which has 2 replicas per shard. That didn’t work, still got the keyFile error. I needed to both update server.pem and replace the 2-replica JSON file with a 1-replica file in order to start a shared cluster with TLS, auth, and the latest MongoDB.

[xdg]

FWIW: In my Perl orchestration tools, I start one server without auth or keyfile, configure the root user, then restart with auth and keyfile. Other servers start with auth and keyfile. Only then do I initiate the replica set off the server with a root user configured. I think that avoids "authentication failed" messages in the log from the system user.

[ShaneHarvey]

Thanks David! I'll take a crack at that process. It might work around this issue.

[ajdavis]

That would definitely be an improvement.

I realized (in SERVER-36460) that the C Driver has always had a bad cert. It has not been testing sharded clusters with auth and TLS in a very long time (ever? for any MongoDB version?) because we'd messed up our tag selectors in our Evergreen config. So it's not the fault of a recent server change that our tests didn't work: they'd never worked and we only found out recently when we enabled them.

That said, I do believe that if there are multiple replicas per shard, with auth, then mongo orchestration must pass --keyFile.

[ShaneHarvey]

That said, I do believe that if there are multiple replicas per shard, with auth, then mongo orchestration must pass --keyFile.

As far as I can tell, MO does set --keyFile when auth is enabled: https://github.com/10gen/mongo-orchestration/blob/v0.6.11/mongo_orchestration/sharded_clusters.py#L177

I confirmed this by launching a cluster using the config you provided. Do you have an example of MO not adding --keyFile? Otherwise I think #253 might resolve some of the failures in CDRIVER-2783.

[ajdavis]

Thanks Shane, well done. I misdiagnosed this. There were a bunch of things going wrong: the C Driver's bad cert, the Mongo Orchestration shutdown bug. I was looking for the cause and it seemed bad that mongod was logging "AuthenticationFailed: It is not possible to authenticate as the __system user on servers started without a --keyFile parameter", so I figured that was related. Turns out that's "normal". I can confirm that the C Driver's old auth-ssl.json does indeed work again.