Fix create db options on secondary shard creation

[chewbranca]

Overview

This is a continuation of the work in #2672 to fix issues around secondary shard creation when nodes re-enter a cluster after a database was created while the node was down (or potentially in maintenance mode).

This PR introduces logic to handle the following four scenarios while opening a db in context of fabric_rpc and mem3_rpc:

1. the db already exists
   - just return the db as normal
2. the db does not exist locally, but the dbs db doc with db create options is present
   - load the db create options and create the local shard
3. the db does not exist locally but it's not a sharded database
   - just create the local db as expected
4. the db does not exist locally, it's a sharded database, and the dbs db doc is not present
   - throw an error and prevent creating the db with incomplete settings

This PR changes the behavior of scenario 2) by loading the appropriate config and creating the db with the desired parameters, and of scenario 4) by no longer allowing the db to be created inappropriately.

For scenario three, I've taken the approach of not loading config as a non sharded db won't have a dbs db doc. In particular here, we want to properly handle updates to internal dbs like _dbs and also sys_dbs like _users.

Another open item here is the merging of the db create options and the provided options. Right now, I'm just doing DbOpts ++ Opts basically, but I wrote a functions opts_merge/{2,3} that will merge two proplists, prioritizing the first list of items, and supports atoms. It's... a bit gnarly, and I'm not convinced we should use it. So we need to figure out what to do with the options lists here, but I think we might be ok just combining the two.

While in the general neighborhood, I took the opportunity to de-duplicate some of the shards_db config lookups as well. I've isolated those changes to a dedicated commit.

Testing recommendations

This needs more testing. I've got a WIP commit for adding a test suite here that is not yet complete. I figured it was worthwhile to get this PR out for feedback while continuing to work on the tests.

Related Issues or Pull Requests

#2672

Checklist

• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation

[chewbranca]

BTW... I believe this issue also extends to PSE databases created while a node was out of rotation. Basically anything that relies on database creation config options, which is PSE and PQ (anything else?).

[davisp]

We need to add tests for options actually being applied as well.

[davisp]

Super important change here.

[davisp]

+1

[wohali]

Shouldn't this and #2672 target the 3.x branch, from which our next release will be cut?

[rnewson]

yes. :)

[davisp]

+1