Scale www database #689
Scale www database #689
Conversation
| Its possible to use one or another with politeiawww by specifying the `database` configuration. | ||
|
|
||
| #### Using LevelDB | ||
| LevelDB requires a minimal configuration. The only required config options is `datadir` which specifies where the directory |
There was a problem hiding this comment.
datadir is not a required config setting since it has a default.
|
|
||
| #### Using CockroachDB | ||
| CockroachDB requires the config options `dbhost`, `dbcertdir`, `dbrootcert` to connect to the running db instance. In order to | ||
| setup the Database you can run the same script as specified in the cache instructions: |
There was a problem hiding this comment.
Is the capitalization of Database here intended?
| @@ -241,6 +241,45 @@ to setup Politeia and access it through the UI. | |||
| * Use the [politeia](https://github.com/decred/politeia/tree/master/politeiad/cmd/politeia) tool to interact directly with politeiad. | |||
| * Use any other tools or clients that are listed above. | |||
|
|
|||
| ### Users database setup | |||
There was a problem hiding this comment.
The rest of the markdown file is formatted to adhere to the 80 column limit (unless its a link of code snippet). Can you do the same here?
| `$ politeiawww_dbutil <command> -h` | ||
|
|
||
|
|
||
| ## Setup Configuration File |
There was a problem hiding this comment.
There is an inconsistent naming convention between OS's here. The macOS is putting the conf file in politeiawww/dbutil where as windows and ubuntu put it in politeiawww/cli.
More generally though, politeiawww_dbutil conf and data should probably not be part of the politeiawww directory.
osDataDir/politeiawww_dataload instead of osDataDir/politeiawww/dbutil
|
|
||
| ### Migrating the database | ||
|
|
||
| If you are running an older version of the database you need to migrate |
There was a problem hiding this comment.
I understand the need for a migration tool if you're migrating from leveldb to cockroachdb, but shouldn't version migrations be handled automatically inside of politeiawww?
| ) | ||
|
|
||
| const ( | ||
| ErrorNoUserIdentity = "No user identity found. You must login with a user." |
| DBCertDir string `long:"dbcertdir" description:"Directory containing SSL client certificates"` | ||
| DBRootCert string `long:"dbrootcert" description:"File containing SSL root certificate"` | ||
| EncryptDB bool `long:"encryptdb" description:"If true the database will encrypt/decrypt for saving and retrieving records"` | ||
| RawJSON bool `short:"j" long:"json" description:"Print raw JSON output"` |
There was a problem hiding this comment.
Why do you have RawJSON and Verbose as config params when they're not actually used anywhere in the tool?
|
|
||
| // cleanAndExpandPath expands environment variables and leading ~ in the passed | ||
| // path, cleans the result, and returns it. | ||
| func cleanAndExpandPath(path string) string { |
There was a problem hiding this comment.
This needs to be updated with the changes in #694.
|
|
||
| // NewCDB returns a new cockroachdb context which contains a connection to the | ||
| // specified database that was made using the passed in user and certificates. | ||
| func NewCDB(user, host, net, rootCert, certDir string, dbKey *database.EncryptionKey, cfg *Config) (*cockroachdb, error) { |
There was a problem hiding this comment.
Style nit. This function gets called as cockroachdb.NewCDB(). The CDB is redundant since the package name is already cockroachdb.
| @@ -86,6 +87,8 @@ type backend struct { | |||
|
|
|||
| // Following entries require locks | |||
| userPubkeys map[string]string // [pubkey][userid] | |||
| userEmail map[string]string // [email][userid] | |||
| userUsername map[string]string // [username][userid] | |||
There was a problem hiding this comment.
I don't think we should be adding in-memory caches to www if we're just going to have to remove them in a few weeks. The way I suggest doing this would be to store the user record as:
type User struct {
UserID string // primary key
Username string
Payload []byte // encrypted blob with the rest of the user details
}
That way we can do user lookups using either the userID or username and eliminate the need for a userUsername in-memory cache. The userEmail in-memory cache would still be required temporarily, but since we want to make email optional anyway, we'll need to switch all email lookups in www to be either username or userID lookups. When we do this we can remove the userEmail in-memory cache completely.
|
Implemented by #856. |
This PR rewrites the database interface and adds another database option, CokroachDB, making it more generic and robust.
The full list of changes implemented are:
politeiawww_dbutilwhich now has an implementation similar topoliteiawwwcliand has additional commands to handle the database migration.The setup of cockroachdb can be done by using the same script introduced in the cache PR.
Instructions of how to switch from the old implementation and from leveldb to cockroachdb can be found in the
politeiawww_dbutilpackage README.Other changes implemented in this PR are:
politeiawww.closes #545