[JUJU-210] DQLITE scaffolding work

[achilleasa]

Caveats:

• jujud (for controllers) must be dynamically linked
• to use the new binary we must link it on the same series as the controller node (glibc dependency).

QA steps

Prepare juju 2.9 binaries for initial bootstrap

# We will be using a non-patched 2.9 agent that we can eventually replace with our
# dqlite-enabled binary

git checkout 3dd693fd2c # 2.9/HEAD (at time of testing)
make install

Spin up cross-compilation environment

cd scripts/dqlite

# Setup an environment that mirrors the controller environments so we can
# produce a right jujud binary that can run as-is on the controller boxes. As
# the resulting jujud binaries are dynamically linked, they depend on a
# particular glibc version so we need the mirrored environment to get everything
# working.
#
# The created lxd instance is called 'build-server'.
#
# You only need to do this once.
make bootstrap

Singe node test

# NOTE: make sure to use the same series as the cross-compile build-server instance
juju bootstrap lxd test-dqlite --bootstrap-series bionic --no-gui
juju switch controller

make install-deps-on-controller
make build-and-install-jujud

# Connect to the singular controller via the REPL. This uses lxc exec, rlwrap
# and socat to establish a connection to the REPL exposed via a unix socket in 
# /var/lib/juju/dqlite/juju.sock
#
# And yes, you also get history support! Press CTRL+D to disconnect.
make dqlite-repl

	[+] Connecting to REPL interface on controller: juju-bfc144-0

	Welcome to the REPL for accessing dqlite databases for Juju models.

	Before running any commands you must first connect to a database. To connect
	to a database, type '.open' followed by the model UUID to connect to.

	For a list of supported commands type '.help'

	repl> .open logs
	You are now connected to DB "logs"

	repl@logs> select level, count(*) from logs group by level;
	level   count(*)
	2       12
	3       167
	4       14
	5       39

	Total rows: 4

	repl@logs> {CTRL+D}

HA mode

First, get the singular dqlite-enabled node up and running.

juju enable-ha

# Wait until the two new machines are ha-ready

# The following for loops skip over controller-0
for node in `lxc list | grep juju | awk 'NR>=2 {print $2}' | paste -sd' '`; do
	JUJU_CONTROLLER_BOX=${node} make install-deps-on-controller
done
for node in `lxc list | grep juju | awk 'NR>=2 {print $2}' | paste -sd' '`; do
	JUJU_CONTROLLER_BOX=${node} make build-and-install-jujud
done

# Connect to any of the controllers
# 'make dqlite-repl' always connects to the first one but you can specify any controller instance as follows:
#  'JUJU_CONTROLLER_BOX=foo make dqlite-repl' 
make dqlite-repl

	[+] Connecting to REPL interface on controller: juju-bfc144-0

	Welcome to the REPL for accessing dqlite databases for Juju models.

	Before running any commands you must first connect to a database. To connect
	to a database, type '.open' followed by the model UUID to connect to.

	For a list of supported commands type '.help'

	repl> .open logs
	You are now connected to DB "logs"

	repl@logs> select entity, count(*) FROM logs group by entity;
	entity  count(*)
	machine-0       1821
	machine-1       1167
	machine-2       733

	Total rows: 3

	repl@logs> {CTRL+D}

[achilleasa]

When enabling HA mode, you will get errors such as this in the machine-0 logs:

adding node controller-1 to dqlite cluster: server ID already in use (1)

I have tracked this down to $GOPATH/deps/libraft/src/configuration.c:133 (RAFT_DUPLICATEID) but I have no idea how that value is selected. The dbaccessor worker ensures that each node has a unique ID by converting each controller's API address into a uint64 value.

Here are the contents of /var/lib/juju/dqlite/info.yaml:

$ cat info.yaml
Address: 10.67.242.248:17069
ID: 3297041220608546238
Role: 0

Interestingly, I can also see the controller-1 node (but not controller-2) in the cluster.yaml file on controller-0:

$ cat cluster.yaml
- Address: 10.67.242.248:17069             <--- this is controller-0
  ID: 3297041220608546238
  Role: 0
- Address: 10.67.242.193:17069             <--- this is controller-1
  ID: 825241142
  Role: 2

However, the dbaccessor worker keeps restarting due to the error reported by libraft.

[achilleasa]

When enabling HA mode, you will get errors such as this in the machine-0 logs:

adding node controller-1 to dqlite cluster: server ID already in use (1)

I have tracked this down to $GOPATH/deps/libraft/src/configuration.c:133 (RAFT_DUPLICATEID) but I have no idea how that value is selected. The dbaccessor worker ensures that each node has a unique ID by converting each controller's API address into a uint64 value.

Here are the contents of /var/lib/juju/dqlite/info.yaml:

$ cat info.yaml
Address: 10.67.242.248:17069
ID: 3297041220608546238
Role: 0

Interestingly, I can also see the controller-1 node (but not controller-2) in the cluster.yaml file on controller-0:

$ cat cluster.yaml
- Address: 10.67.242.248:17069             <--- this is controller-0
  ID: 3297041220608546238
  Role: 0
- Address: 10.67.242.193:17069             <--- this is controller-1
  ID: 825241142
  Role: 2

However, the dbaccessor worker keeps restarting due to the error reported by libraft.

I worked with @SimonRichardson and we identified a better approach for assembling the dqlite cluster which got the PR changes working in HA mode (caveat: we must manually remove dead nodes from the cluster).

We also tried dialing up the level to TRACE and deploying charmed-kubernetes to the controller model (on lxd). It seems to be working as expected:

repl@logs> select entity, count(*) FROM logs group by entity;
entity  count(*)
machine-0       55911
machine-1       39694
machine-11      109
machine-12      103
machine-2       6455
machine-3       84
machine-4       116
machine-6       76
machine-7       79
machine-8       85
machine-9       76
unit-etcd-0     23
unit-etcd-2     14
unit-kubeapi-load-balancer-0    14
unit-kubernetes-master-0        14
unit-kubernetes-master-1        14
unit-kubernetes-worker-0        14
unit-kubernetes-worker-2        14

Total rows: 18

Under load(?) we managed to get one of the controllers into a weird state where it suddenly started appending its logs to its own database (verified by connecting the repl to the node and running queries) whereas the other two nodes would happily keep streaming their logs to the original database. After bouncing jujud on the machine with the issue, it reconnected to the original cluster and started streaming logs again. We didn't see anything suspicious in the logs and are currently trying to reproduce and debug.

[manadart]

I prefer nouns over {verb}-er. DBFactory would by my choice.

[achilleasa]

No particular preference here; just wanted to use the idiomatic naming conventions for interfaces (io.Reader/Writer etc.).

In this case in particular, perhaps a DBResolver would be a better name compared to the term "factory"

[SimonRichardson]

Closing this because of no activity, please reopen if you want to move this forward.