Merge pull request #15177 from SimonRichardson/witness-me

#15177

The following brings the 3.0-dqlite feature branch into the develop branch.

### Changes

This brings in the dqlite database to sit along side the mongo database. Currently, only leases are implemented in Juju using dqlite, more controller base configuration and data will be subsequently moved over to dqlite once this branch has landed.

#### Leases/Raft

The whole raft implementation has been removed from Juju completely. This includes the following workers:

 - raft backstop
 - raft clusterer
 - raft log
 - raft transport
 - global clock updater

In addition, the raft API implementation has also been removed. Instead, the lease has changed to handle the store (dqlite db) directly, improving readability and complexity.

### Jujud 

The `jujud` agent is now built using musl (specifically musl-gcc). This allows `juju` to be built statically embedding `dqlite` in the same process. There are still some rough edges when building and testing and when this lands, we expect to see some churn to polish any of those issues.

Using `go test` is expected to still work as is, this is a last-minute change so that we can utilize sqlite directly for local tests. If you require to test with dqlite (linux only), then running `-tags="dqlite"` with builds/tests/installs is required. All CI jobs are required to run with the dqlite tag.

Some notes:

 1. `CGO_ENABLED=1` and `CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"` are required if you're using dqlite directly.
 2. You are expected to install musl directly on your system if you want to build, using `make musl-install`. This will require sudo.
 3. For development purposes we will download dqlite `.a` files from an s3 bucket to facilitate the setup process. The tar file is sha256 summed to ensure no MITM. You can build these locally if you want to bypass s3 using `make dqlite-build-lxd`. This will spin up an lxd container to build. **Do not attempt** to run `make dqlite-build` locally, unless you know what you're doing.
 4. To access dqlite from a controller, use `make repl`, this will open up a pseudo repl when you can then explore the database with. `.open <db name>` and then you can use SQL from there.
 5. Cross compilation to other architectures can be done using `GOARCH` and `GOOS` before `make install` or `make build`.

There are probably some things I've forgotten, expect a discourse post soon, which will highlight the development flow.

----

Two conflicts when merging. The resolution was to bring in the secret backends for the manifold tests and the controller config type changed for `DefaultMigrationMinionWaitMax`.

```
CONFLICT (content): Merge conflict in cmd/jujud/agent/machine/manifolds_test.go
CONFLICT (content): Merge conflict in controller/config.go
```

c141b2e (upstream/3.0-dqlite) Merge pull request #15159 from SimonRichardson/system-install-musl-by-default
83656e2 Merge pull request #15156 from SimonRichardson/change-log-ddl
125c19d Fix static-analysis pipeline (#15168)
5abfa24 Merge pull request #15140 from SimonRichardson/allow-testing-on-mac
1dc60f6 (3.0-dqlite) Merge pull request #15153 from SimonRichardson/content-addressable-deps
5a1cd24 Merge pull request #15150 from jack-w-shaw/JUJU-2615_symlink_sudo
4502d63 Merge pull request #15148 from SimonRichardson/better-install-method
88941dd Merge pull request #15134 from SimonRichardson/bootstrap-dqlite-agent-tests
2551ffc Merge pull request #15130 from SimonRichardson/build-jujud-snap
0180a53 (origin/3.0-dqlite, manadart/3.0-dqlite) Merge pull request #15123 from SimonRichardson/fix-manifold-lease-expiry-tests
fdf9cc7 Merge pull request #15115 from SimonRichardson/remove-jujud-main-test-file
bf58843 Merge pull request #15113 from SimonRichardson/remove-api-raftlease-api-client
f9419c0 Merge pull request #15112 from SimonRichardson/fix-initializing-state-twice
334d557 Merge pull request #15108 from SimonRichardson/github-action-go-build
2ee6e1a Merge pull request #15107 from SimonRichardson/cross-building-jujud
5a93305 Merge pull request #15087 from SimonRichardson/ensure-placement-of-file
da95dc0 Merge pull request #15086 from SimonRichardson/more-sudo-changes
7b86376 Merge pull request #15085 from SimonRichardson/sudo-apt-get
c4d4eb6 Merge pull request #15057 from SimonRichardson/dqlite-local-build
0ac79b3 Merge pull request #15061 from manadart/develop-into-3.0-dqlite
adc20f7 Merge pull request #15043 from SimonRichardson/allow-overriding-arch-machine
8c02f22 Merge pull request #15048 from SimonRichardson/static-analysis-fix
4547c06 Merge pull request #15050 from manadart/dqlite-address-option
d51b324 Merge pull request #15049 from manadart/dqlite-bootstrap-options
3801b78 Merge pull request #15047 from manadart/develop-into-3.0-dqlite
22d5247 Merge pull request #15037 from SimonRichardson/standardise-dqlite-build
25640a2 Merge pull request #15036 from SimonRichardson/remove-batch-fsm-controller-config
dfa4cb1 Merge pull request #15041 from manadart/dqlite-fix-mock
caf9481 Merge pull request #15034 from manadart/develop-into-3.0-dqlite
c91985d Merge pull request #15035 from SimonRichardson/remove-typed-lease-error
42d17be Merge pull request #15009 from SimonRichardson/allow-repl-via-juju-ssh
d798238 Merge pull request #15002 from manadart/dqlite-use-lease-store
e4f0d39 Merge pull request #14918 from manadart/3.0-dqlite-lease-store
8315fb7 Merge pull request #14986 from manadart/dqlite-build-from-tags
a73b947 Merge pull request #14927 from manadart/3.0-dqlite-lease-store-interface
1657a1d Merge pull request #14910 from manadart/3.0-dqlite-db-supply
27b23f3 Merge pull request #14909 from manadart/3.0-into-3.0-dqlite
6adff35 Merge pull request #14756 from manadart/develop-into-3.0-dqlite
37d81ff Merge pull request #14717 from manadart/develop-into-3.0-dqlite
fe2edb8 Merge pull request #14671 from manadart/3.0-simplify-dbaccessor
1a09836 Merge pull request #14604 from manadart/3.0-bootstrap-controller-db
5ad011e Merge pull request #14652 from manadart/develop-into-3.0-dqlite
1c3d250 Merge pull request #14591 from manadart/develop-into-3.0-dqlite
229cd3e Merge pull request #14578 from manadart/3.0-dqlite-simplify
9d715ba Merge pull request #14565 from manadart/develop-into-3.0-dqlite
92ffd88 Merge pull request #14466 from manadart/develop-into-3.0-dqlite
57f67ce Merge pull request #14336 from manadart/develop-into-3.0-dqlite
648d354 Merge pull request #14364 from manadart/update-musl
198621d Merge pull request #14241 from manadart/develop-into-3.0-dqlite
0360db6 Merge pull request #14153 from manadart/develop-into-3.0-dqlite
17950b2 Merge pull request #14053 from manadart/develop-into-3.0-dqlite
9452026 Merge pull request #14016 from manadart/develop-into-3.0-dqlite
741baca Merge pull request #13963 from manadart/develop-into-3.0-dqlite
5449603 Merge pull request #13969 from manadart/dqlite-manifolds
7b612a0 Merge pull request #13944 from SimonRichardson/dqlite-develop

.github/golangci-lint.config.experimental.yaml

@@ -10,3 +10,4 @@ run:
   tests: false
 skip-dirs:
   - acceptancetests
+  - _deps

.github/golangci-lint.config.yaml

@@ -32,9 +32,10 @@ linters:
     - ineffassign
     - misspell
     - unconvert
-    - deadcode
-    - varcheck
+
 run:
   timeout: 30m
+  modules-download-mode: readonly
 skip-dirs:
   - acceptancetests
+  - _deps

.github/workflows/build.yml

@@ -38,6 +38,23 @@ jobs:
         go-version-file: 'go.mod'
         cache: true
 
-    - name: Build
+    - name: "Restore musl cache"
+      id: restore-musl-cache
+      if: (matrix.os == 'linux')
+      uses: actions/cache/restore@v3
+      with:
+        key: ${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('./scripts/dqlite/Makefile') }}
+        path: |
+          ./_deps/musl-${{ runner.arch }}
+
+    - name: "Build"
       run: |
-        GOOS=${{ matrix.platform.os }} GOARCH=${{ matrix.platform.arch }} make go-install
+        GOOS=${{ matrix.platform.os }} GOARCH=${{ matrix.platform.arch }} make MUSL_PLACEMENT=local go-build
+
+    - name: "Save musl cache"
+      uses: actions/cache/save@v3
+      if: (matrix.os == 'linux')
+      with:
+        key: ${{ steps.restore-musl-cache.outputs.cache-primary-key }}
+        path: |
+          ./_deps/musl-${{ runner.arch }}

.github/workflows/cla.yml

@@ -1,5 +1,6 @@
 name: "CLA check"
 on: [pull_request, workflow_dispatch]
+
 permissions:
   contents: read
 

.github/workflows/client-tests.yml

@@ -21,42 +21,40 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        # windows tests is disabled because we require a certain version of
-        # mongo (that's actually going away now though), and choclately was
-        # failing to install it correctly.
-        # os: [ubuntu-latest, macOS-latest, windows-latest]
         os: [ubuntu-latest, macOS-latest]
 
     steps:
-    - name: Checkout
+    - name: "Checkout"
       uses: actions/checkout@v3
 
-    - name: Set up Go
+    - name: "Set up Go"
       uses: actions/setup-go@v3
       with:
         go-version-file: 'go.mod'
         cache: true
 
-    - name: "Install Mongo Dependencies: ubuntu-latest"
+    - name: "Install Mongo Dependencies (Ubuntu)"
       if: (matrix.os == 'ubuntu-latest')
       run: |
         make install-mongo-dependencies
 
-    - name: "Remove Mongo Dependencies: windows-latest"
-      if: (matrix.os == 'windows-latest')
-      uses: crazy-max/ghaction-chocolatey@90deb87d9fbf0bb2f022b91e3bf11b4441cddda5 # v1
-      with:
-        args: uninstall mongodb mongodb.install -y --all-versions
-
-    - name: "Install Mongo Dependencies: windows-latest"
-      if: (matrix.os == 'windows-latest')
-      uses: crazy-max/ghaction-chocolatey@90deb87d9fbf0bb2f022b91e3bf11b4441cddda5 # v1
-      with:
-        args: install mongodb.install --version=5.0.5 --allow-downgrade
+    - name: "Install musl & dqlite  (Ubuntu)"
+      if: (matrix.os == 'ubuntu-latest')
+      shell: bash
+      run: |
+        sudo make musl-install dqlite-install
 
-    - name: "Test client"
+    - name: "Test client (Ubuntu)"
+      if: (matrix.os == 'ubuntu-latest')
+      shell: bash
       run: |
         # Jenkins can perform the full jujud testing.
-        go test -v ./cmd/juju/... -check.v -coverprofile=coverage.txt -covermode=atomic -timeout=15m
-        go test -v ./cmd/plugins/... -check.v
+        TEST_PACKAGES=./cmd/juju/... COVERAGE_CHECK=1 TEST_TIMEOUT="15m" make run-go-tests
+        TEST_PACKAGES=./cmd/plugins/... COVERAGE_CHECK=1 TEST_TIMEOUT="15m" make run-go-tests
+
+    - name: "Test client (macOS)"
+      if: (matrix.os == 'macOS-latest')
       shell: bash
+      run: |
+        go test -v ./cmd/juju/... -check.v -coverprofile=coverage.txt -covermode=atomic -timeout=15m
+        go test -v ./cmd/plugins/... -check.v -coverprofile=coverage.txt -covermode=atomic -timeout=15m

.github/workflows/gen.yml

@@ -19,26 +19,31 @@ jobs:
     if: github.event.pull_request.draft == false
 
     steps:
-      - name: Checkout
+      - name: "Checkout"
         uses: actions/checkout@v3
 
-      - name: Set up Go
+      - name: "Set up Go"
         uses: actions/setup-go@v3
         with:
           go-version-file: 'go.mod'
           cache: true
 
-      - name: go get dependencies
+      - name: "go get dependencies"
         run: go get ./...
 
-      - name: Delete all mocks
+      - name: "Install musl & dqlite"
+        shell: bash
+        run: |
+          sudo make musl-install dqlite-install
+
+      - name: "Delete all mocks"
         shell: bash
         run: |
           for FILE in $(grep '// Code generated by MockGen. DO NOT EDIT.' -r . -l --include \*.go); do
             rm $FILE
           done
 
-      - name: Regenerate code
+      - name: "Regenerate code"
         shell: bash
         run: |
           # Running go generate by itself is slow over a large codebase, where
@@ -51,9 +56,9 @@ jobs:
           #  3. Unique every file, so we only go generate the file once.
           #  4. Using xargs perform go generate in parallel.
           #
-          grep -ir "//go:generate" . | awk -F : '{ print $1 }' | uniq | xargs -n 1 -P 2 -I% go generate -x $(realpath %)
+          make run-go-generate
 
-      - name: Check diff
+      - name: "Check diff"
         if: success() || failure()
         shell: bash
         run: |

.github/workflows/snap.yml

@@ -9,6 +9,7 @@ on:
       - 'snap/**'
       - '.github/workflows/snap.yml'
   workflow_dispatch:
+
 permissions:
   contents: read
 

.github/workflows/static-analysis.yml

@@ -6,6 +6,7 @@ on:
 #   paths:
 #     DON'T SET - these are "required" so they need to run on every PR
   workflow_dispatch:
+
 permissions:
   contents: read
 
@@ -48,10 +49,16 @@ jobs:
         echo "GOPATH=$(go env GOPATH)" >> $GITHUB_ENV
         echo "$(go env GOPATH)/bin" >> $GITHUB_PATH
 
-        curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.46.2
-        sudo snap install shfmt
-        sudo apt install expect
-    
+        sudo add-apt-repository ppa:dqlite/dev -y --no-update
+        sudo apt-get update
+        sudo apt-get install -y \
+          expect \
+          libdqlite-dev \
+          libsqlite3-dev \
+          sqlite3
+
+        curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.50.1
+
     - name: Download Dependencies
       run: go mod download
 
@@ -72,8 +79,13 @@ jobs:
       run: |
         # Explicitly set GOROOT to avoid golangci-lint/issues/3107
         export "GOROOT=$(go env GOROOT)"
+        export "CGO_LDFLAGS=-L$(pwd)/_deps/juju-dqlite-static-lib-deps -luv -lraft -ldqlite -llz4 -lsqlite3"
+
         STATIC_ANALYSIS_JOB=test_static_analysis_go make static-analysis
       shell: bash
+      env:
+        CGO_ENABLED: 1
+        CGO_LDFLAGS_ALLOW: "(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"
 
     - name: "Static Analysis: Python Check"
       if: steps.filter.outputs.static-analysis == 'true' || steps.filter.outputs.python == 'true'
@@ -115,7 +127,13 @@ jobs:
     - name: Install Dependencies
       if: steps.filter.outputs.schema == 'true'
       run: |
-        sudo apt install expect
+        sudo add-apt-repository ppa:dqlite/dev -y --no-update
+        sudo apt-get update
+        sudo apt-get install -y \
+          expect \
+          libdqlite-dev \
+          libsqlite3-dev \
+          sqlite3
 
     - name: Schema Check
       if: steps.filter.outputs.schema == 'true'

.github/workflows/update-brew-formulae.yml

@@ -5,6 +5,7 @@ on:
   schedule:
     - cron:  '0 */12 * * *'
   workflow_dispatch:
+
 permissions:
   contents: read
 

.gitignore

@@ -30,4 +30,5 @@ tests/tmp.*
 *.out
 *.charm
 *.bundle
-*.log
+*.log
+dqlite-deps.tar.bz2