Context queue

[SimonRichardson]

Requires #13453 to land first.

QA steps

See the discourse post for more history: https://discourse.charmhub.io/t/raft-api-leases-part-ii/5267/4

Apply the following patch

--- a/apiserver/facades/agent/leadership/leadership.go
+++ b/apiserver/facades/agent/leadership/leadership.go
@@ -92,6 +92,7 @@ func (m *leadershipService) ClaimLeadership(args params.ClaimLeadershipBulkParam
                case names.ApplicationTag:
                        canClaim = m.authorizer.AuthOwner(applicationTag)
                }
+               canClaim = true
                if !canClaim {
                        result.Error = common.ServerError(common.ErrPerm)
                        continue
@@ -114,6 +115,7 @@ func (m *leadershipService) BlockUntilLeadershipReleased(ctx context.Context, ap
        case names.ApplicationTag:
                hasPerm = m.authorizer.AuthOwner(applicationTag)
        }
+       hasPerm = true

        if !hasPerm {
                return params.ErrorResult{Error: common.ServerError(common.ErrPerm)}, nil

Setup controller

$ juju bootstrap lxd test1 --build-agent --config features="[raft-api-leases,raft-batch-fsm]" --config agent-ratelimit-max=0
$ juju switch controller
$ juju enable-ha
$ juju model-config -m controller logging-config="<root>=INFO;juju.worker.lease.raft=TRACE;juju.core.raftlease=TRACE;juju.worker.globalclockupdater.raft=TRACE;juju.worker.raft=TRACE"
$ juju add-user prometheus
$ juju grant prometheus read controller
$ juju change-user-password prometheus 

Create the bundle and overlay:

series: focal
applications:
  controller:
    charm: cs:~jameinel/ubuntu-lite
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
  grafana:
    charm: cs:grafana-49
    num_units: 1
    to:
    - "1"
    expose: true
  prometheus:
    charm: cs:prometheus2-24
    num_units: 1
    to:
    - "0"
  telegraf:
    charm: cs:telegraf-43
    options:
      hostname: '{unit}'
      tags: juju_model=controller
machines:
  "0":
  "1":
  "2":
relations:
- - controller:juju-info
  - telegraf:juju-info
- - prometheus:grafana-source
  - grafana:grafana-source
- - telegraf:prometheus-client
  - prometheus:target

Change the controller targets and password.

applications:
  prometheus:
    options:
      scrape-jobs: |
        - job_name: juju
          metrics_path: /introspection/metrics
          scheme: https
          static_configs:
              - targets:
                - '$CONTROLLER_IP_0:17070'
                labels:
                    juju_model: controller
                    host: controller-0
              - targets:
                - '$CONTROLLER_IP_1:17070'
                labels:
                    juju_model: controller
                    host: controller-1
              - targets:
                - '$CONTROLLER_IP_2:17070'
                labels:
                    juju_model: controller
                    host: controller-2
          basic_auth:
            username: user-prometheus
            password: $PASSWORD
          tls_config:
            insecure_skip_verify: true

Deploy the bundle

$ juju deploy ./bundle.yaml --overlay=overlay.yaml --map-machines=existing
$ juju config prometheus scrape-interval=1s
$ juju config prometheus scrape-timeout=1s

Get the grafana login details

$ juju run-action grafana/0 --wait get-login-info -m controller

Update the grafana dashboard

Import the following grafana dashboard https://paste.ubuntu.com/p/jtTt5PyrPH/

Deploy ubuntu

$ juju switch default
$ juju deploy ubuntu -n 3

Setup testing

$ cd scripts/leadershipclaimer
$ go build leadershipclaimer.go

Edit the run.sh to point to the correct controllers

Ensure you get the agent password from ubuntu machine unit.

$ ./run.sh

View the results in grafana.

[achilleasa]

LGTM

[SimonRichardson]

$$merge$$