Nodes joining as voters despite -raft-non-voter=true

[jtackaberry]

What version are you running?

8.14.0

Are you using Docker or Kubernetes to run your system?

K8s

Are you running a single node or a cluster?

3 voters + 1 readonly -- or at least that's what was expected, but in practice it's 4 voters.

What did you do?

Deployed 3 voter nodes with the following (where <N> is the node number 0, 1 or 2):

launch command: /bin/rqlited -node-id rqlite-<N> -http-addr 0.0.0.0:4001 -http-adv-addr rqlite-<N>.rqlite-headless.data.svc.cluster.local:4001 -raft-addr 0.0.0.0:4002 -raft-adv-addr rqlite-<N>.rqlite-headless.data.svc.cluster.local:4002 -auth=/config/sensitive/users.json -join-as=_system_rqlite -node-cert=/config/sensitive/node.crt -node-key=/config/sensitive/node.key -node-verify-server-name=rqlite.data.svc.cluster.local -node-ca-cert=/config/sensitive/node-ca.crt -node-verify-client -http-cert=/config/client-tls/tls.crt -http-key=/config/client-tls/tls.key -http-ca-cert=/config/sensitive/client-ca.crt -disco-mode=dns-srv -disco-config={"name":"rqlite-headless","service":"raft"} -bootstrap-expect=3 -join-interval=1s -join-attempts=120 -raft-shutdown-stepdown /rqlite

Then 1 read-only node with:

launch command: /bin/rqlited -node-id rqlite-readonly-0 -http-addr 0.0.0.0:4001 -http-adv-addr rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4001 -raft-addr 0.0.0.0:4002 -raft-adv-addr rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4002 -auth=/config/sensitive/users.json -join-as=_system_rqlite -node-cert=/config/sensitive/node.crt -node-key=/config/sensitive/node.key -node-verify-server-name=rqlite.data.svc.cluster.local -node-ca-cert=/config/sensitive/node-ca.crt -node-verify-client -http-cert=/config/client-tls/tls.crt -http-key=/config/client-tls/tls.key -http-ca-cert=/config/sensitive/client-ca.crt -disco-mode=dns-srv -disco-config={"name":"rqlite-headless","service":"raft"} -raft-cluster-remove-shutdown=true -raft-non-voter=true -join-interval=1s -join-attempts=120 -raft-shutdown-stepdown /rqlite

All nodes deployed and started concurrently from fresh state (empty data volume). Note that the read-only node gets -raft-non-voter=true.

Full startup logs of rqlite-0 (current leader) and rqlite-readonly-0 are attached.

rqlite-readonly-0.log
rqlite-0.log

However the nodes list shows all nodes as being voters. This was run from rqlite-0 (current leader) and rqlite-readonly-0 agrees:

rqlite-2:
  time_s: 120ns
  id: rqlite-2
  api_addr: https://rqlite-2.rqlite-headless.data.svc.cluster.local:4001
  addr: rqlite-2.rqlite-headless.data.svc.cluster.local:4002
  voter: true
  reachable: true
  leader: false
  time: 4.9e-8
rqlite-readonly-0:
  voter: true
  reachable: true
  leader: false
  time: 2.1e-7
  time_s: 491ns
  id: rqlite-readonly-0
  api_addr: https://rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4001
  addr: rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4002
rqlite-0:
  time: 3.2e-7
  time_s: 571ns
  id: rqlite-0
  api_addr: https://rqlite-0.rqlite-headless.data.svc.cluster.local:4001
  addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4002
  voter: true
  reachable: true
  leader: true
rqlite-1:
  api_addr: https://rqlite-1.rqlite-headless.data.svc.cluster.local:4001
  addr: rqlite-1.rqlite-headless.data.svc.cluster.local:4002
  voter: true
  reachable: true
  leader: false
  time: 6e-8
  time_s: 130ns
  id: rqlite-1

What did you expect to happen?

I expected rqlite-readonly-0 to show voter: false in the nodes output.

Please include the Status, Nodes, and Expvar output from each node (or at least the Leader!)

Nodes list included above. The following is from rqlite-0 (current leader):

127.0.0.1:4001> .status
store:
  fsm_index: 0
  open: true
  snapshot_threshold: 8192
  heartbeat_timeout: 1s
  raft:
    last_contact: 0
    last_snapshot_term: 0
    log_size: 32768
    snapshot_version_min: 0
    voter: true
    state: Leader
    applied_index: 3
    bolt:
      PendingPageN: 2
      FreeAlloc: 8192
      FreelistInuse: 32
      TxN: 23
      OpenTxN: 0
      TxStats:
        RebalanceTime: 0
        Spill: 11
        SpillTime: 194508
        PageCount: 22
        CursorCount: 80
        NodeDeref: 0
        Split: 0
        Write: 33
        WriteTime: 190719189
        PageAlloc: 90112
        NodeCount: 21
        Rebalance: 0
      FreePageN: 0
    commit_index: 3
    fsm_pending: 0
    last_snapshot_index: 0
    protocol_version_max: 3
    last_log_index: 3
    latest_configuration: [{Suffrage:Voter ID:rqlite-readonly-0 Address:rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4002} {Suffrage:Voter ID:rqlite-2 Address:rqlite-2.rqlite-headless.data.svc.cluster.local:4002} {Suffrage:Voter ID:rqlite-0 Address:rqlite-0.rqlite-headless.data.svc.cluster.local:4002} {Suffrage:Voter ID:rqlite-1 Address:rqlite-1.rqlite-headless.data.svc.cluster.local:4002}]
    latest_configuration_index: 0
    snapshot_version_max: 1
    term: 2
    last_log_term: 2
    num_peers: 3
    protocol_version: 3
    protocol_version_min: 0
  snapshot_interval: 10s
  request_marshaler:
    force_compression: false
    compression_batch: 50
    compression_size: 1024
  last_applied_index: 3
  node_id: rqlite-0
  reap_read_only_timeout: 0s
  trailing_logs: 10240
  dir_size_friendly: 70 kB
  leader:
    addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4002
    node_id: rqlite-0
  nodes: [map[addr:rqlite-0.rqlite-headless.data.svc.cluster.local:4002 id:rqlite-0 suffrage:Voter] map[addr:rqlite-1.rqlite-headless.data.svc.cluster.local:4002 id:rqlite-1 suffrage:Voter] map[addr:rqlite-2.rqlite-headless.data.svc.cluster.local:4002 id:rqlite-2 suffrage:Voter] map[addr:rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4002 id:rqlite-readonly-0 suffrage:Voter]]
  ready: true
  snapshot_store:
    db_path:
    dir: /rqlite/rsnapshots
    snapshots: []
  apply_timeout: 10s
  sqlite3:
    db_size_friendly: 4.1 kB
    mem_stats:
      page_size: 4096
      soft_heap_limit: 0
      cache_size: -2000
      freelist_count: 0
      hard_heap_limit: 0
      max_page_count: 1073741823
      page_count: 1
    ro_dsn: file:/rqlite/db.sqlite?mode=ro&_fk=false
    rw_dsn: file:/rqlite/db.sqlite?_fk=false
    size: 4096
    version: 3.44.0
    wal_size: 0
    compile_options: [ATOMIC_INTRINSICS=1 COMPILER=gcc-9.4.0 DEFAULT_AUTOVACUUM DEFAULT_CACHE_SIZE=-2000 DEFAULT_FILE_FORMAT=4 DEFAULT_JOURNAL_SIZE_LIMIT=-1 DEFAULT_MMAP_SIZE=0 DEFAULT_PAGE_SIZE=4096 DEFAULT_PCACHE_INITSZ=20 DEFAULT_RECURSIVE_TRIGGERS DEFAULT_SECTOR_SIZE=4096 DEFAULT_SYNCHRONOUS=2 DEFAULT_WAL_AUTOCHECKPOINT=1000 DEFAULT_WAL_SYNCHRONOUS=1 DEFAULT_WORKER_THREADS=0 ENABLE_DBSTAT_VTAB ENABLE_FTS3 ENABLE_FTS3_PARENTHESIS ENABLE_FTS5 ENABLE_RTREE ENABLE_UPDATE_DELETE_LIMIT MALLOC_SOFT_LIMIT=1024 MAX_ATTACHED=10 MAX_COLUMN=2000 MAX_COMPOUND_SELECT=500 MAX_DEFAULT_PAGE_SIZE=8192 MAX_EXPR_DEPTH=1000 MAX_FUNCTION_ARG=127 MAX_LENGTH=1000000000 MAX_LIKE_PATTERN_LENGTH=50000 MAX_MMAP_SIZE=0x7fff0000 MAX_PAGE_COUNT=1073741823 MAX_PAGE_SIZE=65536 MAX_SQL_LENGTH=1000000000 MAX_TRIGGER_DEPTH=1000 MAX_VARIABLE_NUMBER=32766 MAX_VDBE_OP=250000000 MAX_WORKER_THREADS=8 MUTEX_PTHREADS OMIT_DEPRECATED OMIT_LOAD_EXTENSION OMIT_SHARED_CACHE SYSTEM_MALLOC TEMP_STORE=1 THREADSAFE=1]
    conn_pool_stats:
      ro:
        max_idle_time_closed: 0
        max_open_connections: 0
        open_connections: 1
        in_use: 0
        max_idle_closed: 0
        idle: 1
        wait_count: 0
        wait_duration: 0
        max_lifetime_closed: 0
      rw:
        max_idle_closed: 0
        max_lifetime_closed: 0
        in_use: 0
        wait_duration: 0
        idle: 1
        wait_count: 0
        max_idle_time_closed: 0
        max_open_connections: 1
        open_connections: 1
    db_size: 4096
    path: /rqlite/db.sqlite
    pragmas:
      rw:
        synchronous: 0
        wal_autocheckpoint: 0
        foreign_keys: 0
        journal_mode: wal
      ro:
        foreign_keys: 0
        journal_mode: wal
        synchronous: 1
        wal_autocheckpoint: 1000
  addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4002
  db_applied_index: 0
  db_conf:
    fk_constraints: false
  dir: /rqlite
  dir_size: 69632
  election_timeout: 1s
  no_freelist_sync: false
  observer:
    dropped: 0
    observed: 1
  reap_timeout: 0s
build:
  build_time: 2023-12-31T11:35:47-0500
  commit: c08ff99fefc25b364b564e52222560e7b2811c8a
  compiler: gc
  version: v8.14.0
  branch: master
cluster:
  addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4002
  api_addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4001
  https: true
disco:
  dns_name:: _raft._tcp.rqlite-headless.
  last_addresses: [[2001:470:b0d6:fd:ce70:bd4e:b908:3b6d]:4002 10.3.59.109:4002]
  last_contact: 2023-12-31T18:15:39.604313871Z
  mode: dns-srv
  name: rqlite-headless
  service: raft
http:
  tls:
    next_protos: [h2 http/1.1]
    ca_file: /config/sensitive/client-ca.crt
    cert_file: /config/client-tls/tls.crt
    client_auth: NoClientCert
    enabled: true
    key_file: /config/client-tls/tls.key
  auth: enabled
  bind_addr: [::]:4001
  cluster:
    timeout: 30s
    conn_pool_stats:
      10.3.59.109:4002:
        idle: 4
        max_open_connections: 64
        open_connections: 4
      [2001:470:b0d6:fd:ce70:bd4e:b908:3b6d]:4002:
        idle: 4
        max_open_connections: 64
        open_connections: 4
      rqlite-1.rqlite-headless.data.svc.cluster.local:4002:
        idle: 4
        max_open_connections: 64
        open_connections: 4
      rqlite-2.rqlite-headless.data.svc.cluster.local:4002:
        idle: 4
        max_open_connections: 64
        open_connections: 4
      rqlite-readonly-0.rqlite-headless-readonly.data.svc.cluster.local:4002:
        idle: 4
        max_open_connections: 64
        open_connections: 4
    local_node_addr: rqlite-0.rqlite-headless.data.svc.cluster.local:4002
  queue:
    _default:
      batch_size: 128
      max_size: 1024
      sequence_number: 0
      timeout: 50ms
network:
  interfaces:
    eth0:
      flags: up|broadcast|multicast|running
      hardware_address: ca:25:33:fb:40:7b
      addresses: [map[address:10.3.59.109/32] map[address:2001:470:b0d6:fd:ce70:bd4e:b908:3b6d/128] map[address:fe80::c825:33ff:fefb:407b/64]]
    lo:
      flags: up|loopback|running
      hardware_address:
      addresses: [map[address:127.0.0.1/8] map[address:::1/128]]
node:
  current_time: 2023-12-31T18:17:22.917152297Z
  start_time: 2023-12-31T18:15:36.744980221Z
  uptime: 1m46.172172598s
os:
  executable: /bin/rqlited
  hostname: rqlite-0
  page_size: 4096
  pid: 1
  ppid: 0
runtime:
  version: go1.21.5
  GOARCH: amd64
  GOMAXPROCS: 10
  GOOS: linux
  num_cpu: 10
  num_goroutine: 71
127.0.0.1:4001> .expvar
memstats:
  GCSys: 4275904
  Sys: 25007368
  Frees: 163179
  MSpanInuse: 215208
  HeapReleased: 3899392
  MCacheSys: 15600
  HeapIdle: 6791168
  PauseTotalNs: 1130463
  DebugGC: false
  StackSys: 1146880
  OtherSys: 2224120
  NumGC: 7
  NumForcedGC: 0
  EnableGC: true
  HeapAlloc: 6654528
  NextGC: 11322944
  PauseNs: [278387 232750 94880 115760 65995 140236 202455 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
  HeapSys: 15630336
  GCCPUFraction: 0.00003990406084174916
  BySize: [map[Frees:0 Mallocs:0 Size:0] map[Frees:351 Mallocs:1549 Size:8] map[Frees:20335 Mallocs:25652 Size:16] map[Frees:13865 Mallocs:15873 Size:24] map[Frees:15690 Mallocs:17811 Size:32] map[Frees:11133 Mallocs:14269 Size:48] map[Frees:24247 Mallocs:27082 Size:64] map[Frees:22590 Mallocs:25010 Size:80] map[Frees:20479 Mallocs:22855 Size:96] map[Frees:4953 Mallocs:5294 Size:112] map[Frees:4798 Mallocs:5234 Size:128] map[Frees:3876 Mallocs:13350 Size:144] map[Frees:303 Mallocs:435 Size:160] map[Frees:618 Mallocs:954 Size:176] map[Frees:164 Mallocs:249 Size:192] map[Frees:218 Mallocs:534 Size:208] map[Frees:72 Mallocs:93 Size:224] map[Frees:26 Mallocs:41 Size:240] map[Frees:439 Mallocs:562 Size:256] map[Frees:193 Mallocs:823 Size:288] map[Frees:327 Mallocs:562 Size:320] map[Frees:83 Mallocs:254 Size:352] map[Frees:41 Mallocs:113 Size:384] map[Frees:39 Mallocs:227 Size:416] map[Frees:4 Mallocs:39 Size:448] map[Frees:4 Mallocs:4 Size:480] map[Frees:3676 Mallocs:4083 Size:512] map[Frees:137 Mallocs:254 Size:576] map[Frees:12 Mallocs:29 Size:640] map[Frees:14 Mallocs:27 Size:704] map[Frees:3 Mallocs:18 Size:768] map[Frees:43 Mallocs:149 Size:896] map[Frees:119 Mallocs:195 Size:1024] map[Frees:7 Mallocs:98 Size:1152] map[Frees:109 Mallocs:166 Size:1280] map[Frees:36 Mallocs:49 Size:1408] map[Frees:0 Mallocs:4 Size:1536] map[Frees:45 Mallocs:88 Size:1792] map[Frees:9 Mallocs:29 Size:2048] map[Frees:3 Mallocs:134 Size:2304] map[Frees:37 Mallocs:52 Size:2688] map[Frees:24 Mallocs:33 Size:3072] map[Frees:0 Mallocs:1 Size:3200] map[Frees:0 Mallocs:4 Size:3456] map[Frees:56 Mallocs:95 Size:4096] map[Frees:14 Mallocs:36 Size:4864] map[Frees:2 Mallocs:11 Size:5376] map[Frees:0 Mallocs:4 Size:6144] map[Frees:0 Mallocs:0 Size:6528] map[Frees:0 Mallocs:3 Size:6784] map[Frees:2 Mallocs:2 Size:6912] map[Frees:4 Mallocs:18 Size:8192] map[Frees:5 Mallocs:15 Size:9472] map[Frees:0 Mallocs:0 Size:9728] map[Frees:0 Mallocs:0 Size:10240] map[Frees:0 Mallocs:1 Size:10880] map[Frees:2 Mallocs:2 Size:12288] map[Frees:0 Mallocs:1 Size:13568] map[Frees:0 Mallocs:0 Size:14336] map[Frees:1 Mallocs:5 Size:16384] map[Frees:1 Mallocs:1 Size:18432]]
  StackInuse: 1146880
  MCacheInuse: 12000
  LastGC: 1704046635559173448
  Alloc: 6654528
  Mallocs: 198459
  HeapObjects: 35280
  MSpanSys: 260736
  BuckHashSys: 1453792
  PauseEnd: [1704046536741593023 1704046536905444399 1704046543225890590 1704046543506812820 1704046565898747428 1704046596145149054 1704046635559173448 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
  TotalAlloc: 19602432
  Lookups: 0
  HeapInuse: 8839168
uploader:
  last_upload_bytes: 0
  num_uploads_fail: 0
  num_uploads_ok: 0
  num_uploads_skipped: 0
  total_upload_bytes: 0
cluster:
  num_execute_req: 0
  num_get_node_api_req: 61
  num_get_node_api_req_local: 21
  num_get_node_api_resp: 61
  num_join_req: 13
  num_load_req: 0
  num_backup_req: 0
  num_notify_req: 10
  num_query_req: 0
  num_remove_node_req: 0
  num_request_req: 0
  num_client_retries: 0
db:
  query_errors: 0
  request_transactions: 0
  checkpointed_moves: 0
  checkpoints: 0
  queries: 9
  checkpoint_duration_ns: 0
  open_duration_ms: 18
  execution_errors: 0
  executions: 0
  execute_transactions: 0
  query_transactions: 0
  requests: 0
  checkpoint_errors: 0
  checkpointed_pages: 0
downloader:
  download_bytes: 0
  num_downloads_fail: 0
  num_downloads_ok: 0
http:
  authFail: 0
  request_stmts_rx: 0
  queued_executions_leadership_lost: 0
  remote_executions_failed: 0
  queued_executions_not_leader: 0
  authOK: 24
  executions: 0
  loads_aborted: 0
  queries: 0
  queued_executions: 0
  remote_remove_node: 0
  requests: 0
  boot: 0
  loads: 0
  num_status: 1
  queued_executions_failed: 0
  remote_executions: 0
  queued_executions_no_leader: 0
  queued_executions_num_stmts_rx: 0
  queued_executions_unknown_error: 0
  remote_queries: 0
  queued_executions_num_stmts_tx: 0
  queued_executions_wait: 0
  remote_loads: 0
  backups: 0
  execute_stmts_rx: 0
  leader_not_found: 0
  num_readyz: 21
  query_stmts_rx: 0
  remote_queries_failed: 0
  remote_requests: 0
  queued_executions_ok: 0
  remote_backups: 0
  remote_requests_failed: 0
mux:
  num_connections_handled: 36
  num_unregistered_handlers: 0
proto:
  num_precompressed_bytes: 0
  num_requests: 0
  num_uncompressed_bytes: 0
  num_uncompressed_requests: 0
  num_compressed_bytes: 0
  num_compressed_requests: 0
  num_compression_misses: 0
queue:
  num_flush: 0
  num_timeout: 0
  statements_rx: 0
  statements_tx: 0
snapshot:
  latest_persist_duration: 0
  latest_persist_size: 0
  snapshots_reaped: 0
  upgrade_fail: 0
  upgrade_ok: 0
cmdline: [/bin/rqlited -node-id rqlite-0 -http-addr 0.0.0.0:4001 -http-adv-addr rqlite-0.rqlite-headless.data.svc.cluster.local:4001 -raft-addr 0.0.0.0:4002 -raft-adv-addr rqlite-0.rqlite-headless.data.svc.cluster.local:4002 -auth=/config/sensitive/users.json -join-as=_system_rqlite -node-cert=/config/sensitive/node.crt -node-key=/config/sensitive/node.key -node-verify-server-name=rqlite.data.svc.cluster.local -node-ca-cert=/config/sensitive/node-ca.crt -node-verify-client -http-cert=/config/client-tls/tls.crt -http-key=/config/client-tls/tls.key -http-ca-cert=/config/sensitive/client-ca.crt -disco-mode=dns-srv -disco-config={"name":"rqlite-headless","service":"raft"} -bootstrap-expect=3 -join-interval=1s -join-attempts=120 -raft-shutdown-stepdown /rqlite]
store:
  num_auto_restores_failed: 0
  num_wal_snapshots_failed: 0
  num_ignored_joins: 0
  num_loads: 0
  num_recoveries: 0
  num_restores_failed: 0
  num_snapshots_failed: 0
  failed_heartbeat_observed: 0
  leader_changes_dropped: 0
  num_auto_restores: 0
  num_provides: 0
  num_snapshots: 0
  num_snapshots_full: 0
  num_user_snapshots_failed: 0
  num_backups: 0
  num_compressed_commands: 0
  leader_changes_observed: 0
  nodes_reaped_ok: 0
  num_auto_restores_skipped: 0
  num_removed_before_joins: 0
  num_boots: 0
  num_uncompressed_commands: 0
  snapshot_wal_size: 0
  num_joins: 1
  num_restores: 0
  num_wal_snapshots: 0
  snapshot_precompact_wal_size: 0
  nodes_reaped_failed: 0
  num_db_stats_errors: 0
  num_snapshots_incremental: 0
  num_user_snapshots: 0
  snapshot_create_duration: 0
  snapshot_persist_duration: 0

[otoolep]

Thanks @jtackaberry -- lack of sufficient test coverage on my part, not my best work. :-(

Thanks for your close attention to detail.

[jtackaberry]

8.14.1 looks good, thanks @otoolep. Related but tangential question: is there a way to pass the nonvoters parameter to the nodes API when using the .nodes command from the CLI? Nothing major, just a tiny bit more convenient from a cluster diagnostics perspective.

[otoolep]

Sure. #1550