Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore forwarding may be problematic #1088

Closed
otoolep opened this issue Oct 24, 2022 · 7 comments
Closed

Restore forwarding may be problematic #1088

otoolep opened this issue Oct 24, 2022 · 7 comments

Comments

@otoolep
Copy link
Member

otoolep commented Oct 24, 2022 •
edited

There are reports that restoring via a follower is not working correctly. I can't reproduce any issues.

Test case 1: spin up a 2-node cluster by hand and restore through the follower.

Start node 1:

rqlited -node-id 1 data.1

Start node 2 and join it to node 1:

rqlited -http-addr localhost:4003 -raft-addr localhost:4004 -join http://localhost:4001 -node-id 2 data.2

Restore a SQLite file into the cluster via the follower:

curl -v -XPOST localhost:4003/db/load -H "Content-type: application/octet-stream" --data-binary @restore.sqlite
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1:4003...
* TCP_NODELAY set
* connect to ::1 port 4003 failed: Connection refused
*   Trying 127.0.0.1:4003...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4003 (#0)
> POST /db/load HTTP/1.1
> Host: localhost:4003
> User-Agent: curl/7.68.0
> Accept: */*
> Content-type: application/octet-stream
> Content-Length: 8192
> Expect: 100-continue
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Rqlite-Served-By: localhost:4002
< X-Rqlite-Version: 7
< Date: Mon, 24 Oct 2022 13:30:57 GMT
< Content-Length: 14
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
{"results":[]}

Query the leader:

url -G 'localhost:4001/db/query?pretty&timings' --data-urlencode 'q=SELECT * FROM foo'
{
    "results": [
        {
            "columns": [
                "id",
                "name"
            ],
            "types": [
                "integer",
                "text"
            ],
            "values": [
                [
                    1,
                    "fiona"
                ]
            ],
            "time": 0.000144014
        }
    ],
    "time": 0.000647214
}

Query the follower, using none consistency (so the local SQLite database is queried:

curl -G 'localhost:4003/db/query?pretty&timings?level=None' --data-urlencode 'q=SELECT * FROM foo'
{
    "results": [
        {
            "columns": [
                "id",
                "name"
            ],
            "types": [
                "integer",
                "text"
            ],
            "values": [
                [
                    1,
                    "fiona"
                ]
            ]
        }
    ]
}

Test case 2: I did something very similar on Minikube, using the following configs, and restored fine through a follower (the Load Balancer routed me to a follower).
Services:

apiVersion: v1
kind: Service
metadata:
  name: rqlite-svc-internal
spec:
  clusterIP: None
  publishNotReadyAddresses: True
  selector:
    app: rqlite
  ports:
    - protocol: TCP
      port: 4001
      targetPort: 4001
---
apiVersion: v1
kind: Service
metadata:
  name: rqlite-svc
spec:
  type: LoadBalancer
  selector:
    app: rqlite
  ports:
    - protocol: TCP
      port: 4001
      targetPort: 4001

Stateful set:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rqlite
spec:
  selector:
    matchLabels:
      app: rqlite # has to match .spec.template.metadata.labels
  serviceName: rqlite-svc-internal
  replicas: 3 # by default is 1
  podManagementPolicy: "Parallel"
  template:
    metadata:
      labels:
        app: rqlite # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: rqlite
        image: rqlite/rqlite:local2
        args: ["-disco-mode=dns","-disco-config={\"name\":\"rqlite-svc-internal\"}","-bootstrap-expect","3", "-join-interval=1s", "-join-attempts=120"]
        ports:
        - containerPort: 4001
          name: rqlite
        readinessProbe:
          httpGet:
            scheme: HTTP
            path: /readyz
            port: 4001
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: rqlite-file
          mountPath: /rqlite/file
  volumeClaimTemplates:
  - metadata:
      name: rqlite-file
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 1Gi
@sgalsaleh
Copy link
Contributor

I'm frequently getting into a state when loading into a follower does not restore the data. I'm still not 100% sure how to get to that state, but these are the commands I run to reproduce the issue with the current state:

1- Create a table with some data:

curl -XPOST '10.96.1.223:4001/db/execute?pretty' -H "Content-Type: application/json" -d '[
    "CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT, age INTEGER)",
    "INSERT INTO foo(name, age) VALUES(\"fiona\", 20)"
]'

2- Make sure the data is there:

curl -XPOST '10.96.1.223:4001/db/query?pretty' -H "Content-Type: application/json" -d '[
    "SELECT * FROM foo"
]'

3- Take a backup:

curl -f http://10.96.1.223:4001/db/backup -o rqlite.sql

4- Delete the data:

curl -XPOST '10.96.1.223:4001/db/execute?pretty' -H "Content-Type: application/json" -d '[
    "DELETE FROM foo WHERE 1"
]'

5- Make sure data is not here:

curl -XPOST '10.32.12.2:4001/db/query?pretty' -H "Content-Type: application/json" -d '[
    "SELECT * FROM foo"
]'

6- Restore the data directly into a follower or through the load balancer (and making sure the Served-By header is present:

curl -f http://10.96.1.223:4001/db/load -H "Content-type: application/octet-stream" --data-binary @rqlite.sql -v

7- Check that the data is restored:

curl -XPOST '10.32.12.2:4001/db/query?pretty' -H "Content-Type: application/json" -d '[
    "SELECT * FROM foo"
]'

In my case, the data is not restored. I'll attach the output of /nodes and /status for each node shortly.

@sgalsaleh
Copy link
Contributor

/nodes endpoint output for each node:

~$ curl 10.32.12.2:4001/nodes?pretty
{
    "kotsadm-rqlite-0": {
        "api_addr": "http://kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": true,
        "time": 0.000017684
    },
    "kotsadm-rqlite-1": {
        "api_addr": "http://kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.000762598
    },
    "kotsadm-rqlite-2": {
        "api_addr": "http://kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.00065168
    }
}

~$ curl 10.32.4.4:4001/nodes?pretty
{
    "kotsadm-rqlite-0": {
        "api_addr": "http://kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": true,
        "time": 0.000693552
    },
    "kotsadm-rqlite-1": {
        "api_addr": "http://kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.000028445
    },
    "kotsadm-rqlite-2": {
        "api_addr": "http://kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.000734964
    }
}

~$ curl 10.32.0.10:4001/nodes?pretty
{
    "kotsadm-rqlite-0": {
        "api_addr": "http://kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": true,
        "time": 0.000796612
    },
    "kotsadm-rqlite-1": {
        "api_addr": "http://kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.000858562
    },
    "kotsadm-rqlite-2": {
        "api_addr": "http://kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "reachable": true,
        "leader": false,
        "time": 0.000022714
    }
}

@sgalsaleh
Copy link
Contributor

sgalsaleh commented Oct 24, 2022 •
edited

/status of leader node

~$ curl 10.32.12.2:4001/status?pretty
{
    "build": {
        "branch": "master",
        "build_time": "2022-10-23T22:48:23-0400",
        "commit": "27b8409da97c9091915b51d8a2053326aee8e3c7",
        "compiler": "gc",
        "version": "v7.9.1"
    },
    "cluster": {
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "api_addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "https": "false"
    },
    "disco": {
        "last_addresses": [
            "10.32.12.2:4001"
        ],
        "last_contact": "2022-10-24T16:27:25.806407191Z",
        "mode": "dns",
        "name": "kotsadm-rqlite-internal",
        "port": 4001
    },
    "http": {
        "auth": "disabled",
        "bind_addr": "[::]:4001",
        "cluster": {
            "conn_pool_stats": {
                "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 4,
                    "max_open_connections": 64,
                    "open_connections": 4
                },
                "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 4,
                    "max_open_connections": 64,
                    "open_connections": 4
                }
            },
            "local_node_addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "timeout": 30000000000
        },
        "queue": {
            "_default": {
                "batch_size": 16,
                "max_size": 128,
                "sequence_number": 0,
                "timeout": 50000000
            }
        }
    },
    "last_backup_time": "2022-10-24T16:33:10.68341259Z",
    "node": {
        "current_time": "2022-10-24T17:18:15.594999013Z",
        "start_time": "2022-10-24T16:27:25.801483232Z",
        "uptime": "50m49.793516635s"
    },
    "os": {
        "executable": "/bin/rqlited",
        "hostname": "kotsadm-rqlite-0",
        "page_size": 4096,
        "pid": 1,
        "ppid": 0
    },
    "runtime": {
        "GOARCH": "amd64",
        "GOMAXPROCS": 8,
        "GOOS": "linux",
        "num_cpu": 8,
        "num_goroutine": 53,
        "version": "go1.18"
    },
    "store": {
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "apply_timeout": "10s",
        "db_applied_index": 1684,
        "db_conf": {
            "memory": true,
            "fk_constraints": false
        },
        "dir": "/rqlite/file/data",
        "dir_size": 1048576,
        "election_timeout": "1s",
        "fsm_index": 1684,
        "heartbeat_timeout": "1s",
        "leader": {
            "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "node_id": "kotsadm-rqlite-0"
        },
        "no_freelist_sync": false,
        "node_id": "kotsadm-rqlite-0",
        "nodes": [
            {
                "id": "kotsadm-rqlite-0",
                "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-1",
                "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-2",
                "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            }
        ],
        "observer": {
            "dropped": 0,
            "observed": 1
        },
        "raft": {
            "applied_index": 1684,
            "bolt": {
                "FreePageN": 0,
                "PendingPageN": 5,
                "FreeAlloc": 20480,
                "FreelistInuse": 56,
                "TxN": 591,
                "OpenTxN": 0,
                "TxStats": {
                    "PageCount": 7375,
                    "PageAlloc": 30208000,
                    "CursorCount": 6252,
                    "NodeCount": 5459,
                    "NodeDeref": 20,
                    "Rebalance": 0,
                    "RebalanceTime": 0,
                    "Split": 229,
                    "Spill": 5677,
                    "SpillTime": 34332321,
                    "Write": 9055,
                    "WriteTime": 4185697515
                }
            },
            "commit_index": 1684,
            "fsm_pending": 0,
            "last_contact": 0,
            "last_log_index": 1684,
            "last_log_term": 2,
            "last_snapshot_index": 0,
            "last_snapshot_term": 0,
            "latest_configuration": "[{Suffrage:Voter ID:kotsadm-rqlite-0 Address:kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-1 Address:kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-2 Address:kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002}]",
            "latest_configuration_index": 0,
            "log_size": 1048576,
            "num_peers": 2,
            "protocol_version": 3,
            "protocol_version_max": 3,
            "protocol_version_min": 0,
            "snapshot_version_max": 1,
            "snapshot_version_min": 0,
            "state": "Leader",
            "term": 2
        },
        "request_marshaler": {
            "compression_batch": 5,
            "compression_size": 150,
            "force_compression": false
        },
        "snapshot_interval": 30000000000,
        "snapshot_threshold": 8192,
        "sqlite3": {
            "compile_options": [
                "ATOMIC_INTRINSICS=1",
                "COMPILER=gcc-9.4.0",
                "DEFAULT_AUTOVACUUM",
                "DEFAULT_CACHE_SIZE=-2000",
                "DEFAULT_FILE_FORMAT=4",
                "DEFAULT_JOURNAL_SIZE_LIMIT=-1",
                "DEFAULT_MMAP_SIZE=0",
                "DEFAULT_PAGE_SIZE=4096",
                "DEFAULT_PCACHE_INITSZ=20",
                "DEFAULT_RECURSIVE_TRIGGERS",
                "DEFAULT_SECTOR_SIZE=4096",
                "DEFAULT_SYNCHRONOUS=2",
                "DEFAULT_WAL_AUTOCHECKPOINT=1000",
                "DEFAULT_WAL_SYNCHRONOUS=1",
                "DEFAULT_WORKER_THREADS=0",
                "ENABLE_DBSTAT_VTAB",
                "ENABLE_FTS3",
                "ENABLE_FTS3_PARENTHESIS",
                "ENABLE_RTREE",
                "ENABLE_UPDATE_DELETE_LIMIT",
                "MALLOC_SOFT_LIMIT=1024",
                "MAX_ATTACHED=10",
                "MAX_COLUMN=2000",
                "MAX_COMPOUND_SELECT=500",
                "MAX_DEFAULT_PAGE_SIZE=8192",
                "MAX_EXPR_DEPTH=1000",
                "MAX_FUNCTION_ARG=127",
                "MAX_LENGTH=1000000000",
                "MAX_LIKE_PATTERN_LENGTH=50000",
                "MAX_MMAP_SIZE=0x7fff0000",
                "MAX_PAGE_COUNT=1073741823",
                "MAX_PAGE_SIZE=65536",
                "MAX_SQL_LENGTH=1000000000",
                "MAX_TRIGGER_DEPTH=1000",
                "MAX_VARIABLE_NUMBER=32766",
                "MAX_VDBE_OP=250000000",
                "MAX_WORKER_THREADS=8",
                "MUTEX_PTHREADS",
                "OMIT_DEPRECATED",
                "OMIT_LOAD_EXTENSION",
                "OMIT_SHARED_CACHE",
                "SYSTEM_MALLOC",
                "TEMP_STORE=1",
                "THREADSAFE=1"
            ],
            "conn_pool_stats": {
                "ro": {
                    "max_open_connections": 0,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                },
                "rw": {
                    "max_open_connections": 1,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                }
            },
            "db_size": 212992,
            "mem_stats": {
                "cache_size": -2000,
                "freelist_count": 0,
                "hard_heap_limit": 0,
                "max_page_count": 1073741823,
                "page_count": 52,
                "page_size": 4096,
                "soft_heap_limit": 0
            },
            "path": ":memory:",
            "ro_dsn": "file:/dnjPgPjciMhMKCqrlDfi?mode=ro\u0026vfs=memdb\u0026_txlock=deferred\u0026_fk=false",
            "rw_dsn": "file:/dnjPgPjciMhMKCqrlDfi?mode=rw\u0026vfs=memdb\u0026_txlock=immediate\u0026_fk=false",
            "version": "3.38.5"
        },
        "startup_on_disk": false,
        "trailing_logs": 10240
    }
}

@sgalsaleh
Copy link
Contributor

/status of first follower node:

~$ curl 10.32.4.4:4001/status?pretty
{
    "build": {
        "branch": "master",
        "build_time": "2022-10-23T22:48:23-0400",
        "commit": "27b8409da97c9091915b51d8a2053326aee8e3c7",
        "compiler": "gc",
        "version": "v7.9.1"
    },
    "cluster": {
        "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "api_addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "https": "false"
    },
    "disco": {
        "last_addresses": [
            "10.32.4.4:4001",
            "10.32.12.2:4001",
            "10.32.0.10:4001"
        ],
        "last_contact": "2022-10-24T16:28:09.746695135Z",
        "mode": "dns",
        "name": "kotsadm-rqlite-internal",
        "port": 4001
    },
    "http": {
        "auth": "disabled",
        "bind_addr": "[::]:4001",
        "cluster": {
            "conn_pool_stats": {
                "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 2,
                    "max_open_connections": 64,
                    "open_connections": 8
                },
                "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 4,
                    "max_open_connections": 64,
                    "open_connections": 4
                }
            },
            "local_node_addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "timeout": 30000000000
        },
        "queue": {
            "_default": {
                "batch_size": 16,
                "max_size": 128,
                "sequence_number": 0,
                "timeout": 50000000
            }
        }
    },
    "node": {
        "current_time": "2022-10-24T17:19:01.552424278Z",
        "start_time": "2022-10-24T16:28:09.742168495Z",
        "uptime": "50m51.810256164s"
    },
    "os": {
        "executable": "/bin/rqlited",
        "hostname": "kotsadm-rqlite-1",
        "page_size": 4096,
        "pid": 1,
        "ppid": 0
    },
    "runtime": {
        "GOARCH": "amd64",
        "GOMAXPROCS": 8,
        "GOOS": "linux",
        "num_cpu": 8,
        "num_goroutine": 32,
        "version": "go1.18"
    },
    "store": {
        "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "apply_timeout": "10s",
        "db_applied_index": 0,
        "db_conf": {
            "memory": true,
            "fk_constraints": false
        },
        "dir": "/rqlite/file/data",
        "dir_size": 1048576,
        "election_timeout": "1s",
        "fsm_index": 1710,
        "heartbeat_timeout": "1s",
        "leader": {
            "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "node_id": "kotsadm-rqlite-0"
        },
        "no_freelist_sync": false,
        "node_id": "kotsadm-rqlite-1",
        "nodes": [
            {
                "id": "kotsadm-rqlite-0",
                "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-1",
                "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-2",
                "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            }
        ],
        "observer": {
            "dropped": 0,
            "observed": 1
        },
        "raft": {
            "applied_index": 1710,
            "bolt": {
                "FreePageN": 0,
                "PendingPageN": 5,
                "FreeAlloc": 20480,
                "FreelistInuse": 56,
                "TxN": 75,
                "OpenTxN": 0,
                "TxStats": {
                    "PageCount": 7466,
                    "PageAlloc": 30580736,
                    "CursorCount": 5258,
                    "NodeCount": 5527,
                    "NodeDeref": 20,
                    "Rebalance": 0,
                    "RebalanceTime": 0,
                    "Split": 233,
                    "Spill": 5760,
                    "SpillTime": 35258379,
                    "Write": 9154,
                    "WriteTime": 4219951839
                }
            },
            "commit_index": 1710,
            "fsm_pending": 0,
            "last_contact": "34.21242ms",
            "last_log_index": 1710,
            "last_log_term": 2,
            "last_snapshot_index": 0,
            "last_snapshot_term": 0,
            "latest_configuration": "[{Suffrage:Voter ID:kotsadm-rqlite-0 Address:kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-1 Address:kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-2 Address:kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002}]",
            "latest_configuration_index": 0,
            "log_size": 1048576,
            "num_peers": 2,
            "protocol_version": 3,
            "protocol_version_max": 3,
            "protocol_version_min": 0,
            "snapshot_version_max": 1,
            "snapshot_version_min": 0,
            "state": "Follower",
            "term": 2
        },
        "request_marshaler": {
            "compression_batch": 5,
            "compression_size": 150,
            "force_compression": false
        },
        "snapshot_interval": 30000000000,
        "snapshot_threshold": 8192,
        "sqlite3": {
            "compile_options": [
                "ATOMIC_INTRINSICS=1",
                "COMPILER=gcc-9.4.0",
                "DEFAULT_AUTOVACUUM",
                "DEFAULT_CACHE_SIZE=-2000",
                "DEFAULT_FILE_FORMAT=4",
                "DEFAULT_JOURNAL_SIZE_LIMIT=-1",
                "DEFAULT_MMAP_SIZE=0",
                "DEFAULT_PAGE_SIZE=4096",
                "DEFAULT_PCACHE_INITSZ=20",
                "DEFAULT_RECURSIVE_TRIGGERS",
                "DEFAULT_SECTOR_SIZE=4096",
                "DEFAULT_SYNCHRONOUS=2",
                "DEFAULT_WAL_AUTOCHECKPOINT=1000",
                "DEFAULT_WAL_SYNCHRONOUS=1",
                "DEFAULT_WORKER_THREADS=0",
                "ENABLE_DBSTAT_VTAB",
                "ENABLE_FTS3",
                "ENABLE_FTS3_PARENTHESIS",
                "ENABLE_RTREE",
                "ENABLE_UPDATE_DELETE_LIMIT",
                "MALLOC_SOFT_LIMIT=1024",
                "MAX_ATTACHED=10",
                "MAX_COLUMN=2000",
                "MAX_COMPOUND_SELECT=500",
                "MAX_DEFAULT_PAGE_SIZE=8192",
                "MAX_EXPR_DEPTH=1000",
                "MAX_FUNCTION_ARG=127",
                "MAX_LENGTH=1000000000",
                "MAX_LIKE_PATTERN_LENGTH=50000",
                "MAX_MMAP_SIZE=0x7fff0000",
                "MAX_PAGE_COUNT=1073741823",
                "MAX_PAGE_SIZE=65536",
                "MAX_SQL_LENGTH=1000000000",
                "MAX_TRIGGER_DEPTH=1000",
                "MAX_VARIABLE_NUMBER=32766",
                "MAX_VDBE_OP=250000000",
                "MAX_WORKER_THREADS=8",
                "MUTEX_PTHREADS",
                "OMIT_DEPRECATED",
                "OMIT_LOAD_EXTENSION",
                "OMIT_SHARED_CACHE",
                "SYSTEM_MALLOC",
                "TEMP_STORE=1",
                "THREADSAFE=1"
            ],
            "conn_pool_stats": {
                "ro": {
                    "max_open_connections": 0,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                },
                "rw": {
                    "max_open_connections": 1,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                }
            },
            "db_size": 212992,
            "mem_stats": {
                "cache_size": -2000,
                "freelist_count": 0,
                "hard_heap_limit": 0,
                "max_page_count": 1073741823,
                "page_count": 52,
                "page_size": 4096,
                "soft_heap_limit": 0
            },
            "path": ":memory:",
            "ro_dsn": "file:/rAbaMpOnMJpjcdAHhICF?mode=ro\u0026vfs=memdb\u0026_txlock=deferred\u0026_fk=false",
            "rw_dsn": "file:/rAbaMpOnMJpjcdAHhICF?mode=rw\u0026vfs=memdb\u0026_txlock=immediate\u0026_fk=false",
            "version": "3.38.5"
        },
        "startup_on_disk": false,
        "trailing_logs": 10240
    }
}

@sgalsaleh
Copy link
Contributor

/status of 2nd follower node:

~$ curl 10.32.0.10:4001/status?pretty
{
    "build": {
        "branch": "master",
        "build_time": "2022-10-23T22:48:23-0400",
        "commit": "27b8409da97c9091915b51d8a2053326aee8e3c7",
        "compiler": "gc",
        "version": "v7.9.1"
    },
    "cluster": {
        "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "api_addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4001",
        "https": "false"
    },
    "disco": {
        "last_addresses": [
            "10.32.0.10:4001",
            "10.32.4.4:4001",
            "10.32.12.2:4001"
        ],
        "last_contact": "2022-10-24T16:28:09.828974635Z",
        "mode": "dns",
        "name": "kotsadm-rqlite-internal",
        "port": 4001
    },
    "http": {
        "auth": "disabled",
        "bind_addr": "[::]:4001",
        "cluster": {
            "conn_pool_stats": {
                "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 2,
                    "max_open_connections": 64,
                    "open_connections": 7
                },
                "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002": {
                    "idle": 4,
                    "max_open_connections": 64,
                    "open_connections": 4
                }
            },
            "local_node_addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "timeout": 30000000000
        },
        "queue": {
            "_default": {
                "batch_size": 16,
                "max_size": 128,
                "sequence_number": 0,
                "timeout": 50000000
            }
        }
    },
    "node": {
        "current_time": "2022-10-24T17:19:45.888642185Z",
        "start_time": "2022-10-24T16:28:09.825037916Z",
        "uptime": "51m36.063604755s"
    },
    "os": {
        "executable": "/bin/rqlited",
        "hostname": "kotsadm-rqlite-2",
        "page_size": 4096,
        "pid": 1,
        "ppid": 0
    },
    "runtime": {
        "GOARCH": "amd64",
        "GOMAXPROCS": 8,
        "GOOS": "linux",
        "num_cpu": 8,
        "num_goroutine": 33,
        "version": "go1.18"
    },
    "store": {
        "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
        "apply_timeout": "10s",
        "db_applied_index": 0,
        "db_conf": {
            "memory": true,
            "fk_constraints": false
        },
        "dir": "/rqlite/file/data",
        "dir_size": 1048576,
        "election_timeout": "1s",
        "fsm_index": 1734,
        "heartbeat_timeout": "1s",
        "leader": {
            "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
            "node_id": "kotsadm-rqlite-0"
        },
        "no_freelist_sync": false,
        "node_id": "kotsadm-rqlite-2",
        "nodes": [
            {
                "id": "kotsadm-rqlite-0",
                "addr": "kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-1",
                "addr": "kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            },
            {
                "id": "kotsadm-rqlite-2",
                "addr": "kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002",
                "suffrage": "Voter"
            }
        ],
        "observer": {
            "dropped": 0,
            "observed": 1
        },
        "raft": {
            "applied_index": 1734,
            "bolt": {
                "FreePageN": 0,
                "PendingPageN": 5,
                "FreeAlloc": 20480,
                "FreelistInuse": 56,
                "TxN": 74,
                "OpenTxN": 0,
                "TxStats": {
                    "PageCount": 7586,
                    "PageAlloc": 31072256,
                    "CursorCount": 5326,
                    "NodeCount": 5621,
                    "NodeDeref": 20,
                    "Rebalance": 0,
                    "RebalanceTime": 0,
                    "Split": 236,
                    "Spill": 5857,
                    "SpillTime": 37615533,
                    "Write": 9297,
                    "WriteTime": 3974975373
                }
            },
            "commit_index": 1734,
            "fsm_pending": 0,
            "last_contact": "24.705874ms",
            "last_log_index": 1734,
            "last_log_term": 2,
            "last_snapshot_index": 0,
            "last_snapshot_term": 0,
            "latest_configuration": "[{Suffrage:Voter ID:kotsadm-rqlite-0 Address:kotsadm-rqlite-0.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-1 Address:kotsadm-rqlite-1.kotsadm-rqlite-internal.default.svc.cluster.local:4002} {Suffrage:Voter ID:kotsadm-rqlite-2 Address:kotsadm-rqlite-2.kotsadm-rqlite-internal.default.svc.cluster.local:4002}]",
            "latest_configuration_index": 0,
            "log_size": 1048576,
            "num_peers": 2,
            "protocol_version": 3,
            "protocol_version_max": 3,
            "protocol_version_min": 0,
            "snapshot_version_max": 1,
            "snapshot_version_min": 0,
            "state": "Follower",
            "term": 2
        },
        "request_marshaler": {
            "compression_batch": 5,
            "compression_size": 150,
            "force_compression": false
        },
        "snapshot_interval": 30000000000,
        "snapshot_threshold": 8192,
        "sqlite3": {
            "compile_options": [
                "ATOMIC_INTRINSICS=1",
                "COMPILER=gcc-9.4.0",
                "DEFAULT_AUTOVACUUM",
                "DEFAULT_CACHE_SIZE=-2000",
                "DEFAULT_FILE_FORMAT=4",
                "DEFAULT_JOURNAL_SIZE_LIMIT=-1",
                "DEFAULT_MMAP_SIZE=0",
                "DEFAULT_PAGE_SIZE=4096",
                "DEFAULT_PCACHE_INITSZ=20",
                "DEFAULT_RECURSIVE_TRIGGERS",
                "DEFAULT_SECTOR_SIZE=4096",
                "DEFAULT_SYNCHRONOUS=2",
                "DEFAULT_WAL_AUTOCHECKPOINT=1000",
                "DEFAULT_WAL_SYNCHRONOUS=1",
                "DEFAULT_WORKER_THREADS=0",
                "ENABLE_DBSTAT_VTAB",
                "ENABLE_FTS3",
                "ENABLE_FTS3_PARENTHESIS",
                "ENABLE_RTREE",
                "ENABLE_UPDATE_DELETE_LIMIT",
                "MALLOC_SOFT_LIMIT=1024",
                "MAX_ATTACHED=10",
                "MAX_COLUMN=2000",
                "MAX_COMPOUND_SELECT=500",
                "MAX_DEFAULT_PAGE_SIZE=8192",
                "MAX_EXPR_DEPTH=1000",
                "MAX_FUNCTION_ARG=127",
                "MAX_LENGTH=1000000000",
                "MAX_LIKE_PATTERN_LENGTH=50000",
                "MAX_MMAP_SIZE=0x7fff0000",
                "MAX_PAGE_COUNT=1073741823",
                "MAX_PAGE_SIZE=65536",
                "MAX_SQL_LENGTH=1000000000",
                "MAX_TRIGGER_DEPTH=1000",
                "MAX_VARIABLE_NUMBER=32766",
                "MAX_VDBE_OP=250000000",
                "MAX_WORKER_THREADS=8",
                "MUTEX_PTHREADS",
                "OMIT_DEPRECATED",
                "OMIT_LOAD_EXTENSION",
                "OMIT_SHARED_CACHE",
                "SYSTEM_MALLOC",
                "TEMP_STORE=1",
                "THREADSAFE=1"
            ],
            "conn_pool_stats": {
                "ro": {
                    "max_open_connections": 0,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                },
                "rw": {
                    "max_open_connections": 1,
                    "open_connections": 1,
                    "in_use": 0,
                    "idle": 1,
                    "wait_count": 0,
                    "wait_duration": 0,
                    "max_idle_closed": 0,
                    "max_idle_time_closed": 0,
                    "max_lifetime_closed": 0
                }
            },
            "db_size": 212992,
            "mem_stats": {
                "cache_size": -2000,
                "freelist_count": 0,
                "hard_heap_limit": 0,
                "max_page_count": 1073741823,
                "page_count": 52,
                "page_size": 4096,
                "soft_heap_limit": 0
            },
            "path": ":memory:",
            "ro_dsn": "file:/djcdHjKrcOrPCgaCCdki?mode=ro\u0026vfs=memdb\u0026_txlock=deferred\u0026_fk=false",
            "rw_dsn": "file:/djcdHjKrcOrPCgaCCdki?mode=rw\u0026vfs=memdb\u0026_txlock=immediate\u0026_fk=false",
            "version": "3.38.5"
        },
        "startup_on_disk": false,
        "trailing_logs": 10240
    }
}

@otoolep otoolep mentioned this issue Oct 24, 2022
Merged
@otoolep
Copy link
Member Author

otoolep commented Oct 24, 2022

Fixed by #1089

@otoolep otoolep closed this as completed Oct 24, 2022
@otoolep
Copy link
Member Author

otoolep commented Oct 24, 2022

Issue was the load commands between nodes were limited at 2^16 bits in size. This was too small, and has been increased to 2^64 bytes in size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@otoolep @sgalsaleh