Scylla bench failed to create keyspace table

[igorsimb]

Installation details

Kernel Version: 5.15.0-1019-aws
Scylla version (or git commit hash): 5.1.0~rc1-20220902.d10aee15e7e9 with build-id c127c717ecffa082ce97b94100d62b2549abe486
Cluster size: 6 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

• longevity-10gb-3h-5-1-db-node-78ae0112-8 (176.34.80.148 | 2a05:d018:12e3:f000:8a38:85b5:8fe5:771f) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-7 (54.75.127.156 | 2a05:d018:12e3:f000:14ce:a626:6162:dfe4) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-6 (176.34.146.162 | 2a05:d018:12e3:f000:affa:ad40:fa63:4369) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-5 (34.245.102.145 | 2a05:d018:12e3:f000:9213:4097:b3dd:195c) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-4 (54.75.118.167 | 2a05:d018:12e3:f000:738b:7afd:7e00:e71e) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-3 (3.250.79.48 | 2a05:d018:12e3:f000:9317:bc87:eecd:3666) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-2 (34.245.15.190 | 2a05:d018:12e3:f000:104e:4f54:e859:c91c) (shards: 7)
• longevity-10gb-3h-5-1-db-node-78ae0112-1 (52.214.248.193 | 2a05:d018:12e3:f000:669d:4e40:da51:6385) (shards: 7)

OS / Image: ami-03f1a2d7537cf2e31 (aws: eu-west-1)

Test: longevity-10gb-3h-ipv6-test
Test id: 78ae0112-a4b1-46b5-ae78-95e4f4a74067
Test name: scylla-5.1/longevity/longevity-10gb-3h-ipv6-test
Test config file(s):

• longevity-10gb-3h.yaml

Issue description

Scylla bench was supposed to create a keyspace table, but it failed with the following error:

2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:738b:7afd:7e00:e71e\" connectAddress=\"2a05:d018:12e3:f000:738b:7afd:7e00:e71e\" peer=\"2a05:d018:12e3:f000:738b:7afd:7e00:e71e\" rpc_address=\"2a05:d018:12e3:f000:738b:7afd:7e00:e71e\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:738b:7afd:7e00:e71e\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"5f78e037-6477-4a82-abdd-02b785f53073\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:affa:ad40:fa63:4369\" connectAddress=\"2a05:d018:12e3:f000:affa:ad40:fa63:4369\" peer=\"2a05:d018:12e3:f000:affa:ad40:fa63:4369\" rpc_address=\"2a05:d018:12e3:f000:affa:ad40:fa63:4369\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:affa:ad40:fa63:4369\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"02f7c76f-8502-407e-9325-933c0802b6f7\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:669d:4e40:da51:6385\" connectAddress=\"2a05:d018:12e3:f000:669d:4e40:da51:6385\" peer=\"2a05:d018:12e3:f000:669d:4e40:da51:6385\" rpc_address=\"2a05:d018:12e3:f000:669d:4e40:da51:6385\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:669d:4e40:da51:6385\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"97da62fb-0746-40e4-8ec1-72c431066312\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:9317:bc87:eecd:3666\" connectAddress=\"2a05:d018:12e3:f000:9317:bc87:eecd:3666\" peer=\"2a05:d018:12e3:f000:9317:bc87:eecd:3666\" rpc_address=\"2a05:d018:12e3:f000:9317:bc87:eecd:3666\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:9317:bc87:eecd:3666\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"13e1ada7-9145-4948-a373-081a5706e278\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:9213:4097:b3dd:195c\" connectAddress=\"2a05:d018:12e3:f000:9213:4097:b3dd:195c\" peer=\"2a05:d018:12e3:f000:9213:4097:b3dd:195c\" rpc_address=\"2a05:d018:12e3:f000:9213:4097:b3dd:195c\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:9213:4097:b3dd:195c\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"9ee31e37-4304-46d5-b54a-2eb1442c411d\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:31 error: failed to connect to "[HostInfo hostname=\"2a05:d018:12e3:f000:104e:4f54:e859:c91c\" connectAddress=\"2a05:d018:12e3:f000:104e:4f54:e859:c91c\" peer=\"<nil>\" rpc_address=\"2a05:d018:12e3:f000:104e:4f54:e859:c91c\" broadcast_address=\"2a05:d018:12e3:f000:104e:4f54:e859:c91c\" preferred_ip=\"<nil>\" connect_addr=\"2a05:d018:12e3:f000:104e:4f54:e859:c91c\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"4165fc5d-bba1-46cd-8735-aeb5ecafaf01\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: Keyspace 'ks_truncate_large_partition' does not exist
2022/09/07 13:32:32 no connections were made when creating the session

Scylla bench is expected to create a keyspace, but for some reason it failed to do so.

• Restore Monitor Stack command: $ hydra investigate show-monitor 78ae0112-a4b1-46b5-ae78-95e4f4a74067
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 78ae0112-a4b1-46b5-ae78-95e4f4a74067

Logs:

• db-cluster-78ae0112.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/78ae0112-a4b1-46b5-ae78-95e4f4a74067/20220907_154731/db-cluster-78ae0112.tar.gz
• monitor-set-78ae0112.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/78ae0112-a4b1-46b5-ae78-95e4f4a74067/20220907_154731/monitor-set-78ae0112.tar.gz
• loader-set-78ae0112.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/78ae0112-a4b1-46b5-ae78-95e4f4a74067/20220907_154731/loader-set-78ae0112.tar.gz
• sct-runner-78ae0112.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/78ae0112-a4b1-46b5-ae78-95e4f4a74067/20220907_154731/sct-runner-78ae0112.tar.gz
• parallel-timelines-report-78ae0112.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/78ae0112-a4b1-46b5-ae78-95e4f4a74067/20220907_154731/parallel-timelines-report-78ae0112.tar.gz

Jenkins job URL

[fruch]

fixed in 5c4d702228342f39df8c6d01b810e78299bbb659