Issues when rpc_address != listen_address - Scylla Manager 3.1

[ricardoborenstein]

Experiencing issues with their ScyllaDB setup where they have separate rpc_address and listen_address.
Prospect is able to do CQL clients only on rpc_address, but the Scylla Manager is set up using the listen_address of a node and the cluster list works fine. However, when they try to do sctool restore, it gives an error saying "get CQL cluster session: gocql: unable to create session: unable to discover protocol version: dial tcp [<listen_address>]:9042: connect: connection refused". It seems that it is trying to perform some CQL operation on listen_address, which is not going to work.

Submit tried to update the cluster in sctool using --host <rpc_address> but then it errors out saying "connect: connection refused - make sure the IP is correct and access to port 10001 is unblocked." This is probably because rpc_address port 10001 isn't listening and it only listens on listen_address.
To address this issue, I suggest two solutions. Firstly, I recommended that prospect updates the cluster in Scylla Manager to use the rpc_address instead of the listen_address. This can be done with the sctool cluster update command, specifying the -host option with the rpc_address. Secondly, I suggested that the prospect changes the Scylla configuration on the node to listen for CQL connections on the listen_address. This would involve changing the rpc_address at scylla.yaml file to the listen_address, and then restart the Scylla service.
However, tried both solutions, but they failed.

# sctool status

╭────┬──────────┬──────────┬────────────────────────────┬─────────────┬──────┬─────────┬────────┬───────┬──────────────────────────────────────╮
│UN  │ CQL      │ REST     │    Address                 │ Uptime      │ CPUs │ Memory  │ Scylla │ Agent │ Host ID                              │
├────┼──────────┼──────────┼────────────────────────────┼─────────────┼──────┼─────────┼────────┼───────┼──────────────────────────────────────┤
│ UN │ UP (8ms) │ UP (0ms) │ xxxx:xxxx:x:x:xx:xxx:664:11 │ 2360h11m51s │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ 080acc55-8710-4502-9ea3-0ef30689d8d4 │
│ UN │ UP (8ms) │ UP (1ms) │ xxxx:xxxx:x:x:xx:xxx:664:12 │ 2359h28m5s  │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ bafe23f9-1228-493f-b0e1-f13c8db1a1be │
│ UN │ UP (7ms) │ UP (0ms) │ xxxx:xxxx:x:x:xx:xxx:664:13 │ 213h6m1s    │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ 95d7086f-eaac-417d-9db9-153e5e0c09ac │
│ UN │ UP (9ms) │ UP (0ms) │ xxxx:xxxx:x:x:xx:xxx:664:14 │ 213h18m41s  │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ ea407b4e-9f5e-436d-b349-daabe11cde1f │
│ UN │ UP (8ms) │ UP (5ms) │ xxxx:xxxx:x:x:xx:xxx:664:15 │ 2162h8m11s  │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ b3e81033-cee1-48fa-94a9-7074a7d15979 │
╰────┴──────────┴──────────┴────────────────────────────┴─────────────┴──────┴─────────┴────────┴───────┴──────────────────────────────────────╯
root@stg-scylladbmanager ~ Thu Jun 01 09:51:50
#
root@stg-scylladbmanager ~ Thu Jun 01 09:51:50
#
root@stg-scylladbmanager ~ Thu Jun 01 09:51:50
# telnet xxxx:xxxx:x:x:xx:xxx:664:11 9042
Trying 2a0b:b582::10:123:64:11...
telnet: connect to address 2a0b:b582::10:123:64:11: Connection refused
root@stg-scylladbmanager ~ Thu Jun 01 09:51:52
# telnet xxxx:xxxx:x:x:xx:xxx:648:11 9042
Trying xxxx:xxxx::xx:xxx:48:11...
Connected to xxxx:xxxx:x:x:xx:xxx:648:11.
Escape character is '^]'.
^CConnection closed by foreign host.# telnet xxxx:xxxx:x:x:xx:xxx:664:11 10001
Trying 2a0b:b582::10:123:64:11...
Connected to xxxx:xxxx:x:x:xx:xxx:664:11.
Escape character is '^]'.
^CConnection closed by foreign host.
root@stg-scylladbmanager ~ Thu Jun 01 09:54:59
# telnet xxxx:xxxx:x:x:xx:xxx:664:48 10001
Trying 2a0b:b582::10:123:64:48...
telnet: connect to address 2a0b:b582::10:123:64:48: No route to host


# cat /etc/scylla/scylla.yaml | grep -e "listen_address\|rpc_address" | grep -v "#"
listen_address: 2a0b:b582::10:123:64:11
rpc_address: xxxx:xxxx::xx:xxx:48:11# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192 eth1 eth2 eth3
sources:
services: dhcpv6-client ssh zabbix-agent
ports: 161/udp 9042/tcp 7001/tcp 7199/tcp 123/udp 61621/tcp 10001/tcp 9180/tcp 3000/tcp 9100/tcp 1514/tcp 19042/tcp 10050/tcp
protocols:
forward: no
masquerade: yes
forward-ports:
source-ports:
icmp-blocks:
rich rules:# telnet 2a0b:b582::10:123:64:11 9042
Trying 2a0b:b582::10:123:64:11...
telnet: connect to address 2a0b:b582::10:123:64:11: Connection refusedroot@stg-scylladb101 ~ Thu Jun 01 09:58:30
# telnet 2a0b:b582::10:123:64:11 10001
Trying 2a0b:b582::10:123:64:11...
Connected to 2a0b:b582::10:123:64:11.
Escape character is '^]'.
^CConnection closed by foreign host.root@stg-scylladb101 ~ Thu Jun 01 09:58:36
# telnet xxxx:xxxx::xx:xxx:48:11 9042
Trying xxxx:xxxx::xx:xxx:48:11...
Connected to xxxx:xxxx::xx:xxx:48:11.
Escape character is '^]'.
^CConnection closed by foreign host.root@stg-scylladb101 ~ Thu Jun 01 09:58:48
# telnet xxxx:xxxx::xx:xxx:48:11 10001
Trying xxxx:xxxx::xx:xxx:48:11...
telnet: connect to address xxxx:xxxx::xx:xxx:48:11: Connection refused

1. Setting(adding) listen_address in: scylla-manager-agent.yaml

Jun 05 12:04:11 stg-scylladb101 scylla-manager-agent[3619903]: STARTUP ERROR: yaml: unmarshal errors:
Jun 05 12:04:11 stg-scylladb101 scylla-manager-agent[3619903]:   line 2: field listen_address not found in type agent.Config

2. Set https: xxxx:xxxx::xx:xxx:48:11:10001 [same as rpc_address] in: scylla-manager-agent.yaml

Jun 05 11:56:31 stg-scylladb101 scylla-manager-agent[3619686]: STARTUP ERROR: HTTPS server start: listen tcp: address xxxx:xxxx::xx:xxx:48:11:10001: too many colons in address

3. Set https: 0.0.0.0:10001 [ to listen to all address both rpc and listen_address] in: scylla-manager-agent.yamlAgent works fine, listens to both Set xxxx:xxxx::xx:xxx:48:11:10001 and 2a0b:b582::10:123:64:11:10001
   Scylla-Manager nodes can also reach it(verified via telnet).

Trying xxxx:xxxx::xx:xxx:48:11...
Connected to xxxx:xxxx:x:x:xx:xxx:648:11.
Escape character is '^]'.

Updated Cluster:

# sctool cluster update -c "ClusterName" --host xxxx:xxxx:x:x:xx:xxx:648:11
root@stg-scylladbmanager ~ Mon Jun 05 11:34:18

Yet it shows IP address of subnet 64:

╭────┬───────────┬──────────┬────────────────────────────┬─────────────┬──────┬─────────┬────────┬───────┬──────────────────────────────────────╮
│    │ CQL       │ REST     │ Address                    │ Uptime      │ CPUs │ Memory  │ Scylla │ Agent │ Host ID                              │
├────┼───────────┼──────────┼────────────────────────────┼─────────────┼──────┼─────────┼────────┼───────┼──────────────────────────────────────┤
│ UN │ UP (9ms)  │ UP (3ms) │ xxxx:xxxx:x:x:xx:xxx:664:11 │ 2457h57m35s │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ 080acc55-8710-4502-9ea3-0ef30689d8d4 │
│ UN │ UP (10ms) │ UP (4ms) │ xxxx:xxxx:x:x:xx:xxx:664:12 │ 2457h11m19s │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ bafe23f9-1228-493f-b0e1-f13c8db1a1be │
│ UN │ UP (10ms) │ UP (0ms) │ xxxx:xxxx:x:x:xx:xxx:664:13 │ 310h51m15s  │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ 95d7086f-eaac-417d-9db9-153e5e0c09ac │
│ UN │ UP (9ms)  │ UP (2ms) │ xxxx:xxxx:x:x:xx:xxx:664:14 │ 311h1m40s   │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ ea407b4e-9f5e-436d-b349-daabe11cde1f │
│ UN │ UP (7ms)  │ UP (2ms) │ xxxx:xxxx:x:x:xx:xxx:664:15 │ 2259h51m10s │ 4    │ 31.180G │ 5.1.5  │ 3.1.0 │ b3e81033-cee1-48fa-94a9-7074a7d15979 │
╰────┴───────────┴──────────┴────────────────────────────┴─────────────┴──────┴─────────┴────────┴───────┴──────────────────────────────────────╯

And restore fails:

# sctool restore  --cluster "ClusterName"  -L 's3:scylla-backup01-stg-20bda0ad' --
num-retries 1 -T sm_20230531105239UTC -K cycling.cyclist_base --restore-tables --dry-run --show-tables
Error: get restore target: list all views of cluster 4e81165b-7057-4de6-b2bf-2e553c684ad7: get CQL cluster session: gocql: unable to create session: unable to discover protocol version: dial tcp [xxxx:xxxx::xx:xxx:64:14]:9042: connect: connection refused
Trace ID: wxOQApYdSyWTPb7mXiJNxQ (grep in scylla-manager logs)

Archive.zip

[AdamStawarz]

@karol-kokoszka could you take a look ?

[tzach]

@fgelcer do we have IPv6 tests for Manager?

[karol-kokoszka]

We don't have testing environment for scylla-manager that would let us to test (integration-tests) against IPv6 setup.
Let me address that first.

I'll create another docker-compose that will setup the infra with IPv6 and then will reproduce the issue.

[ShlomiBalalis]

@fgelcer do we have IPv6 tests for Manager?

We do, but it did not include a restore

[ricardoborenstein]

Can we have a list of what was tested with Scylla Manager 3.1 and IPV6?

cc @vladzcloudius @gcarmin

[ricardoborenstein]

Hello, @karol-kokoszka @ShlomiBalalis
Any update? We need to move with this ASAP since is the end of POC and this is one of the KPIs that needs be to accomplished

@mykaul @roydahan - Can you help me?

[fgelcer]

Hello, @karol-kokoszka @ShlomiBalalis Any update? We need to move with this ASAP since is the end of POC and this is one of the KPIs that needs be to accomplished

@mykaul @roydahan - Can you help me?

@ricardoborenstein , here is the test we run using IPv6 --> https://github.com/scylladb/scylla-cluster-tests/blob/405f9eb84957c57b523278545ea1e06809079bc3/mgmt_cli_test.py#L361

and some of the results --> https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.1/job/centos-sanity-ipv6-test/ (last good result was on April 19th)
and here is the same test, but on master, from Argus PoV --> https://argus.scylladb.com/workspace?state=WyJkMTM0NmMxZC01YzNkLTQwYmItOWM2ZC04OWFiZWJkNDNjZmIiXQ

@ShlomiBalalis , why don't we see these tests for 3.1 on Argus? cc @k0machi

[gcarmin]

thanks @fgelcer - so from the links you've shared I understand that for Scyla manger with IPV6 you are testing only:

1. Run the repair test.
   2) Run test_mgmt_cluster test.
   3) test_mgmt_cluster_healthcheck
   4) test_client_encryption

So it means no backup and restore.
Also - from the 2nd link - it seems like that all tests where failing.

BTW - is this test is only for IPV6?
Do we have the same for general manager tests?

In my POV - it means that manager 3.1 does not support IPV6
@vladzcloudius @harel-z @tomer-sandler @AdamStawarz @noellymedina @ricardoborenstein @hopugop FYI.

[mykaul]

@karol-kokoszka - please triage this (3.2?)

[fgelcer]

thanks @fgelcer - so from the links you've shared I understand that for Scyla manger with IPV6 you are testing only:

1. Run the repair test.
   2) Run test_mgmt_cluster test.
   3) test_mgmt_cluster_healthcheck
   4) test_client_encryption

So it means no backup and restore. Also - from the 2nd link - it seems like that all tests where failing.

@gcarmin , well, the documentation is wrong, cause it does run basic_backup --> https://github.com/scylladb/scylla-cluster-tests/blob/405f9eb84957c57b523278545ea1e06809079bc3/mgmt_cli_test.py#LL371C37-L371C37
but right, i don't see it running any restore here

@ShlomiBalalis , please fix the docstring and add a simple restore test to this sanity test ASAP

BTW - is this test is only for IPV6?
Do we have the same for general manager tests?

this test runs ALSO for IPv6, but we have it being called in several different tests:

$ grep -r test_manager_sanity .
./mgmt_cli_test.py:    def test_manager_sanity(self):
./jenkins-pipelines/manager/enterprise-ami-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/ubuntu20-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/centos-manager-sanity-ipv6.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/centos-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/debian10-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/debian11-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager/ubuntu22-manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',
./jenkins-pipelines/manager-sanity.jenkinsfile:    test_name: 'mgmt_cli_test.MgmtCliTest.test_manager_sanity',

[fgelcer]

So it means no backup and restore. Also - from the 2nd link - it seems like that all tests where failing.

@gcarmin , in this run , the repair failed, but the backup (as it is the 1st called test) passed successful... @ShlomiBalalis , can you please summarize the results + failures + issues on this sanity run of manager 3.1 test?

[ricardoborenstein]

@karol-kokoszka , do you need more info? Do we have a workaround for it or should we get the prospect a new release?

[ricardoborenstein]

Hi team,
@mykaul
@karol-kokoszka

We need a solution for this - the prospect is waiting.

[mykaul]

Hi team, @mykaul @karol-kokoszka

We need a solution for this - the prospect is waiting.

Unless it's a misconfiguration, it's clear the agent doesn't support IPv6 adress, and I don't anticipate a solution quickly here. Probably in 3.2.
@karol-kokoszka - can you assign someone to validate this, or perhaps there's a workaround?

[mykaul]

I'm not sure you should set https: xxxx:xxxx::xx:xxx:48:11:10001 in the yaml file of the agent (I think it should be without the https?)

[mykaul]

dial tcp [xxxx:xxxx::xx:xxx:64:14]:9042: connect: connection refused - did you check if anything is listening on that port using telnet? You seem to have tested other stuff using it, but not this one?

[ricardoborenstein]

I'm not sure you should set https: xxxx:xxxx::xx:xxx:48:11:10001 in the yaml file of the agent (I think it should be without the https?)

No, I just checked on our documentation - should have https - https://manager.docs.scylladb.com/stable/config/scylla-manager-agent-config.html#scylla-manager-agent-config

dial tcp [xxxx:xxxx::xx:xxx:64:14]:9042: connect: connection refused - did you check if anything is listening on that port using telnet? You seem to have tested other stuff using it, but not this one?

I just asked the prospect to run the command.

[mykaul]

@ricardoborenstein - I'm not so sure - https://stackoverflow.com/questions/50347441/tls-dial-returns-too-many-colons-in-address ?

[ricardoborenstein]

@ricardoborenstein - I'm not so sure - https://stackoverflow.com/questions/50347441/tls-dial-returns-too-many-colons-in-address ?

I'm not sure too, but if yes, where should we change something?

[mykaul]

@ricardoborenstein - I'm not so sure - https://stackoverflow.com/questions/50347441/tls-dial-returns-too-many-colons-in-address ?

I'm not sure too, but if yes, where should we change something?

What happens without the 'http://' ? Just the IPv6 address? (again, guessing, if we have the setup - I wouldn't bother the customer with this!)

[ricardoborenstein]

@mykaul As @karol-kokoszka is OOO, who can handle this to investigate until his back?

Just let's make sure that we have something to give to the customer, it's been 2 days and we did not have any triage yet.
Could you please prioritize?

[AdamStawarz]

@DoronArazii ^^

[ricardoborenstein]

@mykaul @DoronArazii @karol-kokoszka - Any updates?

[DoronArazii]

@mykaul @DoronArazii @karol-kokoszka - Any updates?

I'm not sure what's your expectations regarding the updates.

IIUC the bottom line is - Manager 3.1 does not support IPV6
It needs to be assess by Dev (@karol-kokoszka / @Michal-Leszczynski) for Manager 3.2
/Cc @tzach

[tzach]

IIUC the bottom line is - Manager 3.1 does not support IPV6

Manager 3.1 supports IPv6 as Manager 3.0.
We need to understand the root cause of the issue.
As a follow-up, we need to add tests and improve the doc as necessary.

[ricardoborenstein]

IIUC the bottom line is - Manager 3.1 does not support IPV6

Manager 3.1 supports IPv6 as Manager 3.0. We need to understand the root cause of the issue. As a follow-up, we need to add tests and improve the doc as necessary.

@fgelcer created a task to add the IPV6 and Restore
#https://github.com/scylladb/qa-tasks/issues/1209

[ricardoborenstein]

Hi @tzach @mykaul @DoronArazii

What the next steps here?
I need to give some answer to the customer :/