mms is not working for me, nodetool status got error

[Yang-Rongquan]

follow the instruction: https://github.com/scylladb/scylla-code-samples/tree/master/mms

nodetool status got error

[root@5c70acb5c2e8 /]# nodetool status nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused) See 'nodetool help' or 'nodetool help <command>'.

[root@5c70acb5c2e8 /]# java -version openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

[root@5c70acb5c2e8 /]# cqlsh Connection error: ('Unable to connect to any servers', {'172.18.0.4': error(111, "Tried connecting to [('172.18.0.4', 9042)]. Last error: Connection refused")})

[tzach]

@Yang-Rongquan thanks for reporting
This issue is some time the result of Scylla process take time to start
Can you please retry the nodetool command after a few minute and share the result?

[Yang-Rongquan]

@tzach tried after cluster up for 30 minutes, still get the same error.

[tzach]

Please include full error log (docker logs)

[Yang-Rongquan]

yang@Mac:~/projects/scylla-code-samples/mms$ docker logs mms_scylla-node2_1
running: (['/opt/scylladb/scripts/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/opt/scylladb/scripts/scylla_io_setup'],)
2021-04-28 06:02:14,748 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/node-exporter.conf" during parsing
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/rsyslog.conf" during parsing
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2021-04-28 06:02:14,748 INFO Included extra file "/etc/supervisord.conf.d/sshd-server.conf" during parsing
2021-04-28 06:02:14,782 INFO RPC interface 'supervisor' initialized
2021-04-28 06:02:14,782 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2021-04-28 06:02:14,782 INFO supervisord started with pid 27
2021-04-28 06:02:15,785 INFO spawned: 'scylla' with pid 30
2021-04-28 06:02:15,787 INFO spawned: 'scylla-housekeeping' with pid 31
2021-04-28 06:02:15,789 INFO spawned: 'sshd' with pid 32
2021-04-28 06:02:15,791 INFO spawned: 'scylla-jmx' with pid 33
2021-04-28 06:02:15,793 INFO spawned: 'rsyslog' with pid 35
2021-04-28 06:02:15,795 INFO spawned: 'node-exporter' with pid 39
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
The key fingerprint is:
SHA256:zJqqvxrJoix77+5SQi8JuEyXdkV7GiIw5LtpwCzZnhY root@93087eb6c6b0
The key's randomart image is:
+--[ED25519 256]--+
|.+ .. |
|. o .. |
|.. ....o . |
|=+o+...o+ |
|BEo. .S |
|+++o o |
|.+B+ o |
|=oo. . |
|++oXO. |
+----[SHA256]-----+
time="2021-04-28T06:02:15Z" level=info msg="Starting node_exporter (version=0.17.0, branch=HEAD, revision=f6f6194a436b9a63d0439abc585c76b19a206b21)" source="node_exporter.go:82"
time="2021-04-28T06:02:15Z" level=info msg="Build context (go=go1.11.2, user=root@322511e06ced, date=20181130-15:51:33)" source="node_exporter.go:83"
time="2021-04-28T06:02:15Z" level=info msg="Enabled collectors:" source="node_exporter.go:90"
time="2021-04-28T06:02:15Z" level=info msg=" - arp" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - bcache" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - bonding" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - conntrack" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - cpu" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - diskstats" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - edac" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - entropy" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - filefd" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - filesystem" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - hwmon" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - infiniband" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - interrupts" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - ipvs" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - loadavg" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - mdadm" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - meminfo" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - netclass" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - netdev" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - netstat" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - nfs" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - nfsd" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - sockstat" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - stat" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - textfile" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - time" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - timex" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - uname" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - vmstat" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - xfs" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg=" - zfs" source="node_exporter.go:97"
time="2021-04-28T06:02:15Z" level=info msg="Listening on :9100" source="node_exporter.go:111"
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --memory 750M --smp 1 --overprovisioned --listen-address 172.18.0.2 --rpc-address 172.18.0.2 --seed-provider-parameters seeds=scylla-node1,scylla-node2 --api-address 0.0.0.0 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, memory: 750M, smp: 1, overprovisioned, listen-address: 172.18.0.2, rpc-address: 172.18.0.2, seed-provider-parameters: seeds=scylla-node1,scylla-node2, api-address: 0.0.0.0, blocked-reactor-notify-ms: 999999999]
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 786432000 available 500000000)
2021-04-28 06:02:16,331 INFO exited: scylla (exit status 1; not expected)
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:oSoxkPjjBfV+PZNY1EIEldj0HgOnwFY25o17A4C234Q root@93087eb6c6b0
The key's randomart image is:
+---[RSA 4096]----+
| . o+X%o. |
|.. . .o +B+O. |
|+ . ..o...= |
| o . ...E=.= o |
| = . o.So + |
| . = . .. .+ . |
| o . |
| . |
| |
+----[SHA256]-----+
2021-04-28 06:02:17,049 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-28 06:02:17,050 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-28 06:02:17,050 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-28 06:02:17,050 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-04-28 06:02:17,050 INFO success: node-exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2021-04-28 06:02:17,702 INFO spawned: 'scylla' with pid 70
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --memory 750M --smp 1 --overprovisioned --listen-address 172.18.0.2 --rpc-address 172.18.0.2 --seed-provider-parameters seeds=scylla-node1,scylla-node2 --api-address 0.0.0.0 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, memory: 750M, smp: 1, overprovisioned, listen-address: 172.18.0.2, rpc-address: 172.18.0.2, seed-provider-parameters: seeds=scylla-node1,scylla-node2, api-address: 0.0.0.0, blocked-reactor-notify-ms: 999999999]
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 786432000 available 500000000)
2021-04-28 06:02:17,902 INFO exited: scylla (exit status 1; not expected)
2021-04-28 06:02:19,907 INFO spawned: 'scylla' with pid 80
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --memory 750M --smp 1 --overprovisioned --listen-address 172.18.0.2 --rpc-address 172.18.0.2 --seed-provider-parameters seeds=scylla-node1,scylla-node2 --api-address 0.0.0.0 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, memory: 750M, smp: 1, overprovisioned, listen-address: 172.18.0.2, rpc-address: 172.18.0.2, seed-provider-parameters: seeds=scylla-node1,scylla-node2, api-address: 0.0.0.0, blocked-reactor-notify-ms: 999999999]
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 786432000 available 500000000)
2021-04-28 06:02:20,100 INFO exited: scylla (exit status 1; not expected)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 966, in send
self.connect()
File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 728, in create_connection
raise err
File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 64, in get_url
return urllib.request.urlopen(path).read().decode('utf-8')
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 197, in
args.func(args)
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 123, in check_version
current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 82, in get_api
return get_json_from_url("http://" + api_address + path)
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 74, in get_json_from_url
retval = result.get(timeout=5)
File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
2021-04-28 06:02:23,266 INFO spawned: 'scylla' with pid 102
Scylla version 4.1.0-0.20200618.b443b2574a8 with build-id 650a4b0c03ae680a9fbf7166c992e99e29b37123 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --memory 750M --smp 1 --overprovisioned --listen-address 172.18.0.2 --rpc-address 172.18.0.2 --seed-provider-parameters seeds=scylla-node1,scylla-node2 --api-address 0.0.0.0 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, memory: 750M, smp: 1, overprovisioned, listen-address: 172.18.0.2, rpc-address: 172.18.0.2, seed-provider-parameters: seeds=scylla-node1,scylla-node2, api-address: 0.0.0.0, blocked-reactor-notify-ms: 999999999]
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 786432000 available 500000000)
2021-04-28 06:02:23,461 INFO exited: scylla (exit status 1; not expected)
2021-04-28 06:02:24,463 INFO gave up: scylla entered FATAL state, too many start retries too quickly

[tzach]

From the following error

Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 786432000 available 500000000)

Its look like you do not have enough RAM to run Scylla, probably becuse you have more Docker instances running.
You can try running all Scylla instances with --smp 1 --memory 750M --overprovisioned 1 this will limit the memory of each, and optimized Scylla to run in an overprovisioned environment

[Yang-Rongquan]

Thanks @tzach
After installing docker, I don't change the settings. Just check the setting, the Mem is configured as 2G.
After changing it to 6G, it works fine now.
docker logs is good tool, thanks for the sharing.