Ubuntu AMI: Scylla server isn't up after reboot #8482

amoskong · 2021-04-14T10:52:12Z

Installation details
Scylla version (or git commit hash): ami-06490d953837d4e6b (eu-north-1)
Instance type: i3.large
Cluster size: 1
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu 20.04 AMI

Description

Created an instance by Ubuntu AMI, wait a while unit scylla-server is up. Then reboot the instance by executing 'sudo reboot'.
Recheck the scylla-server status when system is up.

In my test with 4.6.dev-0.20210408.a8c90a5848, the scylla-server isn't up after reboot, and there is only some mount error.

System log:

system_logs.txt

Mount Errors in reboot

Apr 14 10:38:11 ip-172-31-9-147 systemd[1]: local-fs.target: Found ordering cycle on var-lib-scylla.mount/start
Apr 14 10:38:11 ip-172-31-9-147 systemd[1]: local-fs.target: Found dependency on local-fs.target/start
Apr 14 10:38:11 ip-172-31-9-147 systemd[1]: local-fs.target: Job var-lib-scylla.mount/start deleted to break ordering cycle starting with local-fs.target/start
Apr 14 10:38:11 ip-172-31-9-147 systemd[1]: Created slice scylla.slice.

### after reboot
scyllaadm@ip-172-31-9-147:~$ sudo systemctl status scylla-server
● scylla-server.service - Scylla Server
     Loaded: loaded (/lib/systemd/system/scylla-server.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/scylla-server.service.d
             └─capabilities.conf, dependencies.conf, mounts.conf, sysconfdir.conf
     Active: inactive (dead)
scyllaadm@ip-172-31-9-147:~$ echo $?
3

scyllaadm@ip-172-31-9-147:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30428560 9415740  20996436  31% /
devtmpfs         7805140       0   7805140   0% /dev
tmpfs            7810956       0   7810956   0% /dev/shm
tmpfs            1562192     816   1561376   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs            7810956       0   7810956   0% /sys/fs/cgroup
/dev/loop0         33152   33152         0 100% /snap/amazon-ssm-agent/2996
/dev/loop2         56832   56832         0 100% /snap/core18/1944
/dev/loop1         34176   34176         0 100% /snap/amazon-ssm-agent/3552
/dev/loop3         31872   31872         0 100% /snap/snapd/10707
/dev/loop4         72192   72192         0 100% /snap/lxd/19647
/dev/loop5         69376   69376         0 100% /snap/lxd/18150
/dev/loop6         56832   56832         0 100% /snap/core18/1997
/dev/loop7         33152   33152         0 100% /snap/snapd/11588
tmpfs            1562188       0   1562188   0% /run/user/1000
scyllaadm@ip-172-31-9-147:~$

/Cc @yarongilor @bentsi @roydahan /Cc @syuu1228

The text was updated successfully, but these errors were encountered:

amoskong · 2021-04-14T10:56:56Z

scyllaadm@ip-172-31-9-147:~$ sudo systemctl list-units|grep scylla
  session-5.scope                                               loaded active running   Session 5 of user scyllaadm                                                  
  scylla-node-exporter.service                                  loaded active running   Prometheus exporter for machine metrics                                      
  scylla-helper.slice                                           loaded active active    Slice used to run companion programs to Scylla. Memory, CPU and IO restricted
  scylla-server.slice                                           loaded active active    Slice used to run Scylla. Maximum priority for IO and CPU                    
  scylla.slice                                                  loaded active active    scylla.slice

Last login: Wed Apr 14 10:53:46 2021 from 113.201.56.38

   _____            _ _       _____  ____  
  / ____|          | | |     |  __ \|  _ \ 
 | (___   ___ _   _| | | __ _| |  | | |_) |
  \___ \ / __| | | | | |/ _` | |  | |  _ < 
  ____) | (__| |_| | | | (_| | |__| | |_) |
 |_____/ \___|\__, |_|_|\__,_|_____/|____/ 
               __/ |                       
              |___/                        

Version:
       4.6.dev-0.20210408.a8c90a5848
Nodetool:
	nodetool help
CQL Shell:
	cqlsh
More documentation available at: 
	http://www.scylladb.com/doc/
By default, Scylla sends certain information about this node to a data collection server. For information, see http://www.scylladb.com/privacy/

    Failed mounting RAID volume!

ScyllaDB aborted startup because of RAID volume missing.
To see status, run
 'systemctl status scylla-server'

This EC2 instance is optimized for Scylla.

amoskong · 2021-04-14T11:11:39Z

scyllaadm@ip-172-31-9-147:~$ sudo systemctl status scylla.slice
● scylla.slice
     Loaded: loaded
     Active: active since Wed 2021-04-14 10:58:27 UTC; 11min ago
      Tasks: 4
     Memory: 12.7M
     CGroup: /scylla.slice
             └─scylla-helper.slice
               └─scylla-node-exporter.service
                 └─457 /opt/scylladb/node_exporter/node_exporter --collector.interrupts

Warning: journal has been rotated since unit was started, output may be incomplete.
scyllaadm@ip-172-31-9-147:~$ sudo systemctl status scylla-server.slice
● scylla-server.slice - Slice used to run Scylla. Maximum priority for IO and CPU
     Loaded: loaded (/lib/systemd/system/scylla-server.slice; static; vendor preset: enabled)
     Active: active since Wed 2021-04-14 10:58:27 UTC; 12min ago
      Tasks: 0
     Memory: 0B (swap max: 1B)
        CPU: 0
     CGroup: /scylla.slice/scylla-server.slice

Warning: journal has been rotated since unit was started, output may be incomplete.
scyllaadm@ip-172-31-9-147:~$ sudo systemctl status scylla-helper.slice
● scylla-helper.slice - Slice used to run companion programs to Scylla. Memory, CPU and IO restricted
     Loaded: loaded (/lib/systemd/system/scylla-helper.slice; static; vendor preset: enabled)
    Drop-In: /etc/systemd/system/scylla-helper.slice.d
             └─memory.conf
     Active: active since Wed 2021-04-14 10:58:27 UTC; 12min ago
      Tasks: 4
     Memory: 12.7M (high: 1.1G max: 1.3G limit: 1.3G)
        CPU: 27ms
     CGroup: /scylla.slice/scylla-helper.slice
             └─scylla-node-exporter.service
               └─457 /opt/scylladb/node_exporter/node_exporter --collector.interrupts

Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=thermal_zone
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=time
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=timex
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=udp_queues
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=uname
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=vmstat
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=xfs
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:112 collector=zfs
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=node_exporter.go:191 msg="Listening on" address=>
Apr 14 10:58:33 ip-172-31-9-147 node_exporter[457]: level=info ts=2021-04-14T10:58:33.925Z caller=tls_config.go:170 msg="TLS is disabled and it ca>

yarongilor · 2021-04-14T11:18:29Z

The issue originally happened in scylla-master/gemini-/gemini-1tb-10h:
The SCT error was:

2021-04-09 09:16:15.810: (DisruptionEvent Severity.ERROR): type=RebuildStreamingErr subtype=end node=Node gemini-1tb-10h-master-db-node-8595859b-5 [13.51.159.232 | 10.0.0.224] (seed: False) duration=981 error=RetryError[Wait for: Node gemini-1tb-10h-master-db-node-8595859b-5 [13.51.159.232 | 10.0.0.224] (seed: False): Waiting for DB services to be up: timeout - 300 seconds - expired]
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/wait.py", line 58, in wait_for
res = retry.call(func, **kwargs)
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 358, in call
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 331, in iter
raise retry_exc.reraise()
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 168, in reraise
raise self
tenacity.RetryError: RetryError[<Future at 0x7fd2ec6698b0 state=finished returned bool>]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2904, in wrapper
result = method(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3780, in disrupt
self.call_random_disrupt_method(disrupt_methods=self.disrupt_methods_list)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1082, in call_random_disrupt_method
self.execute_disrupt_method(disrupt_method)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1089, in execute_disrupt_method
disrupt_method()
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2586, in disrupt_rebuild_streaming_err
self.break_streaming_task_and_rebuild(task='rebuild')
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2570, in break_streaming_task_and_rebuild
self.target_node.wait_db_up(verbose=True, timeout=300)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1328, in wait_db_up
wait.wait_for(func=self.db_up, step=60, text=text, timeout=timeout, throw_exc=True)
File "/home/ubuntu/scylla-cluster-tests/sdcm/wait.py", line 69, in wait_for
raise RetryError(err)
tenacity.RetryError: RetryError[Wait for: Node gemini-1tb-10h-master-db-node-8595859b-5 [13.51.159.232 | 10.0.0.224] (seed: False): Waiting for DB services to be up: timeout - 300 seconds - expired]

Installation details
Kernel version: 5.4.0-1035-aws
Scylla version (or git commit hash): 4.6.dev-0.20210408.a8c90a5848
Cluster size: 3 nodes (i3.4xlarge)
Scylla running with shards number (live nodes):
gemini-1tb-10h-master-db-node-8595859b-1 (13.51.86.128 | 10.0.1.37): 14 shards
gemini-1tb-10h-master-db-node-8595859b-3 (13.51.108.164 | 10.0.0.195): 14 shards
gemini-1tb-10h-master-db-node-8595859b-5 (13.51.159.232 | 10.0.0.224): 14 shards
Scylla running with shards number (terminated nodes):
gemini-1tb-10h-master-db-node-8595859b-2 (13.51.47.215 | 10.0.2.138): 14 shards
gemini-1tb-10h-master-db-node-8595859b-4 (13.48.104.13 | 10.0.2.35): 14 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-06490d953837d4e6b (aws: eu-north-1)

Gemini command:
/$HOME/gemini -d --duration 36000s --warmup 7200s -c 100 -m mixed -f --non-interactive --cql-features normal --max-mutation-retries 5 --max-mutation-retries-backoff 500ms --async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms --replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '3'}" --oracle-replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '1'}" --test-cluster=10.0.1.37,10.0.2.138,10.0.0.195 --outfile /home/centos/gemini_result_9feb1b18-2bb7-4adf-bf9c-7ace981d862c.log --seed 83 --oracle-cluster=10.0.0.214
Gemini version: 1.7.4

Test: gemini-1tb-10h
Test name: gemini_test.GeminiTest.test_load_random_with_nemesis
Test config file(s):

gemini-1tb-10h.yaml

Issue description

====================================

PUT ISSUE DESCRIPTION HERE

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor 8595859b-f1ec-4828-934f-4ec486e41940
Show all stored logs command: $ hydra investigate show-logs 8595859b-f1ec-4828-934f-4ec486e41940

Test id: 8595859b-f1ec-4828-934f-4ec486e41940

Logs:
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_092140/grafana-screenshot-gemini-1tb-10h-scylla-per-server-metrics-nemesis-20210409_092518-gemini-1tb-10h-master-monitor-node-8595859b-1.png
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_092140/grafana-screenshot-overview-20210409_092140-gemini-1tb-10h-master-monitor-node-8595859b-1.png
db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_093003/db-cluster-8595859b.zip
loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_093003/loader-set-8595859b.zip
monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_093003/monitor-set-8595859b.zip
sct-runner - https://cloudius-jenkins-test.s3.amazonaws.com/8595859b-f1ec-4828-934f-4ec486e41940/20210409_093003/sct-runner-8595859b.zip

Jenkins job URL

syuu1228 · 2021-04-15T20:25:11Z

I found that mdmonitor.service does not started on the AMI: #8494
But the AMI does not able to resolve by fix #8494, it seems like has more problem.
systemd detected ordering cycle, deleted var-lib-systemd-coredump.mount and var-lib-scylla.mount, so RAID volume failed to mount:

scyllaadm@ip-172-31-43-110:~$ sudo journalctl -xe|grep local-fs.target
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Found ordering cycle on var-lib-systemd-coredump.mount/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Found dependency on var-lib-scylla.mount/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Found dependency on local-fs.target/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Job var-lib-systemd-coredump.mount/start deleted to break ordering cycle starting with local-fs.target/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Found ordering cycle on var-lib-scylla.mount/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Found dependency on local-fs.target/start
Apr 15 19:57:59 ip-172-31-43-110 systemd[1]: local-fs.target: Job var-lib-scylla.mount/start deleted to break ordering cycle starting with local-fs.target/start
-- Subject: A start job for unit local-fs.target has finished successfully
-- A start job for unit local-fs.target has finished successfully.

syuu1228 · 2021-04-15T20:26:49Z

Also systemd-analyze says same error:

root@ip-172-31-43-110:~# systemd-analyze verify default.target
snap-core18-1944.mount: Unit is bound to inactive unit dev-loop5.device. Stopping, too.
snap-lxd-19647.mount: Unit is bound to inactive unit dev-loop7.device. Stopping, too.
snap-lxd-18150.mount: Unit is bound to inactive unit dev-loop4.device. Stopping, too.
snap-snapd-11588.mount: Unit is bound to inactive unit dev-loop3.device. Stopping, too.
snap-amazon\x2dssm\x2dagent-2996.mount: Unit is bound to inactive unit dev-loop0.device. Stopping, too.
snap-snapd-10707.mount: Unit is bound to inactive unit dev-loop6.device. Stopping, too.
snap-core18-1997.mount: Unit is bound to inactive unit dev-loop2.device. Stopping, too.
var-lib-scylla.mount: Found ordering cycle on local-fs.target/start
var-lib-scylla.mount: Found dependency on var-lib-systemd-coredump.mount/start
var-lib-scylla.mount: Found dependency on var-lib-scylla.mount/start
var-lib-scylla.mount: Job local-fs.target/start deleted to break ordering cycle starting with var-lib-scylla.mount/start

syuu1228 · 2021-04-15T22:09:44Z

Seems like DefaultDependencies causes ordering cycle.
After added "DefaultDependencies=no" to var-lib-scylla.mount and var-lib-systemd-coredump.mount, these services become able to launch at startup.

scyllaadm@ip-172-31-43-110:~$ systemctl status var-lib-scylla.mount
● var-lib-scylla.mount - Scylla data directory
     Loaded: loaded (/etc/systemd/system/var-lib-scylla.mount; enabled; vendor >
     Active: active (mounted) since Thu 2021-04-15 22:06:43 UTC; 42s ago
      Where: /var/lib/scylla
       What: /dev/md0
      Tasks: 0 (limit: 76216)
     Memory: 92.0K
     CGroup: /system.slice/var-lib-scylla.mount

Apr 15 22:06:43 ip-172-31-43-110 systemd[1]: Mounting Scylla data directory...
Apr 15 22:06:43 ip-172-31-43-110 systemd[1]: Mounted Scylla data directory.
scyllaadm@ip-172-31-43-110:~$ systemctl status var-lib-systemd-coredump.mount 
● var-lib-systemd-coredump.mount - Save coredump to scylla data directory
     Loaded: loaded (/etc/systemd/system/var-lib-systemd-coredump.mount; enable>
     Active: active (mounted) since Thu 2021-04-15 22:06:43 UTC; 48s ago
      Where: /var/lib/systemd/coredump
       What: /dev/md0
      Tasks: 0 (limit: 76216)
     Memory: 28.0K
     CGroup: /system.slice/var-lib-systemd-coredump.mount

Apr 15 22:06:43 ip-172-31-43-110 systemd[1]: Mounting Save coredump to scylla d>
Apr 15 22:06:43 ip-172-31-43-110 systemd[1]: Mounted Save coredump to scylla da>

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes scylladb#8482

amoskong · 2021-04-19T08:16:33Z

The issue occurred in https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-cdc-100gb-4h-test/214/console
test id: a9a3b229-7baa-46d9-b143-64edf1281938
db-cluster : https://cloudius-jenkins-test.s3.amazonaws.com/a9a3b229-7baa-46d9-b143-64edf1281938/20210419_054039/db-cluster-a9a3b229.zip
The scylla-server in db-node-5 wasn't up after reboot.

ami-02e10cb6a73685809 (eu-north-1)
distro: Ubuntu20
4.6.dev-0.20210418.dbd0b9a3ef

juliayakovlev · 2021-04-19T08:18:08Z

Same problem in longevity-cdc.

Installation details
Kernel version: 5.4.0-1035-aws
Scylla version (or git commit hash): 4.6.dev-0.20210418.dbd0b9a3ef
Cluster size: 6 nodes (i3.4xlarge)
Scylla running with shards number (live nodes):
longevity-cdc-100gb-4h-master-db-node-14fa21e8-1 (13.51.55.251 | 10.0.1.228): 14 shards
longevity-cdc-100gb-4h-master-db-node-14fa21e8-2 (13.51.157.85 | 10.0.3.115): 14 shards
longevity-cdc-100gb-4h-master-db-node-14fa21e8-3 (13.53.37.224 | 10.0.2.12): 14 shards
longevity-cdc-100gb-4h-master-db-node-14fa21e8-4 (13.53.214.90 | 10.0.1.150): 14 shards
longevity-cdc-100gb-4h-master-db-node-14fa21e8-5 (13.48.24.157 | 10.0.2.135): 14 shards
longevity-cdc-100gb-4h-master-db-node-14fa21e8-7 (13.49.64.188 | 10.0.2.200): 14 shards
Scylla running with shards number (terminated nodes):
longevity-cdc-100gb-4h-master-db-node-14fa21e8-6 (13.49.72.88 | 10.0.1.185): 14 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-069f3ff348a725731 (aws: eu-north-1)

Test: longevity-cdc-100gb-4h-test
Test name: longevity_test.LongevityTest.test_custom_time
Test config file(s):

longevity-cdc-100gb-4h.yaml

Restore Monitor Stack command: $ hydra investigate show-monitor 14fa21e8-06ca-44d1-881b-1e10fa7b7378
Show all stored logs command: $ hydra investigate show-logs 14fa21e8-06ca-44d1-881b-1e10fa7b7378

Test id: 14fa21e8-06ca-44d1-881b-1e10fa7b7378

Logs:

db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/14fa21e8-06ca-44d1-881b-1e10fa7b7378/20210418_221052/db-cluster-14fa21e8.zip

sct-runner - https://cloudius-jenkins-test.s3.amazonaws.com/14fa21e8-06ca-44d1-881b-1e10fa7b7378/20210418_221052/sct-runner-14fa21e8.zip

Jenkins job URL

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes #8482 Closes #8495 (cherry picked from commit 0b01e1a)

All mount units generated by systemd-fstab-generator have "Before=local-fs.target", it is opposite of our mount units. Seems like it is the reason we got "ordering cycle" error on scylladb#8482, we need to move local-fs.target to Before= to fix the error. Fixes scylladb#8761

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes scylladb#8482 Closes scylladb#8495 (cherry picked from commit 0b01e1a)

avikivity · 2021-10-28T15:54:57Z

Already backported to all vulnerable branches, removing "Backport candidate" label.

syuu1228 mentioned this issue Apr 15, 2021

scylla_raid_setup: Ubuntu does not able to enable mdmonitor.service #8494

Closed

syuu1228 added a commit to syuu1228/scylla that referenced this issue Apr 15, 2021

dist: add DefaultDependencies=no to .mount units

5b61a64

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes scylladb#8482

syuu1228 added a commit to syuu1228/scylla that referenced this issue Apr 15, 2021

dist: add DefaultDependencies=no to .mount units

c36da93

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes scylladb#8482

syuu1228 added a commit to syuu1228/scylla that referenced this issue Apr 15, 2021

dist: add DefaultDependencies=no to .mount units

76c9c00

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes scylladb#8482

syuu1228 mentioned this issue Apr 15, 2021

dist: add DefaultDependencies=no to .mount units #8495

Closed

slivne assigned syuu1228 and penberg Apr 18, 2021

slivne added high bug cloud/aws AWS related issues labels Apr 18, 2021

scylladb-promoter closed this as completed in 0b01e1a Apr 20, 2021

scylladb-promoter added the Backport candidate label Apr 20, 2021

syuu1228 mentioned this issue May 31, 2021

scylla node failed to start scylla service after reboot. #8716

Closed

penberg pushed a commit that referenced this issue May 31, 2021

dist: add DefaultDependencies=no to .mount units

d92a266

To avoid ordering cycle error on Ubuntu, add DefaultDependencies=no on .mount units. Fixes #8482 Closes #8495 (cherry picked from commit 0b01e1a)

syuu1228 mentioned this issue May 31, 2021

dist: The way to fix 'ordering cycle' issue on var-lib-scylla.mount might be wrong #8761

Closed

syuu1228 mentioned this issue May 31, 2021

Revert adding DefaultDependencies=no, move local-fs.target to Before= #8762

Open

avikivity removed the Backport candidate label Oct 28, 2021

DoronArazii added this to the 4.6 milestone May 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu AMI: Scylla server isn't up after reboot #8482

Ubuntu AMI: Scylla server isn't up after reboot #8482

amoskong commented Apr 14, 2021 •

edited

amoskong commented Apr 14, 2021 •

edited

amoskong commented Apr 14, 2021

yarongilor commented Apr 14, 2021 •

edited

syuu1228 commented Apr 15, 2021

syuu1228 commented Apr 15, 2021

syuu1228 commented Apr 15, 2021

amoskong commented Apr 19, 2021 •

edited

juliayakovlev commented Apr 19, 2021

avikivity commented Oct 28, 2021

Ubuntu AMI: Scylla server isn't up after reboot #8482

Ubuntu AMI: Scylla server isn't up after reboot #8482

Comments

amoskong commented Apr 14, 2021 • edited

Description

System log:

Mount Errors in reboot

amoskong commented Apr 14, 2021 • edited

amoskong commented Apr 14, 2021

yarongilor commented Apr 14, 2021 • edited

syuu1228 commented Apr 15, 2021

syuu1228 commented Apr 15, 2021

syuu1228 commented Apr 15, 2021

amoskong commented Apr 19, 2021 • edited

juliayakovlev commented Apr 19, 2021

avikivity commented Oct 28, 2021

amoskong commented Apr 14, 2021 •

edited

amoskong commented Apr 14, 2021 •

edited

yarongilor commented Apr 14, 2021 •

edited

amoskong commented Apr 19, 2021 •

edited