alias centos user added linked to scylladm should also hard link /home/centos directory to /home/scylladm

[slivne]

currently we add scylladm user and also add an alias to it centos

[shlomi@localhost scylla-machine-image]$ cat aws/ami/files/99_user_alias.cfg
runcmd:
  - /sbin/userdel centos
  - /sbin/groupdel centos
  - /sbin/useradd -o -u 1001 -g scyllaadm -d /home/scyllaadm centos
  - chown scyllaadm:scyllaadm /home/centos
  - ln -sf /home/centos/.bash_profile /home/scyllaadm/.bash_profile

this change is not complete since we leave behind the old /home/centos directory - we should remove that directory and hard link it to /home/scyllaadm

alternatively - we can reuse the /home/centos directory and make the /home/scyllaadm a hard link to it

[avikivity]

hard links don't work with directories. But a symlink can work.

[fgelcer]

this is the 1st version i'm seeing this failure 4.5.dev-0.20210311.f6fc32c8d with build-id ba2a77c9d7aacdabd3ca87ebfec0b6b79ad91880

it is on our longevity 200gb with TLS enabled. (this is the test config longevity-200GB-48h-verifier-LimitedMonkey-tls.yaml)

[fgelcer]

this is the 1st version i'm seeing this failure 4.5.dev-0.20210311.f6fc32c8d with build-id ba2a77c9d7aacdabd3ca87ebfec0b6b79ad91880

it is on our longevity 200gb with TLS enabled. (this is the test config longevity-200GB-48h-verifier-LimitedMonkey-tls.yaml)

if that is of any help

[slivne]

@penberg / @syuu1228 this is blocking the cloud from using the new AMI's please prioritize

[syuu1228]

@slivne i will check it now.

[yarongilor]

possibly reproduced similarly with an error of:

2021-04-09 03:12:54.497: (TestFrameworkEvent Severity.ERROR), source=LongevityTest.SetUp()
exception=[Node longevity-tls-50gb-3d-master-db-node-670b5bf6-1 [13.48.190.14 | 10.0.1.195] (seed: True)] NodeSetupFailed: Encountered a bad command exit code!
Command: 'bash -cxe "\nmkdir -p ~/.cassandra/\ncp /tmp/ssl_conf/client/cqlshrc ~/.cassandra/\nsudo mkdir -p /etc/scylla/\nsudo rm -rf /etc/scylla/ssl_conf/\nsudo mv -f /tmp/ssl_conf/ /etc/scylla/\n"'
Exit code: 1
Stdout:
Stderr:
+ '[' -z '' ']'
+ return
+ case $- in
+ return
+ mkdir -p /home/scyllaadm/.cassandra/
mkdir: cannot create directory ‘/home/scyllaadm/.cassandra/’: Permission denied
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3426, in node_setup
cl_inst.node_setup(_node, **setup_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 4079, in node_setup
self.node_config_setup(node, ','.join(self.seed_nodes_ips), self.get_endpoint_snitch())
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 923, in node_config_setup
node.config_setup(**setup_params)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1809, in config_setup
self.config_client_encrypt()
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1899, in config_client_encrypt
self.remoter.run('bash -cxe "%s"' % setup_script)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 604, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 61, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 596, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 537, in _run_execute
result = connection.run(**command_kwargs)
File "<decorator-gen-3>", line 2, in run
File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 30, in opens
return method(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 721, in run
return self._run(self._remote_runner(), command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/context.py", line 101, in _run
return runner.run(command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 291, in run
return self._run_body(command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 442, in _run_body
raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'bash -cxe "\nmkdir -p ~/.cassandra/\ncp /tmp/ssl_conf/client/cqlshrc ~/.cassandra/\nsudo mkdir -p /etc/scylla/\nsudo rm -rf /etc/scylla/ssl_conf/\nsudo mv -f /tmp/ssl_conf/ /etc/scylla/\n"'
Exit code: 1
Stdout:
Stderr:
+ '[' -z '' ']'
+ return
+ case $- in
+ return
+ mkdir -p /home/scyllaadm/.cassandra/
mkdir: cannot create directory ‘/home/scyllaadm/.cassandra/’: Permission denied

Installation details
Kernel version: 5.4.0-1035-aws
Scylla version (or git commit hash): 4.6.dev-0.20210408.a8c90a5848
Cluster size: 6 nodes (i3.4xlarge)
Scylla running with shards number (live nodes):
longevity-tls-50gb-3d-master-db-node-670b5bf6-1 (13.48.190.14 | 10.0.1.195): 14 shards
longevity-tls-50gb-3d-master-db-node-670b5bf6-2 (13.48.138.213 | 10.0.0.84): 14 shards
longevity-tls-50gb-3d-master-db-node-670b5bf6-3 (13.53.214.116 | 10.0.3.102): 14 shards
longevity-tls-50gb-3d-master-db-node-670b5bf6-4 (13.49.64.185 | 10.0.2.10): 14 shards
longevity-tls-50gb-3d-master-db-node-670b5bf6-5 (13.51.174.200 | 10.0.2.183): 14 shards
longevity-tls-50gb-3d-master-db-node-670b5bf6-6 (13.51.48.216 | 10.0.1.200): 14 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-06490d953837d4e6b (aws: eu-north-1)

Test: longevity-50gb-3days
Test name: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-50GB-3days-authorization-and-tls-ssl.yaml

Issue description

====================================

PUT ISSUE DESCRIPTION HERE

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor 670b5bf6-21d8-4e8b-81dc-6148975204b6
Show all stored logs command: $ hydra investigate show-logs 670b5bf6-21d8-4e8b-81dc-6148975204b6

Test id: 670b5bf6-21d8-4e8b-81dc-6148975204b6

Logs:
db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/670b5bf6-21d8-4e8b-81dc-6148975204b6/20210409_031349/db-cluster-670b5bf6.zip
loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/670b5bf6-21d8-4e8b-81dc-6148975204b6/20210409_031349/loader-set-670b5bf6.zip
monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/670b5bf6-21d8-4e8b-81dc-6148975204b6/20210409_031349/monitor-set-670b5bf6.zip
sct-runner - https://cloudius-jenkins-test.s3.amazonaws.com/670b5bf6-21d8-4e8b-81dc-6148975204b6/20210409_031349/sct-runner-670b5bf6.zip

Jenkins job URL