Cannot provision resources with multiple az's

[soyacz]

When trying to run perf test with multiple az's, provision step failed this way:

/home/ubuntu/sct-results/20240514-092120-239404
11:21:20  Symlink `/home/ubuntu/sct-results/latest' updated to `/home/ubuntu/sct-results/20240514-092120-239404'
11:21:26  syslog-ng listen on port 32768 (config: /tmp/syslog-ngtbcfvx_l.conf)
11:21:26  Provision aws cloud resources
11:21:26  Capacity reservation is not enabled. Skipping reservation.
11:21:26  [eu-west-1] Creating {count} on-demand instances using AMI id 'ami-0ad4bafa74681b4ee' with following parameters:
11:21:26  {'ImageId': 'ami-0ad4bafa74681b4ee', 'KeyName': 'scylla-qa-ec2', 'InstanceType': 'i3en.2xlarge', 'UserData': 'Content-Type: multipart/mixed; boundary="===============2835613571025166995=="\nMIME-Version: 1.0\n\n--===============2835613571025166995==\nContent-Type: x-scylla/json\nMIME-Version: 1.0\nContent-Disposition: attachment; filename="scylla_machine_image.json"\n\n{\n    "cluster_name": "perf-latency-grow-shrink-ubuntu-db-cluster-89bc68f9",\n    "data_device": "instance_store",\n    "raid_level": 0,\n    "scylla_yaml": {\n        "cluster_name": "perf-latency-grow-shrink-ubuntu-db-cluster-89bc68f9"\n    },\n    "start_scylla_on_first_boot": false\n}\n--===============2835613571025166995==\nContent-Type: text/cloud-config\nMIME-Version: 1.0\nContent-Disposition: attachment; filename="cloud-config.txt"\n\n\n        #cloud-config\n        cloud_final_modules:\n        - [scripts-user, always]\n        \n--===============2835613571025166995==\nContent-Type: text/x-shellscript\nMIME-Version: 1.0\nContent-Disposition: attachment; filename="user-script.txt"\n\n#!/bin/bash\nset -x\nwhile ! systemctl status cloud-init.service | grep "active (exited)"; do sleep 1; done\nif [ -f /var/lib/sct/cloud-init/done ]; then sudo systemctl restart syslog-ng; exit 0; fi\nsystemctl stop sshd || true\nSYSLOG_NG_INSTALLED=""\nif yum --help 2>/dev/null 1>&2 ; then\n    if rpm -q syslog-ng ; then\n        rm /etc/syslog-ng/syslog-ng.conf  # Make sure we have default syslog-ng.conf\n        yum reinstall -y syslog-ng\n        SYSLOG_NG_INSTALLED=1\n    else\n        yum install -y epel-release\n        for n in 1 2 3 4 5 6 7 8 9; do # cloud-init is running it with set +o braceexpand\n            if yum install -y --downloadonly syslog-ng; then\n                break\n            fi\n        done\n\n        for n in 1 2 3; do # cloud-init is running it with set +o braceexpand\n            if yum install -y syslog-ng; then\n                SYSLOG_NG_INSTALLED=1\n                break\n            fi\n            sleep 10\n        done\n    fi\nelif apt-get --help 2>/dev/null 1>&2 ; then\n    if dpkg-query --show syslog-ng ; then\n        rm /etc/syslog-ng/syslog-ng.conf  # Make sure we have default syslog-ng.conf\n        apt-get purge -y syslog-ng*\n        DPKG_FORCE=confmiss apt-get --reinstall -y install syslog-ng\n        SYSLOG_NG_INSTALLED=1\n    else\n        cat /etc/apt/sources.list\n        for n in 1 2 3 4 5 6 7 8 9; do # cloud-init is running it with set +o braceexpand\n            if apt-get -y update 2>&1 | tee /tmp/syslog_ng_install.output || grep NO_PUBKEY         /tmp/syslog_ng_install.output; then\n                break\n            fi\n        done\n\n        for n in 1 2 3; do # cloud-init is running it with set +o braceexpand\n            while ! find /proc/*/fd -lname /var/lib/dpkg/lock-frontend -exec false {} + -quit ; do\n                sleep 1\n            done\n            apt-get install -y syslog-ng || true\n            if dpkg-query --show syslog-ng ; then\n                SYSLOG_NG_INSTALLED=1\n                break\n            fi\n        done\n    fi\nelse\n    echo "Unsupported distro"\nfi\n\nsource_name=`cat /etc/syslog-ng/syslog-ng.conf | tr -d "\\n" | tr -d "\\r" | sed -r "s/\\};/\\};\\n/g;         s/source /\\nsource /g" | grep -P "^source.*system\\(\\)" | cut -d" " -f2`\ndisk_buffer_option=""\nif syslog-ng -V | grep disk; then\n    disk_buffer_option="disk-buffer(\n            mem-buf-size(1048576)\n            disk-buf-size(104857600)\n            reliable(yes)\n            dir(\\"/var/log\\")\n        )"\nfi\n\nif grep -P "keep-timestamp\\([^)]+\\)" /etc/syslog-ng/syslog-ng.conf; then\n    sed -i -r "s/keep-timestamp([ ]*yes[ ]*)/keep-timestamp(no)/g" /etc/syslog-ng/syslog-ng.conf\nelse\n    sed -i -r "s/([ \t]*options[ \t]*\\\\{)/\\\\1\\n  keep-timestamp(no);\\n/g" /etc/syslog-ng/syslog-ng.conf\nfi\n\nif ! grep "destination remote_sct" /etc/syslog-ng/syslog-ng.conf; then\n    cat <<EOF >>/etc/syslog-ng/syslog-ng.conf\ndestination remote_sct {\n    syslog(\n        "10.4.0.204"\n        transport("tcp")\n        port(32768)\n        throttle(10000)\n        $disk_buffer_option\n    );\n};\n\nEOF\nfi\n\nif ! grep -P "log {.*destination\\\\(remote_sct\\\\)" /etc/syslog-ng/syslog-ng.conf; then\n    echo "log { source($source_name); destination(remote_sct); };" >> /etc/syslog-ng/syslog-ng.conf\nfi\n\nif [ ! -z "" ]; then\n    if grep "rewrite r_host" /etc/syslog-ng/syslog-ng.conf; then\n        sed -i -r "s/rewrite r_host \\{ set\\(\\"[^\\"]+\\"/rewrite r_host { set(\\"\\"/" /etc/syslog-ng/syslog-ng.conf\n    else\n        echo "rewrite r_host { set(\\"\\", value(\\"HOST\\")); };" >>  /etc/syslog-ng/syslog-ng.conf\n        sed -i -r "s/destination\\(remote_sct\\);[ \\t]*\\};/destination\\(remote_sct\\); rewrite\\(r_host\\); \\};/" /etc/syslog-ng/syslog-ng.conf\n    fi\nfi\nsystemctl restart syslog-ng  || true\n\nif [ -f "/etc/security/limits.d/20-nproc.conf" ]; then\n    sed -i -e "s/^\\*[[:blank:]]*soft[[:blank:]]*nproc[[:blank:]]*.*/*\t\tsoft\tnproc\t\tunlimited/"     /etc/security/limits.d/20-nproc.conf || true\nelse\n    echo "*    hard    nproc    unlimited" > /etc/security/limits.d/20-nproc.conf || true\nfi\n\nsed -i "s/#MaxSessions \\(.*\\)$/MaxSessions 1000/" /etc/ssh/sshd_config || true\nsed -i "s/#MaxStartups \\(.*\\)$/MaxStartups 60/" /etc/ssh/sshd_config || true\nsed -i "s/#LoginGraceTime \\(.*\\)$/LoginGraceTime 15s/" /etc/ssh/sshd_config || true\nsed -i "s/#ClientAliveInterval \\(.*\\)$/ClientAliveInterval 60/" /etc/ssh/sshd_config || true\nsed -i "s/#ClientAliveCountMax \\(.*\\)$/ClientAliveCountMax 10/" /etc/ssh/sshd_config || true\n\nif (( $(ssh -V 2>&1 | tr -d "[:alpha:][:blank:][:punct:]" | cut -c-2) >= 88 )); then\n    systemctl stop sshd || true\n    sed -i "s/#PubkeyAuthentication \\(.*\\)$/PubkeyAuthentication yes/" /etc/ssh/sshd_config || true\n    sed -i -e "\\$aPubkeyAcceptedAlgorithms +ssh-rsa" /etc/ssh/sshd_config || true\n    sed -i -e "\\$aHostKeyAlgorithms +ssh-rsa" /etc/ssh/sshd_config || true\n    systemctl restart sshd || true\nfi\nsystemctl restart sshd || true\nsystemctl start sshd || true\nmkdir -p /var/lib/sct/cloud-init && touch /var/lib/sct/cloud-init/done\n--===============2835613571025166995==--\n', 'NetworkInterfaces': [{'DeviceIndex': 0, 'SubnetId': 'subnet-09ee8fe63f544306b', 'Groups': ['sg-0691c61126d99dd41']}], 'IamInstanceProfile': {'Name': 'qa-scylla-manager-backup-instance-profile'}, 'BlockDeviceMappings': [{'DeviceName': '/dev/sda1', 'Ebs': {'VolumeType': 'gp3', 'VolumeSize': 30}}], 'Placement': {'AvailabilityZone': 'eu-west-1a'}}
11:21:27  Created instances: [ec2.Instance(id='i-03850dd5fe8bbed1b')].
11:21:27  ProvisionPlan: Instances has been provisioned using "On Demand":
11:21:27  [ec2.Instance(id='i-03850dd5fe8bbed1b')]
11:21:28  Traceback (most recent call last):
11:21:28    File "/home/ubuntu/scylla-cluster-tests/./sct.py", line 1798, in <module>
11:21:28      cli.main(prog_name="hydra")
11:21:28    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
11:21:28      rv = self.invoke(ctx)
11:21:28    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
11:21:28      return _process_result(sub_ctx.command.invoke(sub_ctx))
11:21:28    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
11:21:28      return ctx.invoke(self.callback, **ctx.params)
11:21:28    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
11:21:28      return __callback(*args, **kwargs)
11:21:28    File "/home/ubuntu/scylla-cluster-tests/./sct.py", line 229, in provision_resources
11:21:28      layout.provision()
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_provision/aws/layout.py", line 34, in provision
11:21:28      self.db_cluster.provision()
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_provision/aws/cluster.py", line 228, in provision
11:21:28      instance_parameters = self._instance_parameters(region_id=region_id, availability_zone=az_id)
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_provision/aws/cluster.py", line 216, in _instance_parameters
11:21:28      return AWSInstanceParams(**params_builder.dict(exclude_none=True, exclude_unset=True, exclude_defaults=True))
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/provision/common/builders.py", line 69, in dict
11:21:28      prop_value = getattr(self, prop)
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_provision/aws/instance_parameters_builder.py", line 75, in NetworkInterfaces
11:21:28      output.append({'DeviceIndex': index, **self._network_interface_params(interface_index=index)})
11:21:28    File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_provision/aws/instance_parameters_builder.py", line 135, in _network_interface_params
11:21:28      'SubnetId': self._ec2_subnet_ids[self.region_id][self.availability_zone + interface_index],  # pylint: disable=invalid-sequence-index
11:21:28  IndexError: list index out of range
11:21:29  Cleaning SSH agent
11:21:29  Agent pid 3506 killed

from sct log:

< t:2024-05-14 09:21:54,726 f:aws_region.py   l:117  c:sdcm.utils.aws_region p:DEBUG > Found Subnets: {'Subnets': [{'AvailabilityZone': 'eu-west-1a', 'AvailabilityZoneId': 'euw1-az3', 'AvailableIpAddressCount': 983, 'CidrBlock': '10.4.0.0/22', 'DefaultForAz': False, 'MapPublicIpOnLaunch': True, 'MapCustomerOwnedIpOnLaunch': False, 'State': 'available', 'SubnetId': 'subnet-09ee8fe63f544306b', 'VpcId': 'vpc-05caf4736ea5af013', 'OwnerId': '797456418907', 'AssignIpv6AddressOnCreation': True, 'Ipv6CidrBlockAssociationSet': [{'AssociationId': 'subnet-cidr-assoc-074070ae2fdad0856', 'Ipv6CidrBlock': '2a05:d018:12e3:f000::/64', 'Ipv6CidrBlockState': {'State': 'associated'}}], 'Tags': [{'Key': 'Name', 'Value': 'SCT-2-subnet-eu-west-1a'}], 'SubnetArn': 'arn:aws:ec2:eu-west-1:797456418907:subnet/subnet-09ee8fe63f544306b', 'EnableDns64': False, 'Ipv6Native': False, 'PrivateDnsNameOptionsOnLaunch': {'HostnameType': 'ip-name', 'EnableResourceNameDnsARecord': False, 'EnableResourceNameDnsAAAARecord': False}}], 'ResponseMetadata': {'RequestId': 'e66b9b62-4465-46cf-862a-d1eb59bcef4b', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'e66b9b62-4465-46cf-862a-d1eb59bcef4b', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'accept-encoding', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '2067', 'date': 'Tue, 14 May 2024 09:21:53 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}
< t:2024-05-14 09:21:54,777 f:aws_region.py   l:117  c:sdcm.utils.aws_region p:DEBUG > Found Subnets: {'Subnets': [{'AvailabilityZone': 'eu-west-1b', 'AvailabilityZoneId': 'euw1-az1', 'AvailableIpAddressCount': 1011, 'CidrBlock': '10.4.4.0/22', 'DefaultForAz': False, 'MapPublicIpOnLaunch': True, 'MapCustomerOwnedIpOnLaunch': False, 'State': 'available', 'SubnetId': 'subnet-08dcdde2fdc3b33b1', 'VpcId': 'vpc-05caf4736ea5af013', 'OwnerId': '797456418907', 'AssignIpv6AddressOnCreation': True, 'Ipv6CidrBlockAssociationSet': [{'AssociationId': 'subnet-cidr-assoc-07e9041da0f9b1a22', 'Ipv6CidrBlock': '2a05:d018:12e3:f001::/64', 'Ipv6CidrBlockState': {'State': 'associated'}}], 'Tags': [{'Key': 'Name', 'Value': 'SCT-2-subnet-eu-west-1b'}], 'SubnetArn': 'arn:aws:ec2:eu-west-1:797456418907:subnet/subnet-08dcdde2fdc3b33b1', 'EnableDns64': False, 'Ipv6Native': False, 'PrivateDnsNameOptionsOnLaunch': {'HostnameType': 'ip-name', 'EnableResourceNameDnsARecord': False, 'EnableResourceNameDnsAAAARecord': False}}], 'ResponseMetadata': {'RequestId': 'f0872eac-5729-4c59-a691-bdfffd8b16b7', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'f0872eac-5729-4c59-a691-bdfffd8b16b7', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'accept-encoding', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '2068', 'date': 'Tue, 14 May 2024 09:21:53 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}
< t:2024-05-14 09:21:54,824 f:aws_region.py   l:117  c:sdcm.utils.aws_region p:DEBUG > Found Subnets: {'Subnets': [{'AvailabilityZone': 'eu-west-1c', 'AvailabilityZoneId': 'euw1-az2', 'AvailableIpAddressCount': 977, 'CidrBlock': '10.4.8.0/22', 'DefaultForAz': False, 'MapPublicIpOnLaunch': True, 'MapCustomerOwnedIpOnLaunch': False, 'State': 'available', 'SubnetId': 'subnet-0890dadff64302b36', 'VpcId': 'vpc-05caf4736ea5af013', 'OwnerId': '797456418907', 'AssignIpv6AddressOnCreation': True, 'Ipv6CidrBlockAssociationSet': [{'AssociationId': 'subnet-cidr-assoc-02d815d9449af314e', 'Ipv6CidrBlock': '2a05:d018:12e3:f002::/64', 'Ipv6CidrBlockState': {'State': 'associated'}}], 'Tags': [{'Key': 'Name', 'Value': 'SCT-2-subnet-eu-west-1c'}], 'SubnetArn': 'arn:aws:ec2:eu-west-1:797456418907:subnet/subnet-0890dadff64302b36', 'EnableDns64': False, 'Ipv6Native': False, 'PrivateDnsNameOptionsOnLaunch': {'HostnameType': 'ip-name', 'EnableResourceNameDnsARecord': False, 'EnableResourceNameDnsAAAARecord': False}}], 'ResponseMetadata': {'RequestId': '7cd63515-85f0-4518-bf7f-1f12e67d1dcb', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '7cd63515-85f0-4518-bf7f-1f12e67d1dcb', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'accept-encoding', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '2067', 'date': 'Tue, 14 May 2024 09:21:53 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}

sct.log.tar.gz

[enaydanov]

We got same problem in 6.0 test runs:

https://argus.scylladb.com/test/2ef1e453-1598-451e-ad1d-508661a3b92d/runs?additionalRuns[]=fb63fc2d-1dc5-485f-8ef3-510a76d43960 :

< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR > Traceback (most recent call last):
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 177, in wrapper
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     return method(*args, **kwargs)
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 119, in inner
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     res = func(*args, **kwargs)
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 883, in setUp
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     self.init_resources()
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1824, in init_resources
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     self.get_cluster_aws(loader_info=loader_info, db_info=db_info,
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1413, in get_cluster_aws
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     self.db_cluster = create_cluster(db_type)
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1376, in create_cluster
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     return ScyllaAWSCluster(
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 854, in __init__
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     super().__init__(
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3922, in __init__
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     super().__init__(*args, **kwargs)
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 109, in __init__
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     super().__init__(cluster_uuid=cluster_uuid,
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3212, in __init__
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     self.add_nodes(nodes_per_az[az_index], dc_idx=dc_idx, rack=rack,
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 892, in add_nodes
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     added_nodes = super().add_nodes(
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 387, in add_nodes
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     instances = self._create_or_find_instances(
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 376, in _create_or_find_instances
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     return self._create_instances(count, ec2_user_data, dc_idx, az_idx, instance_type=instance_type)
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster_aws.py", line 214, in _create_instances
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR >     'SubnetId': self._ec2_subnet_id[dc_idx][az_idx + i],
< t:2024-05-30 11:40:19,717 f:tester.py       l:184  c:sdcm.tester          p:ERROR > IndexError: list index out of range

Please, raise the priority for this.

[fruch]

@juliayakovlev can you take a look at this, seem related to the networking refactor somehow.