ansible scylla node: Deploying a Scylla cluster
This guide will follow the steps required to deploy a Scylla cluster using the ansible-scylla-node role
- An inventory file in a supported format (ini/yaml)
- Ansible 2.8 or higher
- SSH access to all the Scylla nodes configured
The defaults/main.yml file of the role is highly commented. It can be found here:
https://github.com/scylladb/scylla-ansible-roles/blob/master/ansible-scylla-node/defaults/main.yml
Since different operating systems require different dependencies, keys and configuration, the role is designed to be flexible, but might require some careful data gathering before it is used.
To keep things easy, it is recommended to keep the defaults and set only the necessary variables. Open https://scylladb.com and move to the downloads page. There, select your preferred version and platform:
Next, scroll down to get the additional information.
For CentOS all you need is the repo URL:
For Debian you will need additional parameters:
At this point we should have the following parameters:
RHEL/CentOS:
scylla_repos:
- http://repositories.scylladb.com/scylla/repo/UUID/centos/scylladb-4.1.repo
Debian/Ubuntu:
scylla_dependencies:
- software-properties-common
- apt-transport-https
- gnupg2
- dirmngr
scylla_repo_keyfile_urls:
- https://download.opensuse.org/repositories/home:/scylladb:/scylla-3rdparty-stretch/Debian_10.0/Release.key
scylla_repos:
- http://repositories.scylladb.com/scylla/repo/UUID/debian/scylladb-4.1-buster.list
scylla_repo_keyserver: 'hkp://keyserver.ubuntu.com:80'
scylla_repo_keys:
- 5e08fbd8b5d6ec9c
NOTE: Since the requirements for different OS, Scylla distribution and architecture might differ, it is a generally good idea to go through the Scylla downloads page first and populate these variables carefully.
In case Scylla Manager will also be installed in order to configure the cluster, we need to populate some additional variables and update the ones we already have. Open the Manager download page and grab the information it presents:
For CentOS/RHEL all we need is the Manager repo:
scylla_manager_enabled: true
scylla_manager_repo_url: http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo
For Debian/Ubuntu some additional variables need to be set:
scylla_manager_enabled: true
scylla_manager_repo_url: http://downloads.scylladb.com/deb/debian/scylladb-manager-2.1-stretch.list
scylla_manager_repo_keys:
- 6B2BFD3660EF3F5B
- 17723034C56D4B19
Also, if you see additional dependencies that were not yet mentioned in scylla_dependencies above, add those to the list.
Finally, if the Scylla Manager agents will be dealing with backups, you can set scylla_manager_agent_config like this:
scylla_manager_agent_config: |
s3:
access_key_id: 12345678
secret_access_key: QWerty123456789
provider: AWS
region: us-east-1c
endpoint: https://foo.bar.baz
server_side_encryption:
sse_kms_key_id:
upload_concurrency: 2
chunk_size: 50M
use_accelerate_endpoint: false
For full syntax and options, refer to the Scylla Manager documentation.
If the setup will also have Scylla-Monitoring attached, the role can generate and produce a ready to use scylla_servers.yml - the configuration file that will be used later on by the ansible_scylla_monitoring role in order to configure the monitoring stack to monitor the deployed Scylla cluster.
Scylla can be installed online (connect to repos and install from them) or offline to disable any download attempts (some packages, like node_exporter will need to be installed manually)
RAID configuration is flexible with this role. You have several options:
-
Set the
scylla_raid_setupvariable:scylla_raid_setup: - /dev/nvme0n1 - /dev/nvme0n2 - /dev/nvme0n3This setting will attempt to take the 3 disks in the given list and build a RAID-0 array named md0 on top of them, them mount it under
/var/lib/scylla. Ifmd0already exists and contains all 3 disks, the task will skip. If the disks are already formatted or md0 isn't available, the task will fail. -
Use another role or automation to configure RAID. Make sure
scylla_raid_setupis not set. -
Skip RAID configuration altogether (not recommended unless simply testing Scylla outside production). Make sure
scylla_raid_setupis not set.
Since CentOS comes with a rather old kernel, the role has the option to install the current latest kernel available for it on ELRepo. To install an ELRepo kernel set
elrepo_kernel: True
It is possible to install the Scylla binaries but leave the database unconfigured (useful for some specific use cases). To do this set:
install_only: false
The following variables are used to configure the Scylla cluster and nodes.
scylla_cluster_name: MyCluster
Leave this at the default, unless you want all the nodes to have Scylla listening on a specified NIC, i.e. eth0
# default:
scylla_nic: "{{ ansible_default_ipv4.interface }}"
- The version should be either a specific number, e.g. 4.1.2 (for OSS) or 2020.1.2 (for Enterprise) or simply use
latest - The edition can either be
enterpriseoross
scylla_version: latest
scylla_edition: oss
This variable determines the snitch the cluster will be configured to use. The available snitches and their descriptions are available in the Scylla Administration guide.
scylla_snitch: GossipingPropertyFileSnitch
# if GossipingPropertyFileSnitch is used, also set gpfs_prefer_local
gpfs_prefer_local: "true"
Since it is the absolute best practice to run Scylla on homogeneous clusters, it is assumed all the nodes will have the same disk type and count, you have two options:
- Allow the role to measure the node IO throughput on a single node and propagate the gathered benchmark data as a setting to all the nodes (recommended):
scylla_io_probe: True
- Set the IO figures manually:
scylla_io_probe: False
io_properties:
disks:
- mountpoint: /var/lib/scylla/data
read_iops: 10962999
read_bandwidth: 348093056
write_iops: 9722999
write_bandwidth: 352182944
This is an important setting that will populate the cluster's seed list. If running on a multi-DC cluster, it is a good idea to have a few seeds in each cluster. The default setting simply sets the first 3 nodes in the inventory as seeds, which is usually sub-optimal, so this variable is important to set:
scylla_seeds:
- 10.1.0.1
- 10.1.0.2
- 10.1.0.3
- 10.2.0.1
- 10.2.0.2
- 10.2.0.3
These can be altered and reset, but at the default, as shown below, they will use the IPv4 address of the previously set scylla_nic
scylla_listen_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_rpc_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_broadcast_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_api_address: '127.0.0.1'
scylla_api_port: '10000'
If no certificates are present, the role will generate a self-signed CA and node certificates locally and push them out to all the nodes.
WARNING: if a local ./ssl directory is not present and populated by existing certificates, it will be generated and certificates on the nodes will be overwritten with new certificates.
To use existing certificates:
- To use your own CA, place it in
./ssl/ca/SCYLLA_CLUSTER_NAME-ca.pemand derived per-node certificates will be generated from it, if they are not present. - To use your own node certificates, create a directory per hostname (same as in the inventory) in
./ssl/HOSTNAMEand place the .pem file asHOSTNAME.pemand the .crt file asHOSTNAME.crt
The playbook will use the CA certificate to generate a signed host certificate for the .pem file and push them into the nodes. To skip this configuration entirely, set enabled: false for both options or comment out the scylla_ssl variable.
scylla_ssl:
cert_path: /etc/scylla/ # path on the node, where the certificates will be stored
internode:
enabled: true
internode_encryption: all
client:
enabled: false
Set them under the catchall scylla_yaml_params variable as follows:
scylla_yaml_params:
authorizer: CassandraAuthorizer
authenticator: PasswordAuthenticator
enable_sstables_mc_format: true
internode_compression: all
NOTE: In order to avoid outages, changes to scylla.yaml will NOT trigger a service restart. Please use the rolling_restart.yml playbook if cluster-wide configuration changes have occurred.
Scylla does not require swap to be enabled, but it is highly recommended to have some swap allocated. The role does not configure swap, just like it doesn't require for RAID settings to be in place. It is recommended to use another role to configure swap, or provision the target nodes with a swap partition in place.
Here's an example settings file for a cluster:
- Cluster name "MyCluster"
- Enterprise 2020.1 latest
- Internode SSL enabled, Client SSL disabled
- etc...
# parameters.yml
# swap configuration for the swap role
swap_file_size_mb: 2048
#RAID
scylla_raid_setup:
- /dev/nvme0n1
- /dev/nvme0n2
- /dev/nvme0n3
# Scylla Manager agent
scylla_manager_agent_config: |
gcs:
service_account_file: /etc/scylla/gcs.json
scylla_manager_repo_url: http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo
scylla_manager_enabled: true
# monitoring configuration
generate_monitoring_config: true
# Scylla configuration
scylla_api_port: 10000
scylla_api_address: 127.0.0.1
scylla_broadcast_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_rpc_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_listen_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_seeds:
- 10.142.0.3
- 10.142.0.4
- 10.128.0.3
- 10.128.0.4
enable_mc_format: true
scylla_io_probe: true
scylla_snitch: GossipingPropertyFileSnitch
scylla_edition: enterprise
scylla_version: latest
scylla_dependencies:
- curl
- wget
elrepo_kernel: true
scylla_repos:
- http://downloads.scylladb.com.s3.amazonaws.com/rpm/centos/scylla-2020.1.repo
install_type: online
scylla_nic: "{{ ansible_default_ipv4.interface }}"
install_only: false
scylla_cluster_name: MyCluster
scylla_ssl:
cert_path: /etc/scylla/
internode:
enabled: true
internode_encryption: all
client:
enabled: false
scylla_yaml_params:
authorizer: CassandraAuthorizer
authenticator: PasswordAuthenticator
enable_sstables_mc_format: true
internode_compression: all
Some settings are relevant on a per-node basis, and cannot be worked around using ansible's magic variables.
- For
GossipingPropertyFileSnitcheach node has to be configured with a specificdcandrack. - For Cloud-aware snitches (EC2Snitch, Ec2MultiRegionSnitch, GoogleCloudSnitch)
dc_suffixneeds to be set.
Here is our example inventory, using the GPFS snitch across two DCs, note how each DC has 2 seeds in the scylla_seeds variable above.
# inventory.ini
[scylla]
10.142.0.3 dc=us-east1 rack=rack-b
10.142.0.4 dc=us-east1 rack=rack-b
10.142.0.5 dc=us-east1 rack=rack-b
10.128.0.2 dc=us-central1 rack=rack-a
10.128.0.3 dc=us-central1 rack=rack-a
10.128.0.4 dc=us-central1 rack=rack-a
Here is an example playbook using the role:
# scylla_nodes.yml
- name: Scylla node
hosts: scylla
any_errors_fatal: true
tasks:
# Workaround for some Debian versions that might not have XFS support out of the box
- name: install XFSprogs
package:
name: xfsprogs
state: present
become: true
roles:
# This is a nice working role for setting up swap
- { name: geerlingguy.swap, become: true }
- ansible-scylla-node
And the playbook execution:
ansible-playbook -i inventory.ini -e '@parameters.yml' scylla_nodes.yml
When the playbook is done running, depending on the settings provided, you will end up with some produced files:
-
scyllamgr_auth_token.txt: this is the token configured in all the scylla-manager-agent configuration files across the cluster. Use it with theansible-scylla-managerrole to connect the Manager to the Scylla cluster. -
scylla_servers.yml: this file will be useful for setting up Scylla Monitoring, use it with theansible-scylla-monitoringrole in order to configure monitoring for the cluster. -
ssl/caandssl/HOSTdirectories and their contents: These are the certificates generated and deployed in case SSL was enabled with theansible-scylla-noderole. Please store these safely
The role has the facility to adjust sysctl settings, that might be required by the Scylla team or by your cloud platform support for optimal performance. The defaults/main.yml settings contain a set of recommended sysctl settings under the sysctl variable as well as a flag determining whether or not sysctl changes will be applied at all called update_sysctl which has to be set to True for changes to take place.
When using the ELRepo kernel, it will get installed but not activated until the next reboot. In order to switch to the new kernel, please make sure you restart the nodes after applying the role. If done in an active cluster, perform a rolling restart either manually or using the rolling restart playbook.