498 changes: 498 additions & 0 deletions docs/faq.rst
View file

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions docs/getting-started/_common/note-io.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
.. note::

It's important to keep I/O scheduler configuration in sync on nodes with the same hardware.
That's why we recommend skipping running scylla_io_setup when provisioning a new node with exactly the same hardware setup as existing nodes in the cluster.

Instead, we recommend to copy the following files from an existing node to the new node after running scylla_setup and restart scylla-server service (if it is already running):
* /etc/scylla.d/io.conf
* /etc/scylla.d/io_properties.yaml

Using different I/O scheduler configuration may result in unnecessary bottlenecks.
2 changes: 2 additions & 0 deletions docs/getting-started/_common/note-reclaim-space.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.. note:: By default, when a table or a keyspace is removed, a snapshot is taken so that you can restore it later. As a result, the disk space remains the same and is not immediately reclaimed.
Refer to :doc:`this article </troubleshooting/drop-table-space-up/>` or this :ref:`FAQ entry<reclaim-space>`.
206 changes: 206 additions & 0 deletions docs/getting-started/_common/system-configuration-index.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
Configure Scylla
================

System configuration steps are performed automatically by the Scylla RPM and deb packages. For information on getting started with Scylla, see :doc:`Getting Started </getting-started/index>`.

All Scylla AMIs and Docker images are pre-configured by a script with the following steps. This document is provided as a reference.

.. _system-configuration-files-and-scripts:

System Configuration Files and Scripts
--------------------------------------
Several system configuration settings should be applied. For ease of use, the necessary scripts and configuration files are provided. Files are under :code:`dist/common` and :code:`seastar/scripts` in the Scylla source code and installed in the appropriate system locations. (For information on Scylla’s own configuration file, see Scylla Configuration.)

.. list-table:: System Configuration Files
:widths: 50 50
:header-rows: 1

* - File Name
- Description
* - scylla.conf
- Remove system resource limits
* - scylla-server
- Server startup options
* - (written by ``scylla_coredump_setup``, below)
- Configure core dumps to use the ``scylla_save_coredump`` script

Scylla Scripts
--------------

The following scripts are available for you to run for configuring Scylla. Some of these scripts are included in the scylla_setup script. This script is used for configuring Scylla the first time, or when the system hardware changes.


.. list-table:: Scylla Setup Scripts
:widths: 40 60
:header-rows: 1

* - perftune.py
- Configures various system parameters in order to improve the Seastar application performance
* - scylla_bootparam_setup
- Sets the kernel options in the bootloader. In addition, it tunes Linux boot-time parameters for the node that Scylla is running on (e.g. huge page setup).
* - scylla_coredump_setup
- Sets up coredump facilities for Scylla. This may include uninstalling existing crash reporting software for compatibility reasons.
* - scylla_io_setup
- Benchmarks the disks and generates the io.conf and io_properties.yaml files.
* - scylla_ntp_setup
- Configures Network Time Protocol
* - scylla_prepare
- This script is run automatically every time Scylla starts and the machine needs to be tuned.
* - scylla_raid_setup
- Configures RAID and makes an XFS filesystem.
* - scylla_save_coredump
- Compresses a core dump file (Ubuntu only)
* - scylla_setup
- Sets up the Scylla configuration. Many of these scripts are included in the setup script.
* - scylla_stop
- Resets network mode if running in virtio or DPDK mode.
* - scylla_swap_setup
- Configures a swap space on the host.
* - scylla_sysconfig_setup
- Rewrites the /etc/sysconfig/scylla file.


.. list-table:: Scylla Scripts (Not included with Scylla-Setup)
:widths: 40 60
:header-rows: 1

* - Script Name
- Description
* - node_health_check
- Gathers metrics and information on the node, checking that the node is configured correctly.
* - scylla-blocktune
- Tunes the filesystem and block layer (e.g. block size I/O scheduler configuration) for Scylla.
* - scylla_cpuscaling_setup
- Configures the CPU frequency scaling (IOW, puts the CPU in "performance" mode, instead of the slower "powersave" mode).
* - scylla_cpuset_setup
- Configures which CPUs the Scylla server threads run on.
* - scylla_fstrim
- Runs ``fstrim``, which cleans up unused blocks of data from your SSD storage device. It runs automatically if you run scylla_fstrim_set up (see below).
* - scylla_fstrim_setup
- Configures a job so that ``fstrim`` runs automatically.
* - scylla-housekeeping
- Checks if there are new versions of Scylla available, and also shares some telemetry information for us to keep track of what versions are installed on the field.
* - scylla_rsyslog_setup
- Configures the "rsyslog" service, which is used to send logs to a remote server.
* - scylla_selinux_setup
- Disables SELinux for Scylla.

.. _note-io:

.. include:: /getting-started/_common/note-io.rst

Bootloader Settings
-------------------
If Scylla is installed on an Amazon AMI, the bootloader should provide the :code:`clocksource=tsc` and :code:`tsc=reliable` options. This enables an accurate, high-resolution `Time Stamp Counter (TSC) <https://software.intel.com/en-us/blogs/2013/06/20/eliminate-the-dreaded-clocksource-is-unstable-message-switch-to-tsc-for-a-stable>`_ for setting the system time.

This configuration is provided in the file :code:`/usr/lib/scylla/scylla_bootparam_setup`.

Remove Crash Reporting Software
-------------------------------
Remove the :code:`apport-noui` or :code:`abrt` packages if present, and set up a location and file name pattern for core dumps.

This configuration is provided in the file :code:`/usr/lib/scylla/scylla_bootparam_setup`.

Set Up Network Time Synchronization
-----------------------------------
It is highly recommended to enforce time synchronization between Scylla servers.

Run :code:`ntpstat` on all nodes to check that system time is synchronized. If you are running in a virtualized environment and your system time is set on the host, you may not need to run NTP on the guest. Check the documentation for your platform.

If you have your own time servers shared with an application using Scylla, use the same NTP configuration as for your application servers. The script :code:`/usr/lib/scylla/scylla_ntp_setup` provides sensible defaults, using Amazon NTP servers if installed on the Amazon cloud, and other pool NTP servers otherwise.

Set Up RAID and Filesystem
--------------------------
Setting the file system to XFS is the most important and mandatory for production. Scylla will significantly slow down without it.

The script :code:`/usr/lib/scylla/scylla_raid_setup` performs the necessary RAID configuration and XFS filesystem creation for Scylla.

Arguments to the script are

* :code:`-d` specify disks for RAID
* :code:`-r` MD device name for RAID
* :code:`-u` update /etc/fstab for RAID

On the Scylla AMI, the RAID configuration is handled automatically in the :code:`/usr/lib/scylla/scylla_prepare script`.

CPU Pinning
-----------

When installing Scylla, it is highly recommended to use the :doc:`scylla_setup </getting-started/system-configuration>` script.
Scylla should not share CPUs with any CPU consuming process. In addition, when running Scylla on AWS, we recommend pinning all NIC IRQs to CPU0 (due to the same reason). As a result, Scylla should be prevented from running on CPU0 and its hyper-threading siblings. To verify that Scylla is pinning CPU0, use the command below:
If the node has four or fewer CPUs, don't use this option.

To verify:

.. code-block:: shell
cat /etc/scylla.d/cpuset.conf
Example output:

.. code-block:: shell
--cpuset `1-15,17-31`
Networking
----------
On AWS:
^^^^^^^
1. Prevent irqbalance from moving your NICs’ IRQs.
2. Bind all NICs’ HW queues to CPU0:

.. code-block:: shell
for irq in `cat /proc/interrupts | grep <networking iface name> | cut -d":" -f1`
do echo "Binding IRQ $irq to CPU0" echo 1 > /proc/irq/$irq/smp_affinity done
3. Enable RPS and bind RPS queues to CPUs other than CPU0 and its hyper-threading siblings.
4. Enable XPS and distribute all XPS queues among all available CPUs.

The `posix_net_conf.sh <https://github.com/scylladb/seastar/blob/master/scripts/posix_net_conf.sh>`_ script does all of the above.*

On Bare Metal Setups with Multi-Queue NICs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Prevent irqbalance from moving your NICs IRQs.
2. Bind each NIC’s IRQ to a separate CPU.
3. Enable XPS exactly the same way as for AWS above.
4. Set higher values for a listen() socket backlog and for unacknowledged pending connections backlog:

.. code-block:: shell
echo 4096 > /proc/sys/net/core/somaxconn
echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog
The `posix_net_conf.sh <https://github.com/scylladb/seastar/blob/master/scripts/posix_net_conf.sh>`_ script with the :code:`-mq` parameter does all of the above.

Configuring Scylla
------------------
Configuration for Scylla itself is in the :ref:`Scylla Configuration <admin-scylla-configuration>` section of the administration guide.

Development System Configuration
--------------------------------
*The following item is not required in production.*

When working on DPDK support for Scylla, enable hugepages.

.. code-block:: shell
NR_HUGEPAGES=128
mount -t hugetlbfs -o pagesize=2097152 none /mnt/huge
mount -t hugetlbfs -o pagesize=2097152 none /dev/hugepages/
for n in /sys/devices/system/node/node?; do
echo $NR_HUGEPAGES > $n/hugepages/hugepages-2048kB/nr_hugepages;
done
Huge page configuration is written to :code:`/etc/sysconfig/scylla-server` by the script :code:`/usr/lib/scylla/sysconfig_setup`




Related Topics
--------------

:doc:`System Limits </kb/system-limits>` - outlines the system limits which should be set or removed.

.. include:: /rst_include/advance-index.rst
317 changes: 317 additions & 0 deletions docs/getting-started/appendices.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,317 @@


.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. highlight:: cql

Appendices
----------

.. include:: /rst_include/cql-version-index.rst

.. _appendix-A:

Appendix A: CQL Keywords
~~~~~~~~~~~~~~~~~~~~~~~~

CQL distinguishes between *reserved* and *non-reserved* keywords.
Reserved keywords cannot be used as an identifier. They are truly reserved
for the language (but one can enclose a reserved keyword by
double-quotes to use it as an identifier). Non-reserved keywords, however,
only have a specific meaning in a certain context but can be used as
identifiers otherwise. The only *raison d’être* of these non-reserved
keywords is convenience: some keyword are non-reserved when it was
always easy for the parser to decide whether they were used as keywords
or not.

+--------------------+-------------+
| Keyword | Reserved? |
+====================+=============+
| ``ADD`` | yes |
+--------------------+-------------+
| ``AGGREGATE`` | no |
+--------------------+-------------+
| ``ALL`` | no |
+--------------------+-------------+
| ``ALLOW`` | yes |
+--------------------+-------------+
| ``ALTER`` | yes |
+--------------------+-------------+
| ``AND`` | yes |
+--------------------+-------------+
| ``APPLY`` | yes |
+--------------------+-------------+
| ``AS`` | no |
+--------------------+-------------+
| ``ASC`` | yes |
+--------------------+-------------+
| ``ASCII`` | no |
+--------------------+-------------+
| ``AUTHORIZE`` | yes |
+--------------------+-------------+
| ``BATCH`` | yes |
+--------------------+-------------+
| ``BEGIN`` | yes |
+--------------------+-------------+
| ``BIGINT`` | no |
+--------------------+-------------+
| ``BLOB`` | no |
+--------------------+-------------+
| ``BOOLEAN`` | no |
+--------------------+-------------+
| ``BY`` | yes |
+--------------------+-------------+
| ``CALLED`` | no |
+--------------------+-------------+
| ``CLUSTERING`` | no |
+--------------------+-------------+
| ``COLUMNFAMILY`` | yes |
+--------------------+-------------+
| ``COMPACT`` | no |
+--------------------+-------------+
| ``CONTAINS`` | no |
+--------------------+-------------+
| ``COUNT`` | no |
+--------------------+-------------+
| ``COUNTER`` | no |
+--------------------+-------------+
| ``CREATE`` | yes |
+--------------------+-------------+
| ``CUSTOM`` | no |
+--------------------+-------------+
| ``DATE`` | no |
+--------------------+-------------+
| ``DECIMAL`` | no |
+--------------------+-------------+
| ``DELETE`` | yes |
+--------------------+-------------+
| ``DESC`` | yes |
+--------------------+-------------+
| ``DESCRIBE`` | yes |
+--------------------+-------------+
| ``DISTINCT`` | no |
+--------------------+-------------+
| ``DOUBLE`` | no |
+--------------------+-------------+
| ``DROP`` | yes |
+--------------------+-------------+
| ``ENTRIES`` | yes |
+--------------------+-------------+
| ``EXECUTE`` | yes |
+--------------------+-------------+
| ``EXISTS`` | no |
+--------------------+-------------+
| ``FILTERING`` | no |
+--------------------+-------------+
| ``FINALFUNC`` | no |
+--------------------+-------------+
| ``FLOAT`` | no |
+--------------------+-------------+
| ``FROM`` | yes |
+--------------------+-------------+
| ``FROZEN`` | no |
+--------------------+-------------+
| ``FULL`` | yes |
+--------------------+-------------+
| ``FUNCTION`` | no |
+--------------------+-------------+
| ``FUNCTIONS`` | no |
+--------------------+-------------+
| ``GRANT`` | yes |
+--------------------+-------------+
| ``IF`` | yes |
+--------------------+-------------+
| ``IN`` | yes |
+--------------------+-------------+
| ``INDEX`` | yes |
+--------------------+-------------+
| ``INET`` | no |
+--------------------+-------------+
| ``INFINITY`` | yes |
+--------------------+-------------+
| ``INITCOND`` | no |
+--------------------+-------------+
| ``INPUT`` | no |
+--------------------+-------------+
| ``INSERT`` | yes |
+--------------------+-------------+
| ``INT`` | no |
+--------------------+-------------+
| ``INTO`` | yes |
+--------------------+-------------+
| ``JSON`` | no |
+--------------------+-------------+
| ``KEY`` | no |
+--------------------+-------------+
| ``KEYS`` | no |
+--------------------+-------------+
| ``KEYSPACE`` | yes |
+--------------------+-------------+
| ``KEYSPACES`` | no |
+--------------------+-------------+
| ``LANGUAGE`` | no |
+--------------------+-------------+
| ``LIMIT`` | yes |
+--------------------+-------------+
| ``LIST`` | no |
+--------------------+-------------+
| ``LOGIN`` | no |
+--------------------+-------------+
| ``MAP`` | no |
+--------------------+-------------+
| ``MODIFY`` | yes |
+--------------------+-------------+
| ``NAN`` | yes |
+--------------------+-------------+
| ``NOLOGIN`` | no |
+--------------------+-------------+
| ``NORECURSIVE`` | yes |
+--------------------+-------------+
| ``NOSUPERUSER`` | no |
+--------------------+-------------+
| ``NOT`` | yes |
+--------------------+-------------+
| ``NULL`` | yes |
+--------------------+-------------+
| ``OF`` | yes |
+--------------------+-------------+
| ``ON`` | yes |
+--------------------+-------------+
| ``OPTIONS`` | no |
+--------------------+-------------+
| ``OR`` | yes |
+--------------------+-------------+
| ``ORDER`` | yes |
+--------------------+-------------+
| ``PASSWORD`` | no |
+--------------------+-------------+
| ``PERMISSION`` | no |
+--------------------+-------------+
| ``PERMISSIONS`` | no |
+--------------------+-------------+
| ``PRIMARY`` | yes |
+--------------------+-------------+
| ``RENAME`` | yes |
+--------------------+-------------+
| ``REPLACE`` | yes |
+--------------------+-------------+
| ``RETURNS`` | no |
+--------------------+-------------+
| ``REVOKE`` | yes |
+--------------------+-------------+
| ``ROLE`` | no |
+--------------------+-------------+
| ``ROLES`` | no |
+--------------------+-------------+
| ``SCHEMA`` | yes |
+--------------------+-------------+
| ``SELECT`` | yes |
+--------------------+-------------+
| ``SET`` | yes |
+--------------------+-------------+
| ``SFUNC`` | no |
+--------------------+-------------+
| ``SMALLINT`` | no |
+--------------------+-------------+
| ``STATIC`` | no |
+--------------------+-------------+
| ``STORAGE`` | no |
+--------------------+-------------+
| ``STYPE`` | no |
+--------------------+-------------+
| ``SUPERUSER`` | no |
+--------------------+-------------+
| ``TABLE`` | yes |
+--------------------+-------------+
| ``TEXT`` | no |
+--------------------+-------------+
| ``TIME`` | no |
+--------------------+-------------+
| ``TIMESTAMP`` | no |
+--------------------+-------------+
| ``TIMEUUID`` | no |
+--------------------+-------------+
| ``TINYINT`` | no |
+--------------------+-------------+
| ``TO`` | yes |
+--------------------+-------------+
| ``TOKEN`` | yes |
+--------------------+-------------+
| ``TRIGGER`` | no |
+--------------------+-------------+
| ``TRUNCATE`` | yes |
+--------------------+-------------+
| ``TTL`` | no |
+--------------------+-------------+
| ``TUPLE`` | no |
+--------------------+-------------+
| ``TYPE`` | no |
+--------------------+-------------+
| ``UNLOGGED`` | yes |
+--------------------+-------------+
| ``UPDATE`` | yes |
+--------------------+-------------+
| ``USE`` | yes |
+--------------------+-------------+
| ``USER`` | no |
+--------------------+-------------+
| ``USERS`` | no |
+--------------------+-------------+
| ``USING`` | yes |
+--------------------+-------------+
| ``UUID`` | no |
+--------------------+-------------+
| ``VALUES`` | no |
+--------------------+-------------+
| ``VARCHAR`` | no |
+--------------------+-------------+
| ``VARINT`` | no |
+--------------------+-------------+
| ``WHERE`` | yes |
+--------------------+-------------+
| ``WITH`` | yes |
+--------------------+-------------+
| ``WRITETIME`` | no |
+--------------------+-------------+

Appendix B: CQL Reserved Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following type names are not currently used by CQL but are reserved
for potential future use. User-defined types may not use reserved type
names as their name.

+-----------------+
| type |
+=================+
| ``bitstring`` |
+-----------------+
| ``byte`` |
+-----------------+
| ``complex`` |
+-----------------+
| ``enum`` |
+-----------------+
| ``interval`` |
+-----------------+
| ``macaddr`` |
+-----------------+



.. include:: /rst_include/apache-copyrights-index.rst
.. include:: /rst_include/apache-cql-return-index.rst
Binary file added docs/getting-started/checkmark-16.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
352 changes: 352 additions & 0 deletions docs/getting-started/compaction.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,352 @@

.. _compaction:

Compaction
----------


This document describes the compaction strategy options available when creating a table. For more information about creating a table in Scylla, refer to the :ref:`CQL Reference <create-table-statement>`.

By default, Scylla starts a compaction task whenever a new SSTable is written. Compaction merges several SSTables into a new SSTable, which contains only the live data from the input SSTables. Merging several sorted files to get a sorted result is an efficient process, and this is the main reason why SSTables are kept sorted.

The following compaction strategies are supported by Scylla:

* Size-tiered Compaction Strategy (`STCS`_)

* Leveled Compaction Strategy (`LCS`_)

* Incremental Compaction Strategy (`ICS`_)

* Time-window Compaction Strategy (`TWCS`_)

* Date-tiered Compaction Strategy (DTCS) - use `TWCS`_ instead

This page concentrates on the parameters to use when creating a table with a compaction strategy. If you are unsure which strategy to use or want general information on the compaction strategies which are available to Scylla, refer to :doc:`Compaction Strategies </architecture/compaction/compaction-strategies>`.

Common options
^^^^^^^^^^^^^^
The following options are available for all compaction strategies.

.. code-block:: cql
compaction = {
'class' : 'compaction_strategy_name',
'enabled' : (true | false),
'tombstone_threshold' : ratio,
'tombstone_compaction_interval' : sec}
``class`` (default: SizeTieredCompactionStrategy)
Selects the compaction strategy.

It can be one of the following. If you are unsure which one to choose, refer to :doc:`Compaction Strategies </architecture/compaction/compaction-strategies>` :

* SizeTieredCompactionStrategy
* TimeWindowCompactionStrategy
* LeveledCompactionStrategy


=====

``enabled`` (default: true)
Runs background compaction (known as “minor compaction”). Can be one of the following:

* true - runs minor compaction
* false - disable minor compaction

=====

``tombstone_threshold`` (default: 0.2)
The ratio (expressed as a decimal) of garbage-collectable tombstones compared to the data. When this threshold is exceeded on a specific table, a single SSTable compaction begins. Acceptable values are numbers in the range 0 -1.

=====

``tombstone_compaction_interval`` (default: 86400s (1 day))
An SSTable that is suitable for single SSTable compaction, according to tombstone_threshold will not be compacted if it is newer than tombstone_compaction_interval.

=====

.. _STCS:

Size Tiered Compaction Strategy (STCS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When using STCS, SSTables are put in different buckets depending on their size. When an SSTable is bucketed, the average size of the tables is compared to the new table as well as the high and low threshold levels.

The database compares each SSTable size to the average of all SSTable sizes on the node. It calculates ``bucket_low * avg_bucket_size`` and ``bucket_high * avg_bucket_size`` and then compares the result with the average SSTable size. The conditions set for ``bucket_high`` and ``bucket_low`` dictate if successive tables will be added to the same bucket. When compaction begins it merges SSTables whose size in KB are within ``[average-size * bucket_low]`` and ``[average-size * bucket_high]``.

Once the ``min_threashold`` is reached, minor compaction begins.

.. _stcs-options:

STCS options
~~~~~~~~~~~~

The following options only apply to SizeTieredCompactionStrategy:

.. code-block:: cql
compaction = {
'class' : 'SizeTieredCompactionStrategy',
'bucket_high' : factor,
'bucket_low' : factor,
'min_sstable_size' : int,
'min_threshold' : num_sstables,
'max_threshold' : num_sstables}
``bucket_high`` (default: 1.5)
A new SSTable is added to the bucket if the SSTable size is less than bucket_high * the average size of that bucket (and if the bucket_low condition also holds).

For example, if **'bucket_high = 1.5'** and the **SSTable size = 14MB**, does it belong to a bucket with an average size of 10MB?

Yes, because the **SSTable size = 14`**, which is **less** than **'bucket_high' * average bucket size = 15**.

So, the SSTable will be added to the bucket, and the bucket’s average size will be recalculated.

=====

``bucket_low`` (default: 0.5)
A new SSTable is added to the bucket if the SSTable size is more than bucket_low* the average size of that bucket (and if the bucket_high condition also holds).

For example, if **'bucket_high = 0.5'** and the **SSTable size is 10MB**, does it belong to a bucket with an average size is 15MB?

Yes, because the **SSTable size = 10** which is **more** than **'bucket_low' * average bucket size = 7.5**.

So, the SSTable will be added to the bucket, and the bucket’s average size will be recalculated.

=====

``min_sstable_size`` (default: 50)
All SSTables smaller than this number of bytes are put into the same bucket.

=====

``min_threshold`` (default: 4)
Minimum number of SSTables that need to belong to the same size bucket before compaction is triggered on that bucket.
If your SSTables are small, use *min_sstable_size* to define a size threshold (in bytes) below which all SSTables belong to one unique bucket.

.. note:: Enforcement of ``min_threshold`` is controlled by the ``compaction_enforce_min_threshold`` configuration option in the scylla.yaml configuration settings.
By default, ``compaction_enforce_min_threshold: false``, meaning the Size-Tiered Compaction Strategy will compact any bucket containing 2 or more SSTables.
Otherwise, if ``compaction_enforce_min_threshold: true``, the value of ``min_threshold`` is considered and only those buckets that contain at
least ``min_threshold`` SSTables will be compacted.

=====

``max_threshold`` (default: 32)
Maximum number of SSTables that will be compacted together in one compaction step.



.. _LCS:

Leveled Compaction Strategy (LCS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The compaction class LeveledCompactionStrategy (LCS) creates SSTables of a fixed, relatively small size (160 MB by default) that are grouped into levels. Within each level, SSTables are guaranteed to be non-overlapping. Each level (L0, L1, L2 and so on) is ten times as large as the previous level.

.. _lcs-options:

LCS options
~~~~~~~~~~~

.. code-block:: cql
compaction = {
'class' : 'LeveledCompactionStrategy',
'sstable_size_in_mb' : int}
``sstable_size_in_mb`` (default: 160)
This is the target size in megabytes, that will be used as the goal for an SSTable size following a compression.
Although SSTable sizes should be less or equal to sstable_size_in_mb, it is possible that compaction could produce a larger SSTable during compaction. This occurs when data for a given partition key is exceptionally large.

=====

.. _ICS:

Incremental Compaction Strategy (ICS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. versionadded:: 2019.1.4 Scylla Enterprise

.. note:: ICS is only available for Scylla Enterprise customers

When using ICS, SSTable runs are put in different buckets depending on their size.
When an SSTable run is bucketed, the average size of the runs in the bucket is compared to the new run, as well as the ``bucket_high`` and ``bucket_low`` levels.


The database compares each SSTable-run size to the average of all SSTable-run sizes on all buckets in the node.
It calculates ``bucket_low * avg_bucket_size`` and ``bucket_high * avg_bucket_size`` and then compares the result with the ``average SSTable-run size``.
The conditions set for ``bucket_high`` and ``bucket_low`` dictate if successive runs will be added to the same bucket.
When compaction begins it merges SSTable runs whose size in KB are within ``[average-size * bucket_low]`` and ``[average-size * bucket_high]``.


Once there are multiple runs in a bucket, minor compaction begins.
The minimum number of SSTable runs that triggers minor compaction is either 2 or ``min_threshold``, if the ``compaction_enforce_min_threshold``
configuration option is set in the scylla.yaml configuration file.

.. _ics-options:

ICS options
~~~~~~~~~~~~

The following options only apply to IncrementalCompactionStrategy:

.. code-block:: cql
compaction = {
'class' : 'IncrementalCompactionStrategy',
'bucket_high' : factor,
'bucket_low' : factor,
'min_sstable_size' : int,
'min_threshold' : num_sstables,
'max_threshold' : num_sstables,
'sstable_size_in_mb' : int,
'space_amplification_goal' : double}
=====

``bucket_high`` (default: 1.5)
A new SSTable is added to the bucket if the SSTable size is **less than**
bucket_high * the average size of that bucket (and if the bucket_low condition also holds).
For example, if **'bucket_high = 1.5'** and the **SSTable size = 14MB**, does the SSTable belong to a bucket with an average size of 10MB?
Yes, because the **SSTable size = 14**, which is **less** than **'bucket_high' * average bucket size = 15**.
So, the SSTable will be added to the bucket, and the bucket’s average size will be recalculated.

=====

``bucket_low`` (default: 0.5)
A new SSTable is added to the bucket if the SSTable size is **more than**
bucket_low * the average size of that bucket (and if the bucket_high condition also holds).
For example, if **'bucket_high = 0.5'** and the **SSTable size is 10MB**, does the SSTable belong to a bucket with an average size of 15MB?
Yes, because the **SSTable size = 10**, which is **more** than **'bucket_low' * average bucket size = 7.5**.
So, the SSTable will be added to the bucket, and the bucket’s average size will be recalculated.

=====

``min_sstable_size`` (default: 50)
All SSTables smaller than this number of megabytes are put into the same bucket.

Unlike Apache Cassandra, scylla uses **uncompressed** size when bucketing similar-sized tiers together.
Since compaction works on uncompressed data, SSTables containing similar amounts of data should be compacted together, even when they have different compression ratios.

=====

``min_threshold`` (default: 4)
Minimum number of SSTable runs that need to belong to the same size bucket before compaction is triggered on that bucket.

.. note:: Enforcement of ``min_threshold`` is controlled by the ``compaction_enforce_min_threshold`` configuration option in the scylla.yaml configuration settings.
By default, ``compaction_enforce_min_threshold=false``, meaning the Incremental Compaction Strategy will compact any bucket containing 2 or more SSTable runs.
Otherwise, if ``compaction_enforce_min_threshold=true``, the value of ``min_threshold`` is considered and only those buckets that contain at
least ``min_threshold`` SSTable runs will be compacted.

=====

``max_threshold`` (default: 32)
Maximum number of SSTables that will be compacted together in one compaction step.

=====

``sstable_size_in_mb`` (default: 1000)
This is the target size in megabytes, that will be used as the goal for an SSTable size (fragment size) following a compression.

.. _SAG:

=====

``space_amplification_goal`` (default: null)

.. versionadded:: 2020.1.6 Scylla Enterprise

This is a threshold of the ratio of the sum of the sizes of the two largest tiers to the size of the largest tier,
above which ICS will automatically compact the second largest and largest tiers together to eliminate stale data that may have been overwritten, expired, or deleted.
The space_amplification_goal is given as a double-precision floating point number that must be greater than 1.0.

For example, if **'space_amplification_goal = 1.25'** and the largest tier holds **1000GB**,
when the second-largest tier accumulates SSTables with the total size of 250GB or more,
the ``space_amplification_goal`` threshold is crossed and all the SSTables in the largest and second-largest tiers will be compacted together.

=====

.. _TWCS:

Time Window CompactionStrategy (TWCS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The basic concept is that TimeWindowCompactionStrategy will create 1 SSTable per file for a given time window.

.. include:: /rst_include/warning-ttl-twcs.rst

.. _twcs-options:

TWCS options
~~~~~~~~~~~~

.. code-block:: cql
compaction = {
'class' : 'TimeWindowCompactionStrategy',
'compaction_window_unit' : string,
'compaction_window_size' : int,
'expired_sstable_check_frequency_seconds' : int,
'min_threshold' : num_sstables,
'max_threshold' : num_sstables}
``compaction_window_unit`` (default: DAYS)
A time unit used to determine the window size which can be one of the following:

* ``'MINUTES'``

* ``'HOURS'``

* ``'DAYS'``

=====

``compaction_window_size`` (default: 1)
The number of units which will make up a window.

=====

``expired_sstable_check_frequency_seconds`` (default: 600)
Specifies (in seconds) how often Scylla will check for fully expired SSTables, which can be immediately dropped.

=====

``min_threshold`` (default: 4)
Minimum number of SSTables that need to belong to the same size bucket before compaction is triggered on that bucket.

=====

``max_threshold`` (default: 32)
Maximum number of SSTables that will be compacted together in one compaction step.

=====

See Also
^^^^^^^^^

* :doc:`Apache Cassandra Query Language </getting-started/cql/>`

* :doc:`Compaction Strategies </architecture/compaction/compaction-strategies/>`

* :doc:`Compaction Overview </kb/compaction>`

.. include:: /rst_include/apache-copyrights.rst


.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. highlight:: none
23 changes: 23 additions & 0 deletions docs/getting-started/configure.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
Configure Scylla
================

.. raw:: html

<div class="panel callout radius animated">
<div class="row">
<div class="medium-3 columns">
<h5 id="getting-started">Configure Scylla</h5>
</div>
<div class="medium-9 columns">

.. include:: /rst_include/configure-index.rst


.. raw:: html

</div>
</div>
</div>



105 changes: 105 additions & 0 deletions docs/getting-started/consistency-calculator.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
<style>
.calculator {
width: fit-content;
}
.form-inline {
display: flex;
flex-flow: row wrap;
}
.form-inline label {
margin: 5px 10px 5px 0;
}
.form-inline input {
padding: 10px;
margin: 5px 10px 5px 0;
}
</style>
<div class="calculator">
<form class="form-inline" action="">
<div class="input">
<label for="nodes">Nodes</label><input style="width: 8ch;" type="number" min="1" value="3" name="nodes" id="nodes">
</div>
<div class="input">
<label for="replication-factor">Replication factor</label><input style="width: 8ch;" min="1" value="3" type="number" name="replication-factor" id="replication-factor">
</div>
<div class="input">
<label for="read-consistency">Read consistency level</label><select name="read-consistency" id="read-consistency">
<option value="ONE">ONE</option>
<option value="TWO">TWO</option>
<option value="THREE">THREE</option>
<option value="QUORUM">QUORUM</option>
<option value="ALL">ALL</option>
</select>
</div>
<div class="input">
<label for="write-consistency">Write consistency level</label><select name="write-consistency" id="write-consistency">
<option value="ONE">ONE</option>
<option value="TWO">TWO</option>
<option value="THREE">THREE</option>
<option value="QUORUM">QUORUM</option>
<option value="ALL">ALL</option>
</select>
</div>
</form>
<div class="results"></div>
</div>

<script>
function consistency(c, rf) {
switch(c) {
case "ONE":
return 1;
case "TWO":
return 2;
case "THREE":
return 3;
case "QUORUM":
return Math.floor(rf / 2) + 1;
case "ALL":
return rf;
}
}

function calculate() {
const nodes = Number.parseInt(document.querySelector("#nodes").value);
const readConsistency = document.querySelector("#read-consistency").value;
const writeConsistency = document.querySelector("#write-consistency").value;
const replicationFactor = document.querySelector("#replication-factor").value;
const writeConsistencyN = consistency(writeConsistency, replicationFactor);
const readConsistencyN = consistency(readConsistency, replicationFactor);


const resultsDiv = document.querySelector(".results");

if (nodes < replicationFactor) {
resultsDiv.textContent = "Replication factor must be smaller or equal to the number of nodes";
resultsDiv.classList.add("warning", "admonition");
} else if (writeConsistencyN > replicationFactor || readConsistencyN > replicationFactor) {
resultsDiv.textContent = "Consistency level must be lower than replication factor";
resultsDiv.classList.add("warning", "admonition");
} else {
const msg = [];
if (writeConsistencyN + readConsistencyN > replicationFactor) {
msg.push("Your reads are <b>consistent</b>");
} else {
msg.push("Your reads are <b>eventually consistent</b>");
}
msg.push(`You can lose ${replicationFactor - readConsistencyN} nodes without failing reads`);
msg.push(`You can lose ${writeConsistencyN - 1} nodes without data loss`);
msg.push(`Each node holds ~ ${Math.round(replicationFactor / nodes * 100)}% of your data`);

resultsDiv.classList.remove("warning", "admonition");
resultsDiv.innerHTML = msg.join("<br/>");
}
}

document.querySelectorAll(".input input").forEach((element) => {
element.addEventListener("input", calculate);
})

document.querySelectorAll(".input select").forEach((element) => {
element.addEventListener("input", calculate);
})

calculate();
</script>
128 changes: 128 additions & 0 deletions docs/getting-started/consistency.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
==================
Consistency Levels
==================


A :term:`Consistency Level (CL)` is a dynamic value which dictates the number of replicas (in a cluster) that must acknowledge a read or write operation in order for the coordinator node to determine the operation was successful.
CLs can be used with any transaction including LWTs.

This value is set by the client on a per operation basis. For the CQL Shell, the consistency level defaults to ONE for read and write operations.
If there is a conflict in settings, the CQLSH setting supersedes a consistency level global setting.


Syntax
------
.. code-block:: cql
CONSISTENCY <consistency level>
Examples
========

Set CONSISTENCY to force the majority of the nodes to respond:

.. code-block:: cql
CONSISTENCY QUORUM
Set level to serial for LWT read requests:

.. code-block:: cql
CONSISTENCY SERIAL
Consistency level set to SERIAL.

.. _consistency-levels-reference:

Consistency Levels Reference
============================

The following table describes the different levels you can set.

.. list-table::
:widths: 25 25 25 25
:header-rows: 1

* - Consistency Level
- Which Replicas Must Respond
- Consistency
- Availability
* - ANY (Write Only)
- Closest replica, as determined by the snitch. If all replica nodes are down, write succeeds after a :term:`hinted handoff`. Provides low latency, guarantees writes never fail.
- Lowest (WRITE)
- Highest (WRITE)
* - ONE
- The closest replica as determined by the :term:`Snitch`. Consistency requirements are not too strict.
- Lowest (READ)
- Highest (READ)
* - TWO
- The closest two replicas as determined by the Snitch.
-
-
* - THREE
- The closest three replicas as determined by the Snitch.
-
-
* - QUORUM
- A simple majority of all replicas across all datacenters. This CL allows for some level of failure
-
-
* - LOCAL_QUORUM
- Same as QUORUM, but confined to the same datacenter as the coordinator.
- Low in multi-data centers
-
* - ALL
- *All* replicas in the cluster
- Highest
- Lowest (may cause performance issues)
* - EACH_QUORUM (Write Only)
- A simple majority in each datacenter.
- Same across the datacenters.
-
* - LOCAL_ONE
- Same as ONE, but confined to the local datacenter.
-
-
* - SERIAL
- Returns results with the most recent data. Including uncommitted in-flight LWTs. Writes are not supported, but read transactions are supported.
- Linearizable
-
* - LOCAL_SERIAL
- Same as SERIAL, but confined to a local datacenter. Writes are not supported, but read transactions are supported.
- Linearizable for the local DC
-


Display the Current CL in CQLSh
-------------------------------

To display your current CL in your CQLsh session, use the CONSISTENCY Command with no options.

.. code-block:: cql
CONSISTENCY
returns

.. code-block:: cql
Current consistency level is ALL.
Consistency level calculator
----------------------------

.. raw:: html
:file: consistency-calculator.html


Additional Information
----------------------

* :doc:`Fault Tolerance </architecture/architecture-fault-tolerance/>`
* :doc:`Cluster membership changes and LWT consistency </operating-scylla/procedures/cluster-management/membership-changes/>`
* :ref:`Consistency Level Compatibility <consistency-level-read-and-write>`
* :doc:`Consistency Quiz </kb/quiz-administrators/>`
* Take a course on `Consistency Levels at Scylla University <https://university.scylladb.com/courses/scylla-essentials-overview/lessons/high-availability/topic/consistency-level/>`_
28 changes: 28 additions & 0 deletions docs/getting-started/cql.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Cassandra Query Language (CQL)
==============================

.. toctree::
:hidden:

CQLSh the CQL shell <cqlsh>
Data Definition </getting-started/ddl>
Data Manipulation </getting-started/dml>
Expiring Data with Time to Live </getting-started/time-to-live>
Security </operating-scylla/security/index>
Data Types </getting-started/types>
Appendices </getting-started/appendices>
Definitions </getting-started/definitions>
Materialized Views </getting-started/mv>
Functions </getting-started/functions>
JSON </getting-started/json>
Secondary Indexes </getting-started/secondary-indexes>
Compaction </getting-started/compaction>
Consistency Levels </getting-started/consistency>
Reserved Keywords </getting-started/reserved-keywords>
Non-reserved Keywords </getting-started/non-reserved-keywords>

.. include:: /rst_include/cql-index.rst

.. include:: /rst_include/apache-copyrights.rst

.. Operators </getting-started/operators>
537 changes: 537 additions & 0 deletions docs/getting-started/cqlsh.rst
View file

Large diffs are not rendered by default.

989 changes: 989 additions & 0 deletions docs/getting-started/ddl.rst
View file

Large diffs are not rendered by default.

226 changes: 226 additions & 0 deletions docs/getting-started/definitions.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. _UUID: https://en.wikipedia.org/wiki/Universally_unique_identifier

.. highlight:: cql

Definitions
-----------

.. include:: /rst_include/cql-version-index.rst

.. _conventions:

Conventions
^^^^^^^^^^^

To aid in specifying the CQL syntax, we will use the following conventions in this document:

- Language rules will be given in an informal `BNF variant
<http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form#Variants>`_ notation. In particular, we'll use square brackets
(``[ item ]``) for optional items, ``*`` and ``+`` for repeated items (where ``+`` imply at least one).
- The grammar will also use the following convention for convenience: non-terminal term will be lowercase (and link to
their definition) while terminal keywords will be provided "all caps". Note, however, that keywords are
:ref:`identifiers` and are thus case insensitive in practice. We will also define some early construction using
regexp, which we'll indicate with ``re(<some regular expression>)``.
- The grammar is provided for documentation purposes and leaves some minor details out. For instance, the comma on the
last column definition in a ``CREATE TABLE`` statement is optional but supported if present even though the grammar in
this document suggests otherwise. Also, not everything accepted by the grammar is necessarily valid CQL.
- References to keywords or pieces of CQL code in running text will be shown in a ``fixed-width font``.


.. _identifiers:

Identifiers and keywords
^^^^^^^^^^^^^^^^^^^^^^^^

The CQL language uses *identifiers* (or *names*) to identify tables, columns, and other objects. An identifier is a token
matching the regular expression ``[a-zA-Z][a-zA-Z0-9_]*``.

A number of such identifiers, like ``SELECT`` or ``WITH``, are *keywords*. They have a fixed meaning for the language,
and most are reserved. The list of those keywords can be found in :ref:`appendix-A`.

Identifiers and (unquoted) keywords are case insensitive. Thus ``SELECT`` is the same as ``select`` or ``sElEcT``, and
``myId`` is the same than ``myid`` or ``MYID``. A convention often used (in particular by the samples of this
documentation) is to use upper case for keywords and lower case for other identifiers.

There is a second kind of identifier called *quoted identifiers*, defined by enclosing an arbitrary sequence of
characters (non-empty) in double-quotes(``"``). Quoted identifiers are never keywords. Thus ``"select"`` is not a
reserved keyword and can be used to refer to a column (note that using this is particularly advised), while ``select``
would raise a parsing error. Also, contrary to unquoted identifiers and keywords, quoted identifiers are case
sensitive (``"My Quoted Id"`` is *different* from ``"my quoted id"``). A fully lowercase quoted identifier that matches
``[a-zA-Z][a-zA-Z0-9_]*`` is, however, *equivalent* to the unquoted identifier obtained by removing the double-quote (so
``"myid"`` is equivalent to ``myid`` and to ``myId`` but different from ``"myId"``). Inside a quoted identifier, the
double-quote character can be repeated to escape it, so ``"foo "" bar"`` is a valid identifier.

.. note:: *quoted identifiers* allow to declare columns with arbitrary names, and those can sometimes clash with
specific names used by the server. For instance, when using a conditional update, the server will respond with a
result-set containing a special result named ``"[applied]"``. If you’ve declared a column with such a name, this
could potentially confuse some tools and should be avoided. In general, unquoted identifiers should be preferred, but
if you use quoted identifiers, it is strongly advised to avoid any name enclosed by squared brackets (like
``"[applied]"``) and any name that looks like a function call (like ``"f(x)"``).

More formally, we have:

.. code-block::
identifier: `unquoted_identifier` | `quoted_identifier`
unquoted_identifier: re('[a-zA-Z][a-zA-Z0-9_]*')
quoted_identifier: '"' (any character where " can appear if doubled)+ '"'
.. _constants:

Constants
^^^^^^^^^

CQL defines the following kind of *constants*:

.. code-block:: none
constant: `string` | `integer` | `float` | `boolean` | `uuid` | `blob` | NULL
string: '\'' (any character where ' can appear if doubled)+ '\''
: '$$' (any character other than '$$') '$$'
integer: re('-?[0-9]+')
float: re('-?[0-9]+(\.[0-9]*)?([eE][+-]?[0-9+])?') | NAN | INFINITY
boolean: TRUE | FALSE
uuid: `hex`{8}-`hex`{4}-`hex`{4}-`hex`{4}-`hex`{12}
hex: re("[0-9a-fA-F]")
blob: '0' ('x' | 'X') `hex`+
In other words:

- A string constant is an arbitrary sequence of characters enclosed by single-quote(``'``). A single-quote
can be included by repeating it, e.g. ``'It''s raining today'``. Those are not to be confused with quoted
:ref:`identifiers` that use double-quotes. Alternatively, a string can be defined by enclosing the arbitrary sequence
of characters by two dollar characters, in which case single-quote can be used without escaping (``$$It's raining
today$$``). That latter form is often used when defining user-defined functions to avoid having to
escape single-quote characters in function body (as they are more likely to occur than ``$$``).
- Integer, float, and boolean constant are defined as expected. Note, however, than float allows the special ``NaN`` and
``Infinity`` constants.
- CQL supports UUID_ constants.
- Blob content types are provided in hexadecimal and prefixed by ``0x``.
- The special ``NULL`` constant denotes the absence of value.

For how these constants are typed, see the :doc:`data-types <types>` document.

Terms
^^^^^

CQL has the notion of a *term*, which denotes the kind of values that CQL support. Terms are defined by:

.. code-block:: cql
term: `constant` | `literal` | `function_call` | `arithmetic_operation` | `type_hint` | `bind_marker`
literal: `collection_literal` | `udt_literal` | `tuple_literal`
function_call: `identifier` '(' [ `term` (',' `term`)* ] ')'
arithmetic_operation: '-' `term` | `term` ('+' | '-' | '*' | '/' | '%') `term`
type_hint: '(' `cql_type` `)` term
bind_marker: '?' | ':' `identifier`
A term is thus one of:

- A :ref:`constant <constants>`.
- A literal for either :ref:`a collection <collections>`, a user-defined type or a tuple
(see the linked sections for details).
- An arithmetic operation between terms.
- A *type hint*
- A bind marker, which denotes a variable to be bound at execution time. See the section on :ref:`prepared-statements`
for details. A bind marker can be either anonymous (``?``) or named (``:some_name``). The latter form provides a more
convenient way to refer to the variable for binding it and should generally be preferred.


Comments
^^^^^^^^

A comment in CQL is a line beginning by either double dashes (``--``) or double slash (``//``).

Multi-line comments are also supported through enclosure within ``/*`` and ``*/`` (but nesting is not supported).

::

-- This is a comment
// This is a comment too
/* This is
a multi-line comment */

Statements
^^^^^^^^^^

CQL consists of statements that can be divided into the following categories:

- :doc:`Data Definition </getting-started/ddl/>` statements - to define and change how the data is stored (keyspaces and tables).
- :doc:`Data Manipulation </getting-started/dml/>` statements - for selecting, inserting and deleting data.
- :ref:`cql-permissions` statements.
- cql-triggers statements.

All the statements are listed below and are described in the rest of this documentation (see links above):

.. code-block:: cql
cql_statement: `statement` [ ';' ]
statement: `ddl_statement`
: | `dml_statement`
: | `secondary_index_statement`
: | `materialized_view_statement`
: | `role_or_permission_statement`
: | `udf_statement`
: | `udt_statement`
: | `trigger_statement`
ddl_statement: `use_statement`
: | `create_keyspace_statement`
: | `alter_keyspace_statement`
: | `drop_keyspace_statement`
: | `create_table_statement`
: | `alter_table_statement`
: | `drop_table_statement`
: | `truncate_statement`
dml_statement: `select_statement`
: | `insert_statement`
: | `update_statement`
: | `delete_statement`
: | `batch_statement`
trigger_statement: `create_trigger_statement`
: | `drop_trigger_statement`
.. _prepared-statements:

Prepared Statements
^^^^^^^^^^^^^^^^^^^

CQL supports *prepared statements*. Prepared statements are an optimization that allows parsing a query only once but
executes it multiple times with different concrete values.

Any statement that uses at least one bind marker (see :token:`bind_marker`) will need to be *prepared*. After which, the statement
can be *executed* by provided concrete values for each of its markers. The exact details of how a statement is prepared
and then executed depends on the CQL driver used, and you should refer to your driver documentation.


.. include:: /rst_include/apache-copyrights-index.rst
.. include:: /rst_include/apache-cql-return-index.rst
974 changes: 974 additions & 0 deletions docs/getting-started/dml.rst
View file

Large diffs are not rendered by default.

291 changes: 291 additions & 0 deletions docs/getting-started/functions.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,291 @@


.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. highlight:: cql

.. _cql-functions:

.. Need some intro for UDF and native functions in general and point those to it.
.. _udfs:
.. _native-functions:

Functions
---------

CQL supports two main categories of functions:

- The :ref:`scalar functions <scalar-functions>`, which simply take a number of values and produce an output with it.
- The :ref:`aggregate functions <aggregate-functions>`, which are used to aggregate multiple rows of results from a
``SELECT`` statement.

.. In both cases, CQL provides a number of native "hard-coded" functions as well as the ability to create new user-defined
.. functions.
.. .. note:: By default, the use of user-defined functions is disabled by default for security concerns (even when
.. enabled, the execution of user-defined functions is sandboxed and a "rogue" function should not be allowed to do
.. evil, but no sandbox is perfect so using user-defined functions is opt-in). See the ``enable_user_defined_functions``
.. in ``scylla.yaml`` to enable them.
.. A function is identifier by its name:
.. .. code-block::
function_name: [ `keyspace_name` '.' ] `name`

.. _scalar-functions:

Scalar functions
^^^^^^^^^^^^^^^^

.. _scalar-native-functions:

Native functions
~~~~~~~~~~~~~~~~

Cast
````

Supported starting from Scylla version 2.1

The ``cast`` function can be used to convert one native datatype to another.

The following table describes the conversions supported by the ``cast`` function. Scylla will silently ignore any cast converting a cast datatype into its own datatype.

=============== =======================================================================================================
From To
=============== =======================================================================================================
``ascii`` ``text``, ``varchar``
``bigint`` ``tinyint``, ``smallint``, ``int``, ``float``, ``double``, ``decimal``, ``varint``, ``text``,
``varchar``
``boolean`` ``text``, ``varchar``
``counter`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``float``, ``double``, ``decimal``, ``varint``,
``text``, ``varchar``
``date`` ``timestamp``
``decimal`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``float``, ``double``, ``varint``, ``text``,
``varchar``
``double`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``float``, ``decimal``, ``varint``, ``text``,
``varchar``
``float`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``double``, ``decimal``, ``varint``, ``text``,
``varchar``
``inet`` ``text``, ``varchar``
``int`` ``tinyint``, ``smallint``, ``bigint``, ``float``, ``double``, ``decimal``, ``varint``, ``text``,
``varchar``
``smallint`` ``tinyint``, ``int``, ``bigint``, ``float``, ``double``, ``decimal``, ``varint``, ``text``,
``varchar``
``time`` ``text``, ``varchar``
``timestamp`` ``date``, ``text``, ``varchar``
``timeuuid`` ``timestamp``, ``date``, ``text``, ``varchar``
``tinyint`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``float``, ``double``, ``decimal``, ``varint``,
``text``, ``varchar``
``uuid`` ``text``, ``varchar``
``varint`` ``tinyint``, ``smallint``, ``int``, ``bigint``, ``float``, ``double``, ``decimal``, ``text``,
``varchar``
=============== =======================================================================================================

The conversions rely strictly on Java's semantics. For example, the double value 1 will be converted to the text value
'1.0'. For instance::

SELECT avg(cast(count as double)) FROM myTable

Token
`````

The ``token`` function computes a token for a given partition key. The exact signature of the token function
depends on the table concerned and on the partitioner used by the cluster.

The arguments of the ``token`` depend on the type of the partition key columns that are used. The return type depends on
the partitioner in use:

- For Murmur3Partitioner, the return type is ``bigint``.

For instance, in a cluster using the default Murmur3Partitioner, if a table is defined by::

CREATE TABLE users (
userid text PRIMARY KEY,
username text,
)

The ``token`` function accepts single argument of type ``text`` (in that case, the partition key is ``userid``
(there are no clustering columns, so the partition key is the same as the primary key)), and the return type will be
``bigint``.

Uuid
````
The ``uuid`` function takes no parameters and generates a random type 4 uuid suitable for use in ``INSERT`` or
``UPDATE`` statements.

.. _timeuuid-functions:

.. Timeuuid functions
.. ``````````````````
.. ``now``
.. #######
.. The ``now`` function takes no arguments and generates, on the coordinator node, a new unique timeuuid at the
.. time the function is invoked. Note that this method is useful for insertion but is largely non-sensical in
.. ``WHERE`` clauses. For instance, a query of the form::

.. SELECT * FROM myTable WHERE t = now()
.. will never return any result by design, since the value returned by ``now()`` is guaranteed to be unique.
.. ``currentTimeUUID`` is an alias of ``now``.
.. ``minTimeuuid`` and ``maxTimeuuid``
.. ###################################
.. The ``minTimeuuid`` (resp. ``maxTimeuuid``) function takes a ``timestamp`` value ``t`` (which can be `either a timestamp
.. or a date string <timestamps>`) and return a *fake* ``timeuuid`` corresponding to the *smallest* (resp. *biggest*)
.. possible ``timeuuid`` having for timestamp ``t``. So for instance::
.. SELECT * FROM myTable
.. WHERE t > maxTimeuuid('2013-01-01 00:05+0000')
.. AND t < minTimeuuid('2013-02-02 10:00+0000')
.. will select all rows where the ``timeuuid`` column ``t`` is strictly older than ``'2013-01-01 00:05+0000'`` but strictly
.. younger than ``'2013-02-02 10:00+0000'``. Please note that ``t >= maxTimeuuid('2013-01-01 00:05+0000')`` would still
.. *not* select a ``timeuuid`` generated exactly at '2013-01-01 00:05+0000' and is essentially equivalent to ``t >
.. maxTimeuuid('2013-01-01 00:05+0000')``.
.. note:: We called the values generated by ``minTimeuuid`` and ``maxTimeuuid`` *fake* UUID because they do no respect
the Time-Based UUID generation process specified by the `RFC 4122 <http://www.ietf.org/rfc/rfc4122.txt>`__. In
particular, the value returned by these two methods will not be unique. This means you should only use those methods
for querying (as in the example above). Inserting the result of those methods is almost certainly *a bad idea*.

Datetime functions
``````````````````

.. versionadded:: 2.3

Retrieving the current date/time
################################

The following functions can be used to retrieve the date/time at the time where the function is invoked:

===================== ===============
Function name Output type
===================== ===============
``currentTimestamp`` ``timestamp``
``currentDate`` ``date``
``currentTime`` ``time``
``currentTimeUUID`` ``timeUUID``
===================== ===============

For example, to retrieve data up to today, run the following query::

SELECT * FROM myTable WHERE date >= currentDate()

Time conversion functions
#########################

A number of functions are provided to “convert” a ``timeuuid``, a ``timestamp``, or a ``date`` into another ``native``
type.

===================== =============== ===================================================================
Function name Input type Description
===================== =============== ===================================================================
``toDate`` ``timeuuid`` Converts the ``timeuuid`` argument into a ``date`` type
``toDate`` ``timestamp`` Converts the ``timestamp`` argument into a ``date`` type
``toTimestamp`` ``timeuuid`` Converts the ``timeuuid`` argument into a ``timestamp`` type
``toTimestamp`` ``date`` Converts the ``date`` argument into a ``timestamp`` type
``toUnixTimestamp`` ``timeuuid`` Converts the ``timeuuid`` argument into a ``bigInt`` raw value
``toUnixTimestamp`` ``timestamp`` Converts the ``timestamp`` argument into a ``bigInt`` raw value
``toUnixTimestamp`` ``date`` Converts the ``date`` argument into a ``bigInt`` raw value
``dateOf`` ``timeuuid`` Similar to ``toTimestamp(timeuuid)`` (DEPRECATED)
``unixTimestampOf`` ``timeuuid`` Similar to ``toUnixTimestamp(timeuuid)`` (DEPRECATED)
===================== =============== ===================================================================

.. Blob conversion functions
.. `````````````````````````
.. A number of functions are provided to “convert” the native types into binary data (``blob``). For every
.. ``<native-type>`` ``type`` supported by CQL (a notable exceptions is ``blob``, for obvious reasons), the function
.. ``typeAsBlob`` takes a argument of type ``type`` and return it as a ``blob``. Conversely, the function ``blobAsType``
.. takes a 64-bit ``blob`` argument and convert it to a ``bigint`` value. And so for instance, ``bigintAsBlob(3)`` is
.. ``0x0000000000000003`` and ``blobAsBigint(0x0000000000000003)`` is ``3``.
Blob conversion functions
`````````````````````````
A number of functions are provided to “convert” the native types into binary data (``blob``). For every
``<native-type>`` ``type`` supported by CQL (a notable exception is a ``blob``, for obvious reasons), the function
``typeAsBlob`` takes an argument of type ``type`` and returns it as a ``blob``. Conversely, the function ``blobAsType``
takes a 64-bit ``blob`` argument and converts it to a ``bigint`` value. For example, ``bigintAsBlob(3)`` is
``0x0000000000000003`` and ``blobAsBigint(0x0000000000000003)`` is ``3``.

.. _aggregate-functions:

Aggregate functions
^^^^^^^^^^^^^^^^^^^

Aggregate functions work on a set of rows. They receive values for each row and return one value for the whole set.

If ``normal`` columns, ``scalar functions``, ``UDT`` fields, ``writetime``, or ``ttl`` are selected together with
aggregate functions, the values returned for them will be the ones of the first row matching the query.

.. note::
The ``LIMIT`` and ``PER PARTITION LIMIT`` used in the ``SELECT`` query will have no effect if the limit is greater
than or equal to 1 because they are applied to the output of the aggregate functions (which return one value for
the whole set of rows).


Native aggregates
~~~~~~~~~~~~~~~~~

.. _count-function:

Count
`````

The ``count`` function can be used to count the rows returned by a query. Example::

SELECT COUNT (*) FROM plays;
SELECT COUNT (1) FROM plays;

It also can be used to count the non-null value of a given column::

SELECT COUNT (scores) FROM plays;

Max and Min
```````````

The ``max`` and ``min`` functions can be used to compute the maximum and the minimum value returned by a query for a
given column. For instance::

SELECT MIN (players), MAX (players) FROM plays WHERE game = 'quake';

Sum
```

The ``sum`` function can be used, to sum up all the values returned by a query for a given column. For instance::

SELECT SUM (players) FROM plays;

Avg
```

The ``avg`` function can be used to compute the average of all the values returned by a query for a given column. For
instance::

SELECT AVG (players) FROM plays;

.. _user-defined-aggregates-functions:

.. include:: /rst_include/apache-cql-return-index.rst

.. include:: /rst_include/apache-copyrights.rst
Binary file added docs/getting-started/images/checkmark-16.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/getting-started/images/x-mark-16.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
65 changes: 65 additions & 0 deletions docs/getting-started/index.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
Getting Started
===============

.. toctree::
:maxdepth: 2
:hidden:

install-scylla/index
configure
requirements
cql
cqlsh
Scylla Drivers </using-scylla/drivers/index>
Migrate to Scylla </using-scylla/migrate-scylla>
Integration Solutions </using-scylla/integrations/index>
tutorials

.. panel-box::
:title: Scylla Requirements
:id: "getting-started"
:class: my-panel

* :doc:`Scylla System Requirements Guide</getting-started/system-requirements/>`
* :doc:`OS Support by Platform and Version</getting-started/os-support/>`

.. panel-box::
:title: Install and Configure Scylla
:id: "getting-started"
:class: my-panel

* `Install Scylla (Binary Packages, Docker, or EC2) <https://www.scylladb.com/download/>`_ - Links to the ScyllaDB Download Center

* :doc:`Configure Scylla</getting-started/system-configuration/>`
* :doc:`Run Scylla in a Shared Environment </getting-started/scylla-in-a-shared-environment>`
* :doc:`Create a Scylla Cluster - Single Data Center (DC) </operating-scylla/procedures/cluster-management/create-cluster/>`
* :doc:`Create a Scylla Cluster - Multi Data Center (DC) </operating-scylla/procedures/cluster-management/create-cluster-multidc/>`

.. panel-box::
:title: Develop Applications for Scylla
:id: "getting-started"
:class: my-panel

* :doc:`Scylla Drivers</using-scylla/drivers/index>`
* `Get Started Lesson on Scylla University <https://university.scylladb.com/courses/scylla-essentials-overview/lessons/quick-wins-install-and-run-scylla/>`_
* :doc:`Apache Cassandra Query Language (CQL) Reference </getting-started/cql/>`
* :doc:`CQLSh: the CQL shell</getting-started/cqlsh/>`

.. panel-box::
:title: Use Scylla with Third-party Solutions
:id: "getting-started"
:class: my-panel

* :doc:`Migrate to Scylla </using-scylla/migrate-scylla>` - How to migrate your current database to Scylla
* :doc:`Integrate with Scylla </using-scylla/integrations/index>` - Integration solutions with Scylla


.. panel-box::
:title: Tutorials
:id: "getting-started"
:class: my-panel

The tutorials will show you how to use ScyllaDB as a data source for an application.

* `Build an IoT App with sensor simulator and a REST API <https://care-pet.docs.scylladb.com/>`_ - ScyllaDB Tutorial
* `Implement CRUD operations with a TODO App <https://github.com/scylladb/scylla-cloud-getting-started/>`_ - ScyllaDB Cloud Tutorial
11 changes: 11 additions & 0 deletions docs/getting-started/install-scylla/air-gapped-install.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
==============================
Air-gapped Server Installation
==============================

An air-gapped server is a server without any access to external repositories or connections to any network, including the internet.
To install Scylla on an air-gapped server, you first need to download the relevant files from a server that is not air-gapped and then and move the files to the air-gapped servers to complete the installation.

There are two ways to install Scylla on an air-gapped server:

- With root privileges (recommended): download the OS specific packages (rpms and debs) and install them with the package manager (dnf and apt). See `Install Scylla on an Air-gapped Server Using the Packages (Option 2) <https://www.scylladb.com/download/?platform=tar>`_.
- Without root privileges: using the :doc:`Scylla Unified Installer <unified-installer>`.
69 changes: 69 additions & 0 deletions docs/getting-started/install-scylla/config-commands.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
==============================
Scylla Configuration Reference
==============================

This guide describes the commands that you can use to configure your Scylla clusters.
The commands are all sent via the command line in a terminal and sudo or root access is not required as long as you have permission to execute in the directory.

.. caution:: You should **only** use configuration settings which are officially supported.

A list of all Scylla commands can be obtained by running

.. code-block:: none
scylla --help
.. note:: This command displays all Scylla commands as well as Seastar commands. The Seastar commands are listed as Core Options.

For example:

.. code-block:: none
Scylla version 4.2.3-0.20210104.24346215c2 with build-id 0c8faf8bb8a3a0eda9337aad98ed3a6d814a4fa9 starting ...
command used: "scylla --help"
parsed command line options: [help]
Scylla options:
-h [ --help ] show help message
--version print version number and exit
--options-file arg configuration file (i.e.
<SCYLLA_HOME>/conf/scylla.yaml)
--memtable-flush-static-shares arg If set to higher than 0, ignore the
controller's output and set the
memtable shares statically. Do not set
this unless you know what you are doing
and suspect a problem in the
controller. This option will be retired
when the controller reaches more
maturity
--compaction-static-shares arg If set to higher than 0, ignore the
controller's output and set the
compaction shares statically. Do not
set this unless you know what you are
doing and suspect a problem in the
controller. This option will be retired
when the controller reaches more
maturity
.. note:: This is an incomplete screenshot. For the complete file, run the command in a terminal.

Scylla Configuration Files and Scylla Commands
----------------------------------------------

Some Scylla Command Line commands are derived from the `scylla.yaml <https://github.com/scylladb/scylla/blob/master/conf/scylla.yaml>`_ configuration parameters.

For example, in the case of ``cluster_name: 'Test Cluster'`` as seen in the `scylla.yaml <https://github.com/scylladb/scylla/blob/master/conf/scylla.yaml>`_ configuration parameters.

To send this configuration setting with the command line, run:

.. code-block:: none
scylla --cluster-name 'Test Cluster'
As you can see from the example above, the general rule of thumb is:

#. Take a configuration parameter from the scylla.yaml file.
#. Prepend it with ``scylla --``.
#. In any place where there is an underscore, replace with a dash.
#. Run the command in a terminal.

17 changes: 17 additions & 0 deletions docs/getting-started/install-scylla/dev-mod.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Scylla Developer Mode
=====================

If you want to use Scylla in developer mode you need to use the command below (using root privileges)

``sudo scylla_dev_mode_setup --developer-mode 1``


This script will write the developer mode setting into ``/etc/scylla.d/dev-mode.conf``








21 changes: 21 additions & 0 deletions docs/getting-started/install-scylla/disable-housekeeping.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@

=========================================
Scylla Housekeeping and how to disable it
=========================================

It is always recommended to run the latest version of Scylla Open Source or Scylla Enterprise.
The latest stable release version is always available from the `Download Center <https://www.scylladb.com/download/>`_.

When you install Scylla, it installs by default two services: **scylla-housekeeping-restart** and **scylla-housekeeping-daily**. These services check for the latest Scylla version and prompt the user if they are using a version that is older than what is publicly available.
Information about your Scylla deployment, including the Scylla version currently used, as well as unique user and server identifiers, are collected by a centralized service.

To disable these service, update file ``/etc/scylla.d/housekeeping.cfg`` as follow: ``check-version: False``

See also:

* `Scylla privacy <https://www.scylladb.com/privacy/>`_
* :doc:`Getting Started </getting-started/index>`




37 changes: 37 additions & 0 deletions docs/getting-started/install-scylla/index.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
Install Scylla
==============

.. toctree::
:maxdepth: 2
:hidden:

scylla-web-installer
unified-installer
air-gapped-install
rpm-info
disable-housekeeping
dev-mod
config-commands

.. panel-box::
:title: Install Scylla
:id: "getting-started"
:class: my-panel

Keep your versions up-to-date. The two latest versions are supported. Also always install the latest patches for your version.

* Download and install Scylla Server, Drivers and Tools in `Scylla Download Center <https://www.scylladb.com/download/#server/>`_
* :doc:`ScyllaDB Web Installer for Linux <scylla-web-installer>`
* :doc:`Scylla Unified Installer (relocatable executable) <unified-installer>`
* :doc:`Air-gapped Server Installation <air-gapped-install>`
* :doc:`What is in each RPM </getting-started/install-scylla/rpm-info/>`
* :doc:`Create a Scylla Cluster - Single Data Center (DC) </operating-scylla/procedures/cluster-management/create-cluster/>`
* :doc:`Create a Scylla Cluster - Multi Data Center (DC) </operating-scylla/procedures/cluster-management/create-cluster-multidc/>`
* :doc:`Scylla Developer Mode </getting-started/install-scylla/dev-mod>`
* :doc:`Scylla Configuration Command Reference </getting-started/install-scylla/config-commands>`
* `Scylla Housekeeping and how to disable it <disable-housekeeping>`_





13 changes: 13 additions & 0 deletions docs/getting-started/install-scylla/rpm-info.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
What is in each RPM
^^^^^^^^^^^^^^^^^^^

* scylla : Scylla is a highly scalable, eventually consistent, distributed, partitioned row DB. Includes all other packages.
* scylla-server : The Scylla database server
* scylla-conf : Scylla configuration package
* scylla-debuginfo : Debug information for package scylla
* scylla-jmx : Scylla JMX
* scylla-kernel-conf : Scylla configuration package for the Linux kernel
* scylla-machine-image.noarch : Scylla Machine Image
* scylla-tools.noarch : Scylla Tools, like nodetool, cqlsh, cassandra-stress
* scylla-tools-core.noarch : Core files for Scylla tools
* scylla-python3 : A standalone python3 interpreter that can be moved around different Linux machines
61 changes: 61 additions & 0 deletions docs/getting-started/install-scylla/scylla-web-installer.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
==================================
ScyllaDB Web Installer for Linux
==================================

ScyllaDB Web Installer is a platform-agnostic installation script you can run with ``curl`` to install ScyllaDB on Linux.

See `ScyllaDB Download Center <https://www.scylladb.com/download/#server>`_ for information on manually installing ScyllaDB with platform-specific installation packages.

Prerequisites
--------------

Ensure that your platform is supported by the ScyllaDB version you want to install.
See :doc:`OS Support by Platform and Version </getting-started/os-support/>`.

Installing ScyllaDB with Web Installer
---------------------------------------
To install ScyllaDB with Web Installer, run:

.. code:: console
curl -sSf get.scylladb.com/server | sudo bash
By default, running the script installs the latest official version of ScyllaDB Open Source. You can use the following
options to install a different version or ScyllaDB Enterprise:

.. list-table::
:widths: 20 25 55
:header-rows: 1

* - Option
- Acceptable values
- Description
* - ``--scylla-product``
- ``scylla`` | ``scylla-enterprise``
- Specifies the ScyllaDB product to install: Open Source (``scylla``) or Enterprise (``scylla-enterprise``) The default is ``scylla``.
* - ``--scylla-version``
- ``<version number>``
- Specifies the ScyllaDB version to install. You can specify the major release (``x.y``) to install the latest patch for that version or a specific patch release (``x.y.x``). The default is the latest official version.

You can run the command with the ``-h`` or ``--help`` flag to print information about the script.

Examples
---------

Installing ScyllaDB Open Souce 4.6.1:

.. code:: console
curl -sSf get.scylladb.com/server | sudo bash -s -- --scylla-version 4.6.1
Installing the latest patch release for ScyllaDB Open Source 4.6:

.. code:: console
curl -sSf get.scylladb.com/server | sudo bash -s -- --scylla-version 4.6
Installing ScyllaDB Enterprise 2021.1:

.. code:: console
curl -sSf get.scylladb.com/server | sudo bash -s -- --scylla-product scylla-enterprise --scylla-version 2021.1
64 changes: 64 additions & 0 deletions docs/getting-started/install-scylla/unified-installer.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
=================================================
Scylla Unified Installer (relocatable executable)
=================================================

This document covers how to install, uninstall, and upgrade using the Scylla Unified Installer. The Unified Installer is recommended to be used when you do not have root privileges to the server.
If you have root privileges, it is recommended to download the OS specific packages (RPMs and DEBs) and install them with the package manager (dnf and apt).

Supported distros
=================

* CentOS 7 (Only support root offline install)
* CentOS 8
* Ubuntu 18.04 (developer-mode is used if NOFILE rlimit is too low)
* Debian 10

Download and install
====================

For installation without root privileges, follow the instructions on `Scylla Download Center <https://www.scylladb.com/download/?platform=tar>`_

Upgrade / Downgrade/ Uninstall
==============================

.. _unified-installed-upgrade:

Upgrade
-------

The unified package is based on a binary package; it’s not a RPM / DEB packages, so it doesn’t upgrade or downgrade by yum / apt. Currently, only install.sh of scylla supports the upgrade.

Root install:

.. code:: sh
./install.sh --upgrade
Nonroot install

.. code:: sh
./install.sh --upgrade --nonroot
.. note:: the installation script does not upgrade scylla-jmx and scylla-tools. You will have to do this separately.

Uninstall
---------

Root uninstall:

.. code:: sh
sudo ./uninstall.sh
Nonroot uninstall

.. code:: sh
./uninstall.sh --nonroot
Downgrade
---------

To downgrade to your original Scylla version, use the Uninstall_ procedure above and then install the original Scylla packages.
124 changes: 124 additions & 0 deletions docs/getting-started/json.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@


.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. highlight:: cql

.. _cql-json:

JSON Support
------------

.. versionadded:: 2.3

Scylla introduces JSON support to :ref:`SELECT <select-statement>` and :ref:`INSERT <insert-statement>`
statements. This support does not fundamentally alter the CQL API (for example, the schema is still enforced). It simply
provides a convenient way to work with JSON documents.

SELECT JSON
^^^^^^^^^^^

With ``SELECT`` statements, the ``JSON`` keyword can be used to return each row as a single ``JSON`` encoded map. The
remainder of the ``SELECT`` statement behavior is the same.

The result map keys are the same as the column names in a normal result set. For example, a statement like ``SELECT JSON
a, ttl(b) FROM ...`` would result in a map with keys ``"a"`` and ``"ttl(b)"``. However, this is one notable exception:
for symmetry with ``INSERT JSON`` behavior, case-sensitive column names with upper-case letters will be surrounded with
double-quotes. For example, ``SELECT JSON myColumn FROM ...`` would result in a map key ``"\"myColumn\""`` (note the
escaped quotes).

The map values will ``JSON``-encoded representations (as described below) of the result set values.

INSERT JSON
^^^^^^^^^^^

With ``INSERT`` statements, the new ``JSON`` keyword can be used to enable inserting a ``JSON`` encoded map as a single
row. The format of the ``JSON`` map should generally match that returned by a ``SELECT JSON`` statement on the same
table. In particular, case-sensitive column names should be surrounded by double-quotes. For example, to insert into a
table with two columns named "myKey" and "value", you would do the following::

INSERT INTO mytable JSON '{ "\"myKey\"": 0, "value": 0}'

By default (or if ``DEFAULT NULL`` is explicitly used), a column omitted from the ``JSON`` map will be set to ``NULL``,
meaning that any pre-existing value for that column will be removed (resulting in a tombstone being created).
Alternatively, if the ``DEFAULT UNSET`` directive is used after the value, omitted column values will be left unset,
meaning that pre-existing values for those columns will be preserved.


JSON Encoding of Scylla Data Types
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Where possible, Scylla will represent and accept data types in their native ``JSON`` representation. Scylla will
also accept string representations matching the CQL literal format for all single-field types. For example, floats,
ints, UUIDs, and dates can be represented by CQL literal strings. However, compound types, such as collections, tuples,
and user-defined types, must be represented by native ``JSON`` collections (maps and lists) or a JSON-encoded string
representation of the collection.

The following table describes the encodings that Scylla will accept in ``INSERT JSON`` values (and ``fromJson()``
arguments) as well as the format Scylla will use when returning data for ``SELECT JSON`` statements (and
``fromJson()``):

=============== ======================== =============== ==============================================================
Type Formats accepted Return format Notes
=============== ======================== =============== ==============================================================
``ascii`` string string Uses JSON's ``\u`` character escape
``bigint`` integer, string integer String must be valid 64 bit integer
``blob`` string string String should be 0x followed by an even number of hex digits
``boolean`` boolean, string boolean String must be "true" or "false"
``date`` string string Date in format ``YYYY-MM-DD``, timezone UTC
``decimal`` integer, float, string float May exceed 32 or 64-bit IEEE-754 floating point precision in
client-side decoder
``double`` integer, float, string float String must be valid integer or float
``float`` integer, float, string float String must be valid integer or float
``inet`` string string IPv4 or IPv6 address
``int`` integer, string integer String must be valid 32 bit integer
``list`` list, string list Uses JSON's native list representation
``map`` map, string map Uses JSON's native map representation
``smallint`` integer, string integer String must be valid 16 bit integer
``set`` list, string list Uses JSON's native list representation
``text`` string string Uses JSON's ``\u`` character escape
``time`` string string Time of day in format ``HH-MM-SS[.fffffffff]``
``timestamp`` integer, string string A timestamp. Strings constant allows to input :ref:`timestamps
as dates <timestamps>`. Datestamps with format ``YYYY-MM-DD
HH:MM:SS.SSS`` are returned.
``timeuuid`` string string Type 1 UUID. See :token:`constant` for the UUID format
``tinyint`` integer, string integer String must be valid 8 bit integer
``tuple`` list, string list Uses JSON's native list representation
``UDT`` map, string map Uses JSON's native map representation with field names as keys
``uuid`` string string See :token:`constant` for the UUID format
``varchar`` string string Uses JSON's ``\u`` character escape
``varint`` integer, string integer Variable length; may overflow 32 or 64 bit integers in
client-side decoder
=============== ======================== =============== ==============================================================

The fromJson() Function
^^^^^^^^^^^^^^^^^^^^^^^

The ``fromJson()`` function may be used similarly to ``INSERT JSON``, but for a single column value. It may only be used
in the ``VALUES`` clause of an ``INSERT`` statement or as one of the column values in an ``UPDATE``, ``DELETE``, or
``SELECT`` statement. For example, it cannot be used in the selection clause of a ``SELECT`` statement.

The toJson() Function
^^^^^^^^^^^^^^^^^^^^^

The ``toJson()`` function may be used similarly to ``SELECT JSON``, but for a single column value. It may only be used
in the selection clause of a ``SELECT`` statement.


:doc:`Apache Cassandra Query Language </getting-started/cql/>`

.. include:: /rst_include/apache-copyrights.rst
78 changes: 78 additions & 0 deletions docs/getting-started/logging.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
Logging
=======

Logging with the systemd journal (CentOS, Amazon AMI, Ubuntu, Debian)
---------------------------------------------------------------------
On distributions with systemd, Scylla logs are written to the `systemd journal <http://www.freedesktop.org/software/systemd/man/systemd-journald.service.html>`_. You can retrieve log entries with the `journalctl <http://www.freedesktop.org/software/systemd/man/journalctl.html>`_ command.

Listed below are a few useful examples.

* get logs generated by the "scylla" user:

.. code-block:: shell
journalctl _UID=`id -u scylla`
* get logs generated by the "scylla" command:

.. code-block:: shell
journalctl _COMM=scylla
* filter only Scylla logs by priority:

.. code-block:: shell
journalctl _COMM=scylla -p err..emerg
.. code-block:: shell
journalctl _COMM=scylla -p warning
* filter only Scylla logs by date:

.. code-block:: shell
journalctl _COMM=scylla --since="2013-3-16 23:59:59"
.. code-block:: shell
journalctl _COMM=scylla --since "2015-01-10" --until "2015-01-11 03:00"`
.. code-block:: shell
journalctl _COMM=scylla --since yesterday
* filter only Scylla logs since last server boot:
.. code-block:: shell
journalctl _COMM=scylla -b
Logging on Ubuntu 14.04
-----------------------
On Ubuntu 14.04, Scylla writes its initial boot message into :code:`/var/log/upstart/scylla-server.log`.
After Scylla has started, logs are stored in :code:`/var/log/syslog`. Scylla logs can be filter by creating a :code:`rsyslog` configuration file with the following rule (for example, in :code:`/etc/rsyslog.d/10-scylla.conf`)
.. code-block:: shell
:syslogtag, startswith, "scylla" /var/log/scylla/scylla.log
& ~
And then creating the log file with the correct permissions and restarting the service:
.. code-block:: shell
install -o syslog -g adm -m 0640 /dev/null /var/log/scylla/scylla.log
service rsyslog restart
This will send Scylla only logs to :code:`/var/log/scylla/scylla.log`
Logging on Docker
-----------------
Starting from Scylla 1.3, `Scylla Docker <https://hub.docker.com/r/scylladb/scylla/>`_, you should use :code:`docker logs` command to access Scylla server and JMX proxy logs
.. include:: /rst_include/advance-index.rst
184 changes: 184 additions & 0 deletions docs/getting-started/mv.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@


.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. highlight:: cql

.. _materialized-views:

Materialized Views
------------------
Production ready in Scylla Open Source 3.0 and Scylla Enterprise 2019.1.x

.. include:: /rst_include/cql-version-index.rst


Materialized views names are defined by:

.. code-block:: cql
view_name: re('[a-zA-Z_0-9]+')
.. _create-materialized-view-statement:

CREATE MATERIALIZED VIEW
........................

You can create a materialized view on a table using a ``CREATE MATERIALIZED VIEW`` statement:

.. code-block:: cql
create_materialized_view_statement: CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] `view_name` AS
: `select_statement`
: PRIMARY KEY '(' `primary_key` ')'
: WITH `table_options`
For instance::

CREATE MATERIALIZED VIEW monkeySpecies_by_population AS
SELECT * FROM monkeySpecies
WHERE population IS NOT NULL AND species IS NOT NULL
PRIMARY KEY (population, species)
WITH comment='Allow query by population instead of species';

The ``CREATE MATERIALIZED VIEW`` statement creates a new materialized view. Each view is a set of *rows* that
corresponds to the rows that are present in the underlying, or base table, as specified in the ``SELECT`` statement. A
materialized view cannot be directly updated, but updates to the base table will cause corresponding updates in the
view.

Creating a materialized view has three main parts:

- The :ref:`select statement <mv-select>` that restricts the data included in the view.
- The :ref:`primary key <mv-primary-key>` definition for the view.
- The :ref:`options <mv-options>` for the view.

Attempting to create an already existing materialized view will return an error unless the ``IF NOT EXISTS`` option is
used. If it is used, the statement will be a no-op if the materialized view already exists.

.. _mv-select:

MV Select Statement
...................

The select statement of a materialized view creation defines which of the base table is included in the view. That
statement is limited in a number of ways:

- The :ref:`selection <selection-clause>` is limited to those that only select columns of the base table. In other
words, you can't use any function (aggregate or not), casting, term, etc. Aliases are also not supported. You can,
however, use `*` as a shortcut to selecting all columns. Further, :ref:`static columns <static-columns>` cannot be
included in a materialized view (which means ``SELECT *`` isn't allowed if the base table has static columns).
- The ``WHERE`` clause has the following restrictions:

- It cannot include any :token:`bind_marker`.
- The columns that are not part of the *base table* primary key can only be restricted by an ``IS NOT NULL``
restriction. No other restriction is allowed.
- As the columns that are part of the *view* primary key cannot be null, they must always be at least restricted by a
``IS NOT NULL`` restriction (or any other restriction, but they must have one).
- They can also be restricted by relational operations (=, >, <).

- The SELECT statement cannot include **any** of the following:

- An :ref:`ordering clause <ordering-clause>`
- A :ref:`limit <limit-clause>` clause
- An :ref:`ALLOW FILTERING <allow-filtering>` clause.

.. _mv-primary-key:

MV Primary Key
..............

A view must have a primary key, and that primary key must conform to the following restrictions:

- It must contain all the primary key columns of the base table. This ensures that every row in the view corresponds to
exactly one row of the base table.
- It can only contain a single column that is not a primary key column in the base table.

So, for instance, give the following base table definition::

CREATE TABLE t (
k int,
c1 int,
c2 int,
v1 int,
v2 int,
PRIMARY KEY (k, c1, c2)
);

then the following view definitions are allowed::

CREATE MATERIALIZED VIEW mv1 AS
SELECT * FROM t WHERE k IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL
PRIMARY KEY (c1, k, c2);

CREATE MATERIALIZED VIEW mv1 AS
SELECT * FROM t WHERE v1 IS NOT NULL AND k IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL
PRIMARY KEY (v1, k, c1, c2);

but the following ones are **not** allowed::

// Error: cannot include both v1 and v2 in the primary key as both are not in the base table primary key
CREATE MATERIALIZED VIEW mv1 AS
SELECT * FROM t WHERE k IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL AND v1 IS NOT NULL
PRIMARY KEY (v1, v2, k, c1, c2)

// Error: must include k in the primary as it's a base table primary key column
CREATE MATERIALIZED VIEW mv1 AS
SELECT * FROM t WHERE c1 IS NOT NULL AND c2 IS NOT NULL
PRIMARY KEY (c1, c2)


.. _mv-options:

MV Options
..........

A materialized view is internally implemented by a table, and as such, creating a MV allows the :ref:`same options than
creating a table <create-table-options>`.


.. _alter-materialized-view-statement:

ALTER MATERIALIZED VIEW
.......................

After creation, you can alter the options of a materialized view using the ``ALTER MATERIALIZED VIEW`` statement:

.. code-block::
alter_materialized_view_statement: ALTER MATERIALIZED VIEW `view_name` WITH `table_options`
The options that can be updated with an ALTER statement are the same as those used with a CREATE statement (see :ref:`Create table options <create-table-options>`).

.. _drop-materialized-view-statement:

DROP MATERIALIZED VIEW
......................

Dropping a materialized view users the ``DROP MATERIALIZED VIEW`` statement:

.. code-block::
drop_materialized_view_statement: DROP MATERIALIZED VIEW [ IF EXISTS ] `view_name`;
If the materialized view does not exist, the statement will return an error unless ``IF EXISTS`` is used, in which case
the operation is a no-op.


:doc:`Apache Cassandra Query Language </getting-started/cql/>`

.. include:: /rst_include/apache-copyrights.rst
Loading