NoSQL data store using the seastar framework, compatible with Apache Cassandra
C++ Python Other
Latest commit 1857ba0 Jan 12, 2017 @raphaelsc raphaelsc committed with avikivity db: fix bad resource usage distribution when resharding due to refresh
That's because a single shard is used to calculate generation for new
sstables in upload directory, and that will result in that single shard
sharing all the resources with other shards.
For refresh without upload dir, it currently works fine because we
reshuffle column family dir instead.

flush_upload_dir() is now a free function, takes a distributed database
object, and uses calculate_shard_from_sstable_generation() to decide
which shard will move sstable using its own generation namespace.

Fixes #2008.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <b0cccf7bbb61416ff8718bac92fdca90cc5fb9c9.1484253232.git.raphaelsc@scylladb.com>
Permalink
Failed to load latest commit information.
.github Make github issue template less shouty Jun 1, 2016
api storage_service: make is_joined() an immediate function Dec 28, 2016
auth migration_listener: Listen for view schema changes Dec 20, 2016
conf conf/scylla.yaml: Move broadcast_rpc_address to the supported section Dec 29, 2016
cql3 Replace iostream include with iosfwd in headers Jan 17, 2017
db Replace iostream include with iosfwd in headers Jan 17, 2017
dht dht: use nonwrapping_ranges in ring_position_range_sharder Dec 22, 2016
dist dist/redhat: fix path of housekeeping.cfg Jan 19, 2017
docs docs: tracing.md: initial commit Oct 10, 2016
exceptions exceptions: mark function called before construction static Oct 3, 2016
gms Replace iostream include with iosfwd in headers Jan 17, 2017
idl idl: is_short_read() was added in 1.6 Dec 22, 2016
interface Fix pre-ScyllaDB copyright statements Apr 8, 2016
io Fix pre-ScyllaDB copyright statements Apr 8, 2016
licenses Merge "CQL 3.3.1 support" from Pekka Jan 9, 2017
locator Convert to use dht::partition_range_vector and dht::token_range_vector Dec 19, 2016
message storage_proxy: pass maximum result size to replicas Dec 22, 2016
repair Convert to use dht::partition_range_vector and dht::token_range_vector Dec 19, 2016
scripts scripts/scylla_install_pkg: support Debian Dec 6, 2016
seastar @ ff098c8 Merge seastar upstream Jan 19, 2017
service db: fix bad resource usage distribution when resharding due to refresh Jan 19, 2017
sstables sstables: Close _file even if random_access_reader::close() reports e… Jan 18, 2017
streaming streaming::stream_manager: move a collectd counters registration to t… Jan 10, 2017
swagger-ui @ 1b212bb Update swagger-ui for local fix (change URL to not to point to pet st… Jun 25, 2015
tests lsa: Reduce reclamation latency Jan 19, 2017
thrift thrift::server: move collectd counters registration to the metrics re… Jan 10, 2017
tools/scyllatop scyllatop: dump all output to stdout instead of running a fancy conso… Aug 31, 2016
tracing remove scollectd from headers Jan 17, 2017
transport transport::server: move collectd metrics registration to the metrics … Jan 10, 2017
utils lsa: Reduce reclamation latency Jan 19, 2017
.gitattributes Add .gitattributes file to classify C++ source Oct 5, 2015
.gitignore ScyllaTop: top-like tool to see live scylla metrics Feb 23, 2016
.gitmodules dist: move ComboAMI related code to scylla-ami Sep 21, 2015
.gitorderfile gitorderfile: make changes into *.py files appear first May 12, 2015
CONTRIBUTING.md CONTRIBUTING.md: add sections for help and issues Nov 18, 2016
Doxyfile docs: exclude dpdk Jun 24, 2015
IDL.md Add an IDL definition file Jan 24, 2016
LICENSE.AGPL Add the AGPL license Sep 20, 2015
NOTICE.txt Add NOTICE file as required by the Apache license. Dec 24, 2014
ORIGIN Update ORIGIN for gossip and storage_service Dec 1, 2015
README-DPDK.md README: fix typos and paramter syntax Jun 28, 2015
README.md README: Guidelines for contributing Nov 16, 2016
SCYLLA-VERSION-GEN dist: renumber development version as 666.development Jan 12, 2016
atomic_cell.hh Replace iostream include with iosfwd in headers Jan 17, 2017
atomic_cell_hash.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
atomic_cell_or_collection.hh rename memory_usage() to external_memory_usage() where applicable Nov 18, 2016
bytes.cc Fix pre-ScyllaDB copyright statements Apr 8, 2016
bytes.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
bytes_ostream.hh bytes_ostream: make max_chunk_size() an inline function Oct 17, 2016
caching_options.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
canonical_mutation.cc idl: allow writers to use any output stream Dec 22, 2016
canonical_mutation.hh remove db/serializer.hh includes Mar 2, 2016
cartesian_product.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
checked-file-impl.hh db: avoid excessive memory usage during resharding Jan 9, 2017
clustering_bounds_comparator.hh clustering_bounds_comparator: Add tri_comparator Dec 28, 2016
clustering_key_filter.hh Remove clustering_key_filtering_context. Aug 30, 2016
combine.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
compaction_strategy.hh lcs: fix broken token range distribution at higher levels Sep 30, 2016
compatible_ring_position.hh compatible_ring_position: add function to return token Dec 8, 2016
compound.hh Replace iostream include with iosfwd in headers Jan 17, 2017
compound_compat.hh composite: Use operator[] instead of at() Jul 28, 2016
compress.hh cql3: Disable compression on empty properties Nov 9, 2016
configure.py build: track system header changes too Jan 8, 2017
converting_mutation_partition_applier.hh schema: Dense schemas are correctly upgrades Aug 3, 2016
cql_serialization_format.hh Replace iostream include with iosfwd in headers Jan 17, 2017
database.cc db: fix bad resource usage distribution when resharding due to refresh Jan 19, 2017
database.hh db: fix bad resource usage distribution when resharding due to refresh Jan 19, 2017
database_fwd.hh database: keep a pointer to the memtable list in a memtable Nov 21, 2016
db_clock.hh db: Use microsecond precision for server-side timestamps Dec 21, 2016
debug.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
digest_algorithm.hh storage_proxy: avoid calculating digest when only one replica is cont… Nov 17, 2016
disk-error-handler.cc db: make it possible to use custom error handler with io checker Oct 27, 2016
disk-error-handler.hh db: make it possible to use custom error handler with io checker Oct 27, 2016
dns.cc dns: Move gethostbyname to source file Oct 26, 2015
dns.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
enum_set.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
frozen_mutation.cc idl: allow writers to use any output stream Dec 22, 2016
frozen_mutation.hh frozen_mutation: avoid buffer linearization and copy Aug 22, 2016
frozen_schema.cc idl: allow writers to use any output stream Dec 22, 2016
frozen_schema.hh remove db/serializer.hh includes Mar 2, 2016
gc_clock.cc Fix pre-ScyllaDB copyright statements Apr 8, 2016
gc_clock.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
hashing.hh Introduce hashing helpers Jan 8, 2016
hashing_partition_visitor.hh mutations: Row tombstones are now a set of ranges Jun 2, 2016
idl-compiler.py idl: allow writers to use any output stream Dec 22, 2016
init.cc init: move supervisor_notify() out of main.cc Jan 6, 2017
init.hh init: move supervisor_notify() out of main.cc Jan 6, 2017
intrusive_set_external_comparator.hh intrusive_set_external_comparator: avoid using boost::intrusive::valu… Jan 10, 2017
json.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
keys.cc partition_key_view: Implement operator<< Sep 30, 2016
keys.hh keys: add memory_usage() Nov 18, 2016
log.hh Merge seastar upstream Jun 1, 2016
main.cc Merge "avoid excessive memory usage during resharding" from Rapahel Jan 9, 2017
map_difference.hh map_difference: Allow on unordered_map Apr 20, 2016
md5_hasher.hh md5_hasher: add finalize_array() Mar 11, 2016
memtable.cc Get rid of query::partition_range Dec 19, 2016
memtable.hh Get rid of query::partition_range Dec 19, 2016
mutation.cc mutation_partition: take schema in find_row and clustered_row Jan 5, 2017
mutation.hh Replace iostream include with iosfwd in headers Jan 17, 2017
mutation_compactor.hh mutation_compactor: honour stop_iteration from consumers Dec 14, 2016
mutation_partition.cc Implement intrusive set using rbtree_algorithms Jan 5, 2017
mutation_partition.hh Replace iostream include with iosfwd in headers Jan 17, 2017
mutation_partition_applier.hh mutations: Row tombstones are now a set of ranges Jun 2, 2016
mutation_partition_serializer.cc idl: allow writers to use any output stream Dec 22, 2016
mutation_partition_serializer.hh idl: allow writers to use any output stream Dec 22, 2016
mutation_partition_view.cc mutation_partition_view: Avoid unnecessary copy into temporary Jun 14, 2016
mutation_partition_view.hh idl: switch to utils::input_stream Aug 22, 2016
mutation_partition_visitor.hh mutations: Row tombstones are now a set of ranges Jun 2, 2016
mutation_query.cc mutation_query: to_data_query_result enforces row limit Dec 15, 2016
mutation_query.hh Get rid of query::partition_range Dec 19, 2016
mutation_reader.cc Convert to use dht::partition_range_vector and dht::token_range_vector Dec 19, 2016
mutation_reader.hh db: Use incremental selector in partition_presence_checker Dec 19, 2016
noexcept_traits.hh Introduce noexcept_traits Dec 7, 2015
nway_merger.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
partition_builder.hh mutations: Row tombstones are now a set of ranges Jun 2, 2016
partition_range_compat.hh Convert to use dht::partition_range_vector and dht::token_range_vector Dec 19, 2016
partition_slice_builder.cc Merge "Fix query digest mismatch" from Tomasz Apr 8, 2016
partition_slice_builder.hh Merge "Fix query digest mismatch" from Tomasz Apr 8, 2016
partition_version.cc partition_version: Fix corruption of partition_version list Oct 18, 2016
partition_version.hh partition_version: fix const correctness in rows_entry_compare Nov 28, 2016
query-request.hh Get rid of query::partition_range Dec 19, 2016
query-result-reader.hh storage_proxy: avoid calculating digest when only one replica is cont… Nov 17, 2016
query-result-set.cc query: use result_view::consume() where appropriate Aug 22, 2016
query-result-set.hh Merge "Fix query digest mismatch" from Tomasz Apr 8, 2016
query-result-writer.hh idl: allow writers to use any output stream Dec 22, 2016
query-result.hh result_memory_tracker: fix too-short short reads Jan 5, 2017
query.cc query: make result_memory_limiter constants available for linker Dec 22, 2016
query_result_merger.hh query_result_merger: Limit rows Dec 15, 2016
range.hh Replace iostream include with iosfwd in headers Jan 17, 2017
range_tombstone.cc Clean up position_in_partition. Oct 25, 2016
range_tombstone.hh streamed_mutation: add memory_usage() to mutation fragment types Nov 18, 2016
range_tombstone_list.cc range_tombstone_list: use non-template lambda for cloning tombstones Jun 27, 2016
range_tombstone_list.hh range_tombstone_list: add non-const begin() and end() Jul 13, 2016
range_tombstone_to_prefix_tombstone_converter.hh range_tombstone: check for adjacent instead of equal bounds Jun 20, 2016
release.cc Fix pre-ScyllaDB copyright statements Apr 8, 2016
release.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
reversibly_mergeable.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
row_cache.cc row_cache: Fix stats handling for uncached wide partitions Jan 18, 2017
row_cache.hh row_cache: Add counter for wide partition mispopulations Jan 18, 2017
schema.cc cell_comparator: Better fix (i.e. potentially correct) for compound/c… Jan 17, 2017
schema.hh alter_table_statement: Update materialized view Dec 20, 2016
schema_builder.hh schema: Add view_info field Dec 20, 2016
schema_mutations.cc frozen_schema: Support view schemas Dec 20, 2016
schema_mutations.hh frozen_schema: Support view schemas Dec 20, 2016
schema_registry.cc schema_registry: Fix possible hang in maybe_sync() if syncer doesn't … May 11, 2016
schema_registry.hh schema_registry: Track synced state of schema Jan 11, 2016
scylla-blocktune blocktune: fix syntax error in exception handling Oct 23, 2016
scylla-gdb.py scylla-gdb: Fix lookup of symbols in 'scylla ptr' Dec 1, 2016
scylla-housekeeping scylla-housekeeping: Create a uuid file if one is missing Jan 8, 2017
serialization_visitors.hh idl: allow writers to use any output stream Dec 22, 2016
serializer.hh Merge seastar upstream Sep 28, 2016
serializer_impl.hh serializer_impl: add serializer for bool_class<Tag> Dec 14, 2016
sstable_mutation_readers.hh Get rid of query::partition_range Dec 19, 2016
stdx.hh Introduce stdx.hh header file Aug 17, 2016
streamed_mutation.cc streamed_mutation: Fix memory corruption when reader constructor throws Jan 16, 2017
streamed_mutation.hh position_in_partition: Add tri_comparator Dec 28, 2016
supervisor.cc init: move supervisor_notify() out of main.cc Jan 6, 2017
supervisor.hh init: move supervisor_notify() out of main.cc Jan 6, 2017
test.py test.py: remove '.cc' from view_schema_test Dec 20, 2016
timestamp.hh db: Use microsecond precision for server-side timestamps Dec 21, 2016
to_string.hh Fix pre-ScyllaDB copyright statements Apr 8, 2016
tombstone.hh db: Avoiding checking bloom filters during compaction Jul 10, 2016
types.cc cql3: TIME data type support Jan 9, 2017
types.hh Replace iostream include with iosfwd in headers Jan 17, 2017
unimplemented.cc unimplemented: Add materialized views Oct 18, 2016
unimplemented.hh Replace iostream include with iosfwd in headers Jan 17, 2017
validation.cc validation: Add KS validation + convinence methods Apr 19, 2016
validation.hh validation: Add KS validation + convinence methods Apr 19, 2016
version.hh version: Bump Cassandra version to 2.2.8 Jan 9, 2017

README.md

Scylla

Building Scylla

In addition to required packages by Seastar, the following packages are required by Scylla.

Submodules

Scylla uses submodules, so make sure you pull the submodules first by doing:

git submodule init
git submodule update --init --recursive

Building and Running Scylla on Fedora

  • Installing required packages:
sudo dnf install yaml-cpp-devel lz4-devel zlib-devel snappy-devel jsoncpp-devel thrift-devel antlr3-tool antlr3-C++-devel libasan libubsan gcc-c++ gnutls-devel ninja-build ragel libaio-devel cryptopp-devel xfsprogs-devel numactl-devel hwloc-devel libpciaccess-devel libxml2-devel python3-pyparsing lksctp-tools-devel protobuf-devel protobuf-compiler systemd-devel libunwind-devel
  • Build Scylla
./configure.py --mode=release --with=scylla --disable-xen
ninja-build build/release/scylla -j2 # you can use more cpus if you have tons of RAM

  • Run Scylla
./build/release/scylla

  • run Scylla with one CPU and ./tmp as data directory
./build/release/scylla --datadir tmp --commitlog-directory tmp --smp 1
  • For more run options:
./build/release/scylla --help

Building Fedora RPM

As a pre-requisite, you need to install Mock on your machine:

# Install mock:
sudo yum install mock

# Add user to the "mock" group:
usermod -a -G mock $USER && newgrp mock

Then, to build an RPM, run:

./dist/redhat/build_rpm.sh

The built RPM is stored in /var/lib/mock/<configuration>/result directory. For example, on Fedora 21 mock reports the following:

INFO: Done(scylla-server-0.00-1.fc21.src.rpm) Config(default) 20 minutes 7 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-21-x86_64/result

Building Fedora-based Docker image

Build a Docker image with:

cd dist/docker
docker build -t <image-name> .

Run the image with:

docker run -p $(hostname -i):9042:9042 -i -t <image name>

Contributing to Scylla

Guidelines for contributing