RELEASE Scylla 1.0 RC2

The Scylla team is pleased to announce the release of Scylla 1.0 RC2, a Release Candidate toward Scylla 1.0

From now own, and for some time now, only critical bugs (showstoppers) will be fix in branch-1.0. We will continue to fix bugs and add feature on the master branch.

Scylla is an open source, Apache-Cassandra-compatible NoSQL database, with superior performance and consistent low latency.

Noteworthy updates are:

• Developer Mode update

  Developer mode is a way to run reduce Scylla, omitting performance enhancement features. You should normally not run in developer mode, unless your environment does not allow you to run Scylla otherwise. For example, you must use developer mode to run Scylla without XFS or without setting IO queues setting (see below). Running in developer mode will reduce performance and it is not recommended for production.

  To run Scylla in developer mode, you should now have a file /etc/scylla.d/dev-mode.conf with the line

DEV_MODE="--developer-mode=1

• Tune IO queues

  You now must run scylla_io_setup as part of a new Scylla installation. scylla_io_setup runs a short (up to a few minutes) benchmark on your storage and generate /etc/scylla.d/io.conf file for Scylla to use. Once the file is ready, you can start Scylla. There is no need to run scylla-io-setup again, you change your running environment - such as disks and CPU placement. Scylla will not run without io.conf file, unless using developer mode. There is no need to run scylla_io_setup on recommended instance types when using Scylla AMI.

• Controlled Repair

  Scylla now have more control over the resources that repair uses, when competing with serving ordinary requests. Repair's writes now go to separate memtables and bypass the commitlog to allow fair resource allocation vs. ordinary write requests. Also, repair's parallelism is limited to avoid it monopolizing the CPU.

• tcp keepalive

  Keep connection alive in networking environments that have connection tracking.

• Many bug fix

  In particular, handling node failures during repair, failures during shutdown and boot, Ubuntu packaging issues and more. Full list of contributions below.

Get started with Scylla 1.0 RC2 here. Please let us know[1] if you encounter any problem.

Regards Tzach

[1] https://github.com/scylladb/scylla/issues

Scylla Contributions

Asias He (33):
      streaming: Hook streaming with gossip callback
      messaging_service: Stop retrying if node is removed from gossip
      streaming: Remove the unused test code
      db: Introduce column_family_exists
      streaming: Do not send if the cf is deleted
      streaming: Handle cf is deleted when sending STREAM_MUTATION_DONE
      storage_service: Fix effective_ownership
      storage_service: Do not stop messaging_service more than once
      gossip: Do not stop gossip more than once
      storage_service: Make decommission log info instead of debug level
      streaming: Fix a log level in get_column_family_stores
      gossip: Log status info when stopping gossip
      streaming: Reduce STREAM_MUTATION error logging
      messaging_service: Add missed throw
      streaming: Fix rethrow in stream_transfer_task
      storage_service: Fix pending_range_calculator_service
      migration_manager: Make the migration callbacks runs inside seastar thread
      storage_service: Update pending ranges when keyspace is changed
      gossip: Introdcue handle_ack2_msg helper
      gossip: Introduce handle_echo_msg helper
      gossip: Introduce handle_shutdown_msg helper
      gossip: Ignore ack2 message if gosisp is not enabled yet
      gossip: Add comments on external runtime dependency needed by gossip
      storage_service: Drop unused debug code
      storage_service: Move REPLICATION_FINISHED verb to storage_service
      main: Defer initalization of streaming
      storage_service: Drop block_until_update_pending_ranges_finished
      tests: Fix stop of storage_service in cql_test_env
      streaming: Handle cf is deleted after the deletion check
      gossip: Handle unknown application_state when printing
      messaging_service: Take reference of ms in send_message_timeout_and_retry
      gossip: Sync gossip_digest.idl.hh and application_state.hh
      streaming: Complete receive task after the flush

Avi Kivity (5):
      Merge seastar upstream
      Merge seastar upstream
      Merge seastar upstream
      build: prepare for 1.0 release series
      dist: ami: fix AMI_OPT receiving no value

Benoît Canet (1):
      dist/ubuntu: Fix the init script variable sourcing

Calle Wilund (3):
      lists.cc: fix update insert of frozen list
      sstables: Add delete func to rename TOC ensuring table is marked dead
      database: Use disk-marking delete function in discard_sstables

Glauber Costa (20):
      repair: replace a magic number with another magic number
      touch CF directories during startup
      colum_family: remove mutation_count
      column_family: do not open code generation calculation
      database: remove unused parameter
      main: when scanning SSTables, run shard 0 first
      sstables: improve error messages
      database: turn sstable generation number into an optional
      sstables: do not assume mutation_reader will be kept alive
      stream_session: remove duplicated debug message
      stream_session: print debug message for STREAM_MUTATION
      database: create a class for memtable_list
      database: move active_memtable to memtable_list
      database: move add_memtable as a method of the memtable_list
      database: make seal_on_overflow a method of the memtable_list
      priority manager: separate streaming reads from writes
      streaming: add incoming streaming mutations to a different sstable
      streaming memtables: coalesce incoming writes
      database: keep streaming memtables in their own region group
      repair: rework repair code so we can limit parallelism

Gleb Natapov (15):
      log: add space between log level and date in the outpu
      Take pending endpoints into account while checking for sufficient live nodes
      fix EACH_QUORUM handling during bootstrapping
      storage_proxy: always wait for digest read resolver done future
      storage_proxy: send read requests with timeout
      storage_proxy: fix background_reads counter
      storage_proxy: add counter for retries reads
      cleanup forward declaration for key types
      fix developer-mode parameter application on SMP
      make initialization run in a thread
      messaging: enable keepalive tcp option for inter-node communication
      main: Defer storage proxy RPC verb registration after commitlog replay
      enable rpc_keepalive parameter
      messaging: do not delete client during messaging service shutdown
      messaging: do not admit new requests during messaging service shutdown.

Nadav Har'El (6):
      sstables: avoid index and data file over-reads
      sstables: more efficient read of compressed data file
      sstable: fix compressed data file overread
      Fix out-of-range exception when uncompressing 0 bytes
      Allow uncompression at end of file
      sstable: fix use-after-free of temporary ioclass copy

Paweł Dziepak (13):
      lsa: set _active to nullptr in region destructor
      storage_proxy: fix reconciliation with limits
      mutation_partition: do not add ck to result when not asked to
      types: make collection_type_impl::deserialize_mutation_form static
      atomic_cell: do not require type to hash collection cell
      result.idl: keep digest together with result
      md5_hasher: add finalize_array()
      query: add keys and tombstones to result digest
      query: build only result, only digest or both
      specify whether query::result, result_digest or both are needed
      query: add query::result::_digest to pretty printer
      move SCHEMA_CHECK registration to migration_manager
      lsa: update _closed_occupancy after freeing all segments

Pekka Enberg (21):
      Update scylla-ami submodule
      types: Implement to_string for timestamps and dates
      configure: Add boost date_time library as a dependency
      configure: Remove 'scylla_libs' variable
      main: Initialize system keyspace earlier
      build: Fix http/http_response_parser.hh dependency
      message/messaging_service: Remove init_messaging_service() declaration
      main: Defer migration manager RPC verb registration after commitlog replay
      main: Defer REPAIR_CHECKSUM_RANGE RPC verb registration after commitlog replay
      tests/gossip_test: Fix messaging service stop
      dist: Fix '--developer-mode' parsing in scylla_io_setup
      main: Fix broadcast_address and listen_address validation errors
      main: Defer API server hooks until commitlog replay
      Revert "build: prepare for 1.0 release series"
      build: Invoke Seastar build only once
      dist/ubuntu: Use tilde for release candidate builds
      release: prepare for 1.0.rc1
      Merge seastar upstream
      Update scylla-ami submodule
      Merge seastar upstream
      release: prepare for 1.0.rc2

Raphael Carvalho (2):
      service: fix refresh
      Fix corner-case in refresh

Raphael S. Carvalho (7):
      fix storage_service::load_new_sstables() to not disable write permanently
      compaction_manager: update stat pending_tasks properly
      sstables: update comment
      service: improve logging of storage_service::load_new_sstables
      sstables: bail out if toc exists for generation used by write_components
      sstables: fix race condition when writing to the same sstable in parallel
      sstables: make write_simple() safer by using exclusive flag

Shlomi Livne (4):
      fix centos local ami creation (revert some changes)
      dist/ami: Use the actual number of disks instead of AWS meta service
      scylla_io_setup import scylla-server env args
      fix a collision betwen --ami command line param and env

Takuya ASADA (23):
      dist: add /etc/scylla.d/io.conf on Ubuntu
      Revert "Revert "Revert "dist: remove AMI entry from sysconfig, since there is no script refering it"""
      Revert "Revert "dist: align ami option with others (-a --> --ami)""
      dist: export sysconfig for scylla-io-setup.service
      dist: export all entries on /etc/default/scylla-server on Ubuntu
      dist: show message to use XFS for scylla data directory and also notify about developer mode, when iotune fails
      dist: use SEASTAR_IO instead of SCYLLA_IO
      dist: do not auto-start scylla-server job on Ubuntu package install time
      dist: On scylla_sysconfig_setup, don't disable cpu0 on non-AMI environments
      dist: follow sysconfig setting when counting number of cpus on scylla_io_setup
      dist: enable collectd on scylla_setup by default, to make scyllatop usable
      dist: introduce dev-mode.conf and scylla_dev_mode_setup
      dist: on scylla_io_setup print out message both for stdout and syslog
      dist: remove scylla-io-setup.service and make it standalone script
      dist: on scylla_io_setup, SMP and CPUSET should be empty when the parameter not present
      dist: Ubuntu based AMI support
      dist: prevent to show up dialog on apt-get in scylla_raid_setup
      dist: fix broken scylla_install_pkg --local-pkg and --unstable on Ubuntu
      dist: launch scylla_ami_setup on Ubuntu AMI
      dist: allow to run 'sudo scylla_ami_setup' for Ubuntu AMI
      dist: stop using '-p' option on lsblk since Ubuntu doesn't supported it
      dist: allow more requests for i2 instances
      dist/ami: use tilde for release candidate builds

Tomasz Grabiec (34):
      log: Fix operator<<(std::ostream&, const std::exception_ptr&)
      mutation_partition: Unmark cell_entry's copy constructor as noexcept
      mutation_partition: Make row::vector_to_set() exception-safe
      mutation_partition: Add cell_entry constructor which makes an empty cell
      mutation: Add copy assignment operator
      managed_bytes: Make linearization_context::forget() noexcept
      managed_bytes: Make copy assignment exception-safe
      managed_bytes: Mark move-assignment noexcept
      mutation_partition: Make row::merge() tolerate empty row
      mutation_partition: row: Allow storing empty cells internally
      mutation_partition: row: Add empty()
      Introduce the concept of ReversiblyMergeable
      tombstone: Make ReversiblyMergeable
      atomic_cell: Add REVERT flag
      atomic_cell_hash: Specialize appending_hash<> for atomic_cell and collection_mutation
      atomic_cell_or_collection: Introduce as_atomic_cell_ref()
      mutation_partition: Make row ReversiblyMergeable
      mutation_partition: Make row_marker ReversiblyMergeable
      mutation_partition: Make rows_entry ReversiblyMergeable
      mutation_partition: Make row_tombstones_entry ReversiblyMergeable
      mutation_partition: Make intrusive sets ReversiblyMergeable
      mutation_partition: Make apply() atomic even in case of exception
      mutation_partition: Fix friend declarations
      mutation_test: Fix use-after-free
      mutation_test: Make make_blob() return different blob each time
      mutation_test: Hoist make_blob() to a function
      mutation_test: Add more apply() tests
      mutation_test: Add allocation failure stress test for apply()
      perf_simple_query: Make duration configurable
      managed_bytes: Make operator[] work for large blobs as well
      logalloc: Rename tracker::occupancy() to region_occupancy()
      logalloc: Introduce tracker::occupancy()
      Fix assertion in row_cache_alloc_stress
      cql3: batch_statement: Execute statements sequentially

Vlad Zolotarov (6):
      gossiper: use a semaphore instead of a future<> for serializing a timer callback
      gossiper: make _shadow_endpoint_state_map public and rename
      storage_service: distribute gossiper::endpoint_state_map together with token_metadata
      database: store "incremental backup" configuration value in per-shard instance
      api::set_storage_service(): fix the 'nodetool enablebackup' API
      sstables: properly account removal requests

Yoav Kleinberger (1):
      tools/scyllatop: defend against curses "out of screen bounds" error

Scylla JMX Contributions

Amnon Heiman (4):
      MessagingService: Ignore exception on the dropped messages thread
      StorageService: Sort the results of getTokenToEndpointMap
      ProgressInfo: Add creation from json object and json array
      SessionInfo: Add receiving_files and sending_files support

Nadav Har'El (1):
      scylla-jmx: implement forceRepairRangeAsync