Skip to content
The high-performance distributed SQL database for global, internet-scale apps.
C C++ Java PLpgSQL TSQL JavaScript Other
Branch: master
Clone or download
spolitov [#2571] [#2895] Fix MVCC concurrency issue and other issues found in …
…CppCassandraDriverTest.BatchWriteDuringSoftMemoryLimit

Summary:
When an operation is received by a follower, it is added to MvccManager during the "prepare" phase.
But this preparing phase could be delayed under high load, so it happens after this operation is Raft-replicated.
So the following scenario is possible:
1) The follower receives an operation from the leader.
2) Follower wins an election.
3) It becomes the leader and replicates operations from the old leader with a newly added no-op.
4) Operations from the old leader are being prepared.

After (3) we already know that this follower is a fully functional leader, so it could return safe time equal to hybrid time lease expiration.
And (4) happens after that, so we will try to register (in MvccManager) operations with hybrid times that are lower than last returned safe time, leading to a check failure and a crash.

Fixed by changing the place where we register follower operations in MVCC, i.e. StartReplicaOperation.

Also fixed the following issues found by the same test:
1) `RemoteTablet::GetRemoteTabletServers` could try to lock a mutex while the wait is not allowed.
   Fixed by allowing reading stale leader state at this point.
2) In CppCassandraDriverTest.BatchWriteDuringSoftMemoryLimit, the external mini-cluster could produce more than 50MB of logs.
   Increased limit to 512MB.
3) Moved partition check from `AsyncRpc` to `Batcher`, so failure would happen earlier.
4) Removed `DCHECK` for the number of `scheduled_tasks_` to be equal to zero from ServicePool. There could be a race condition when the aborted task was not yet completely aborted, while the new scheduled task is complete. So `check_timeout_task_` will be reset to `kUninitializedScheduledTaskId`, but `scheduled_tasks_` would still be non-zero.
5) Reset of `leader_no_op_committed_` before marking current node as leader. Otherwise, for a very short period of time, we will have an invalid value of `leader_no_op_committed_` from previous leadership.
6) Fix race condition in `Batcher` when `FlushBuffersIfReady` is performed twice. Disallow `FlushBuffersIfReady` in `kTransactionReady` state, and introduce new function `ExecuteOperations` to complete `Batcher` after transaction is ready.

Test Plan: ybd asan --cxx-test cassandra_cpp_driver-test --gtest_filter CppCassandraDriverTest.BatchWriteDuringSoftMemoryLimit -n 20

Reviewers: bogdan, timur, mikhail

Reviewed By: mikhail

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D7560
Latest commit 401e821 Nov 16, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Don't try to build initial system catalog snapshot in the CircleCI build Jun 17, 2019
arcanist_util Add Apache license Sep 7, 2017
architecture Update docdb-encryption-at-rest.md Sep 19, 2019
bin Fix yugaybted import to be python3 compatible Nov 15, 2019
build-support YW build errors should fail YB builds Nov 15, 2019
cloud [#2782] Add a loadbalancer to k8s statefulset.yaml files Nov 11, 2019
cmake_modules [#2608] Add GCC8 build to thirdparty Oct 19, 2019
community/learn Add lesson markdown for learn.yugabyte.com tutorials Sep 1, 2019
docs Automatic download of prebuilt third-party dependencies and Homebrew/… Nov 15, 2019
ent [#2154] CDC Producer Metrics Nov 16, 2019
java [#1237] [YSQL] Transparent read restarts for general queries Nov 15, 2019
licenses Adding POLYFORM license Jul 10, 2019
managed [#2952] Add the missing close client call in SetUniverseKey Nov 16, 2019
python/yb Automatic download of prebuilt third-party dependencies and Homebrew/… Nov 15, 2019
sample #1936: Add chinook sample db to sample directory Jul 30, 2019
src [#2571] [#2895] Fix MVCC concurrency issue and other issues found in … Nov 17, 2019
submodules Automatic download of prebuilt third-party dependencies and Homebrew/… Nov 15, 2019
thirdparty Automatic download of prebuilt third-party dependencies and Homebrew/… Nov 15, 2019
www Some simple visual fixes for YX Aug 13, 2019
.arcconfig Change arcconfig to point to the Phabricator callsign pointing to htt… Jul 16, 2019
.arclint [#2081] [CDC] Setup Java Console Connector for CDC Oct 8, 2019
.clang-format use the google style for the precommit hook Sep 30, 2016
.gitignore Fix build in Clang 9 and add Flex/Bison generated code to compilecmds… Oct 8, 2019
.gitmodules Adding submodules: yugabyte-installation and yugabyte-bash-common Apr 5, 2019
.ycm_extra_conf.py A macro-based DSL for C-to-C++ APIs for PostgreSQL integration Jul 14, 2018
CMakeLists.txt [#2608] Add GCC 8 build type Oct 22, 2019
CONTRIBUTORS.md contributors: add hasheddan (#2653) Oct 19, 2019
Doxyfile Removing some irrelevant files Sep 8, 2017
LICENSE.md Update and rename LICENSE.txt to LICENSE.md Jul 10, 2019
NOTICE.txt Detect the edition based on the Jenkins job name, and add NOTICE.txt … Dec 30, 2017
README.md Update README.md Oct 14, 2019
code_style.xml Ensure our code style enables multiline align to be consistent with g… Jan 25, 2017
python_requirements.txt [#939] [#940] [#941] [#942] Cleanup of stray processes, NFS issues wo… Mar 7, 2019
python_requirements_frozen.txt [#939] [#940] [#941] [#942] Cleanup of stray processes, NFS issues wo… Mar 7, 2019
version.txt [#1886] Adapt release code to work with the new tar.gz format without… Jul 25, 2019
yb_build.sh Automatic download of prebuilt third-party dependencies and Homebrew/… Nov 15, 2019
yb_release Properly deactivate old virtualenv; use the shared third-party direct… May 16, 2018
yb_release_manifest.json Rename yugabytedb to yugabyted Nov 1, 2019

README.md

YugabyteDB


License Documentation Status Ask in forum Slack chat Analytics

What is YugabyteDB?

YugabyteDB is a high-performance, cloud-native distributed SQL database. Here are its salient points:

  • Has a pluggable query layer, and supports two distributed SQL APIs:
  • Automated sharding, Raft consensus replication and distributed transactions, based on the Google Spanner architecture
  • Offers horizontal write scalability, strong write consistency and tunable read consistency (strong reads by default with follower & observer reads as options)
  • Extremely resilient with native failover and repair - can tolerate disk, node, zone and region failures automatically
  • Supports geo-distributed deployments (multi-zone, multi-region, multi-cloud)
  • Built-in enterprise features such as distributed backups, in-flight/at-rest encryption and read replicas (for observer reads)
  • Can be deployed in public clouds and natively inside Kubernetes
  • Best fit for powering massively-scalable, globally-distributed, cloud-native applications that require absolute data correctness and high tolerance to failures
  • 100% open source under the Apache 2.0 license

Read more about YugabyteDB in our Docs.

Get Started

Cannot find what you are looking for? Have a question? Please post your questions or comments on our Community Slack or Forum.

Build Apps

YugabyteDB supports a number of languages and client drivers. Below is a brief list.

Language ORM YSQL Drivers YCQL Drivers
Java Spring/Hibernate PostgreSQL JDBC cassandra-driver-core-yb
Go Gorm pq gocql
NodeJS Sequelize pg cassandra-driver
Python SQLAlchemy psycopg2 yb-cassandra-driver
Ruby ActiveRecord pg yugabyte-ycql-driver
C# EntityFramework npgsql CassandraCSharpDriver
C++ Not tested libpqxx cassandra-cpp-driver
C Not tested libpq Not tested

Architecture

YugabyteDB Architecture

Review detailed architecture in our Docs.

Need Help?

Contribute

As an open source project with a strong focus on the user community, we welcome contributions as GitHub pull requests. See our Contributor Guides to get going. Discussions and RFCs for features happen on the design discussions section of our Forum.

License

Source code in this repository is variously licensed under the Apache License 2.0 and the Polyform Free Trial License 1.0.0. A copy of each license can be found in the licenses directory.

The build produces two sets of binaries:

  • The entire database with all its features (including the enterprise ones) are licensed under the Apache License 2.0
  • The binaries that contain -managed in the artifact and help run a managed service are licensed under the Polyform Free Trial License 1.0.0.

By default, the build options generate only the Apache License 2.0 binaries.

Read More

You can’t perform that action at this time.