Skip to content

@rkarthik007 rkarthik007 released this Apr 17, 2019 · 135 commits to master since this release

NOTE: This page has moved to this location after this release. Please go there to find the latest release.

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.2.4. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.4 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.4, create a new cluster and then load the previously saved data.

YSQL Changes

  • Added support for primary keys. It is now possible to create a table with a primary key. #955
  • Added support for temporary tables. #805
  • Added support for CYCLE option in CREATE SEQUENCE statement. #993
  • Enabled SET TRANSACTION READ WRITE statement. #1081
  • Support RelationType in TableInfo to distinguish between system tables, user tables and user indexes. #906

Bug Fixes

  • Fixed a crash in tserver that is observed while inserting null value into an indexed column. #945
  • Fixed a bug where indexes on system catalog tables were not being updated during DDL. #1083
  • Fixed detection of single row transactions on tables that have primary keys.
  • Fixed MiniMaster startup race condition in test infrastructure. #1040

System Improvements

  • Reduce per-connection overhead for cache preloading. #1090 #1091
  • Faster release of sending buffers to reduce memory consumption when leader sends large updates to followers who are lagging behind. #1079
  • Parallelize DNS resolution while filling system.peers and system.local tables #1000
  • Script for post-processing test results and generating JSON-based result files.
  • Improve stability of QLTransactionTest.IntentsCleanupAfterRestart. #1102

Key Commits

  • [f2ecaa1] [YSQL] Fix single-row-txn detection after primary key change
  • [90680ca] [#1102]: Improve stability of QLTransactionTest.IntentsCleanupAfterRestart
  • [a4c614f] JSON-based test reports
  • [33835b0] [#1079]: Release sending buffers as soon as possible
  • [1bc77da] ENG-4682 (#906): Support RelationType (user table, index, system table) in TableInfo
  • [1dedcd1] ENG-4943 #945 [YSQL] Correct handling for NULL value in indexed column
  • [465b6d6] [YSQL] #1090 #1091 Reduce per-connection overhead for cache preloading
  • [382b89c] [YSQL] ENG-4601 (#805): Add support for temporary tables
  • [5165f01] #1081 [YSQL] Enable READ WRITE transaction
  • [9f31c41] ENG-4927: #1040: Fix MiniMaster startup race condition
  • [be0908b] [YSQL] #955: Handle primary key as an index
  • [4b48e25] ENG-4872: #1000: Parallel DNS resolution while filling system.peers and system.local tables
  • [4096f59] ENG-4900: Add external mini cluster test with TLS support
  • [34ee0fc] [YSQL] #1083: Handle sys catalog index updates
  • [ac52116] #993: [YSQL] Add support for CYCLE option in CREATE SEQUENCE
  • [3149f36] ENG-4363, #668: Support JSON operator on INSERT (Part 1 - Refactoring)
  • [a3ea483]
  • [b3a60ee] [yb-docker-ctl] Save initdb log (#1087)
  • [b131fbc] ENG-4888: #1031: Make YCQL index name optional
  • [9d333f7] #1064: Fix data race with YQL permissions cache
  • [4a56fe1] #1038: Change FLAGS_remote_boostrap_rate_limit_bytes_per_sec to FLAGS_remote_bootstrap_rate_limit_bytes_per_sec
  • [56a6029] #752: Add TLS encryption support to yb-admin.
  • [d9b6385] [YSQL] #1066: Fix memory leak in ResourceOwner
  • [fb4c1a2] Added yb-docker-ctl to CE (#1077)
  • [046b4b6] ENG-4855: #977: Handle restart read in YSQL
  • [30725bf] ENG-4343: #655: Pick transaction read time at tablet server when possible
  • [0341b29] [YSQL] #1067: Buffer create index write operations only for initdb
  • [78ca75a] [#1050]: Add Tablet Server capabilities
  • [9d724d5] ENG-4790: Removing most of yb-loadtester from repo due to migration to standalone yb-sample-apps repo
  • [bbcc67b] ENG-4891: Removing the SQL RPCs button.
  • [14279cc] [YCQL] #1042: Fix ASAN issue in YCQL executor restart
  • [cc3584f] ENG-4800: #952: Use call buffer for receive from socket
  • [8715be3] [YSQL] #1051: Bulk insert in initdb
  • [987abc6] ENG-4877 (#1009): yb-ctl enhancements
  • [0f261a9] [YSQL] 1049 ASAN reported memory leaks in Postgres
  • [9eaf15b] Add a new pattern for NFS directories: /Volumes/net
  • [3a61698] [#1041]: Skip dumping of finished call
  • [a322d95] [YSQL] #711 Add support for all datatypes
  • [7e2c89c] [#1030]: Add transactional retry writes tests 2) Second
  • [request]licated. Fixed by returning AlreadyPresent status and handling it in
  • [callbac] not handled correctly. For instance when request cannot be processed because of service shutdown, we could stop retrying to apply the operation. Fixed by correct handling of such errors.
  • [509c5a2] #921: Fixed: [YCQL] improve error messages related to index DDL
  • [e48b7da] ENG-4468: Always perform only one server rpc call for readiness check.
  • [0f8fdd9] [YSQL] ENG-4831 (#969): Enable drop cascade
  • [85836b5] Allow spaces around equality sign in @RunWith(value=SomeClass) annotation
  • [e533efc] ENG-4844, #949: Treat TUPLE as a keyword
  • [40dd31f] [#984]: Allow running C++ test by specifying only gtest_filter
  • [9cb4f10] ENG-4746: #925: Implement change_blacklist & get_universe_config commands in yb-admin
  • [9e28eee] [YSQL] #995: Raise error on unsupported WHERE condition with secondary index
  • [9fd097e] [#965] Reduce the number of concurrent connections to YB masters and tservers from PostgreSQL backend processes
  • [7ea3cd0] [YSQL] ENG-4839 (#976): Handle postgres session termination correctly in YB mode
  • [f55f1c0] #991: Fixed TestRpc.TestDisconnect
  • [aa8c4ec] #999: [YSQL] SELECTing directly from a sequence throws an error
  • [8d226ce] ENG-4852: Do not pass Datum data types to ybc_pgate
  • [f702d72] [YSQL] #979: Fix ORDER BY query using secondary index
  • [66a94f3] ENG-4840 (#987): Fix incorrect count for system tablet leader count
  • [c847562] #964: Change compaction and remote bootstrap rate defaults.
  • [5ed041c] #788 Optimize select statements where a clustering-key needs to be in a given range
  • [0c00885] [#933]: Leader lease revocation
  • [03b996d] ENG-4746: #925: Migrate missing java yb-cli functionality to yb-admin (Preparation)
Assets 2

Overview of Changes

Theme of this release: PostgreSQL, PostgreSQL, PostgreSQL! YSQL is a distributed PostgreSQL that can scale-out, tolerate failures and has no single point of failure.

Support has been added for a ton of Postgres functionality. The following is a list of some of the functionality:

DDL statements

  • CREATE, DROP and TRUNCATE tables

Data types

  • All primitive types including numeric types (integers and floats), text data types, byte arrays, date-time types
  • Support for UUID and SERIAL
  • Support for JSONB

DML statements

  • INSERT, UPDATE, SELECT and DELETE statements
  • JOIN, WHERE clauses, GROUP BY, ORDER BY, LIMIT, OFFSET and SEQUENCES

Transactions

  • ABORT, ROLLBACK, BEGIN, END, and COMMIT

Miscellaneous

  • Expressions: Built-in functions and operators
  • VIEWs
  • EXPLAIN PLAN
  • PREPARE-BIND-EXECUTE
  • JDBC Support

Key Commits

  • [aef1a1f] ENG-4683 (#874): Fix user table UI to exclude transactions and sequences tables
  • [c60921f] ENG-4681 [YSQL] Support INTERVAL for date/time datatypes
  • [4927d38] [YSQL] ENG-4807 (#957) Fix logs wrongly reporting that YugaByte transactions are not enabled
  • [4e69172] ENG-4780 (#930): Fix postgres server crash while creating index on existing table with data
  • [f004732] #972: ENG-4808: Fixed TabletInvoker::SelectTabletServer
  • [652bb5e] [#959]: Groups threads by stack trace in /threadz
  • [d33eaa9] #665: [YSQL] Enable support for sequences
  • [53a1331] ENG-4801 (#944): yb-ctl improvements for easier onboarding
  • [dce09da] ENG-4585: #796: Created test that covers case when thread pool is not yet ready for intents cleanup
  • [bb40833] ENG-4468: (#871) Add support for checking if tserver is ready for IO requests.
  • [e570c1c] [ENG-3928] [#967] Wait for NFS-based paths to exist in run_tests_on_spark.py
  • [3399b88] [YSQL] #255: Separate YCQL, YSQL and YEDIS namespaces
  • [064d366] ENG-4749: #887: Handle failure of delete at start of iteration in PgLibPqTest.SerializableColoring test
  • [88bb343] #961 Fix instantiation of round-robin policy
  • [1037c47] ENG-4577: Log Replay Count Bugfix and Test #782
  • [57ca114] ENG-4725: #932: Write RPC should take into account retryable_rpc_single_call_timeout_ms
  • [e9424ab] [YSQL] #943 Optimize single-row write transactions to avoid distributed txn path.
  • [c545fb5] ENG-4783: Misc fixes in shared lock manager
  • [45bc122] ENG-4783: Fixed LockBatch move ctor/assignment
  • [3db5d55] Cleanup of stray processes, NFS issues workaround, DXR integration, adjusting RPC timeout and using random PG ports in YSQL
  • [36431db] Update PostgreSQL code as of the following commit:
  • [3952394] [YSQL] ENG-4757: #902: Restart transaction when it was first executed operation and read restart is required
  • [d563e50] ENG-4577: Log Replay Count Bugfix and Test #782
  • [1ed9e95] ENG-3558: #920: Replace our own implementation of VarInt with BIGNUM from OpenSSL
  • [2f6dedc] [#836][ENG-4638] Fix KEYS to not Surface Deleted Keys
  • [7cf6ccd] [#918]: Add a pool of pre-created transaction records to avoid a Raft round-trip for transaction creation on the write path
  • [7b59cc8] #897: Fixed yb-tserver failure on SELECT COUNT(column) where column contains null value
  • [b0314eb] [YCQL] #916 Making STATIC an unreserved-keyword. Removing REFERENCES from keyword list.
  • [c886b86] [YSQL] #917: Support DROP INDEX
  • [81a80ab] [YCQL] (#878) : Add uuid() builtin function support.
  • [99cc736] ENG-4672: Provide more information (version,
  • [a101e74] ENG-4656 (#852): Enabled pg_depend
  • [c839f8c] Avoid using NFS-shared local Maven repository directory for running Java tests on macOS
  • [396ff88] [#914]: Split doc_operation into multiple files
  • [78776a5] [YCQL] #835 Allow OFFSET to be used as column names and identifiers
  • [362502d] ENG-4636 (#834): Make transaction failure messages more user friendly
  • [a746743] #904: ENG-4735: Fixed race between deleting obsolete files and creating new output SST file
  • [845cdb4] ENG-4717: #891: Not submitted write operation should block tablet peer shutdown
  • [1125d1e] #771 [YEDIS] Close connection on pub/sub limit
  • [ae1e30c] #768 Ping in subscribe mode should return a multi-bulk string
  • [dd97847] [YSQL] ENG-4734 Fix data race with ysql_catalog_version #903
  • [08c9bbf] ENG-4410 (#696): Retry resolving master addresses instead of erroring out immediately
  • [3ce7386] ENG-4680 (#873): Save postgres logs in tserver/logs directory
  • [1dd8e4b] ENG-4692 (#868): Enable access share mode for table lock
  • [63a0d61] ENG-4692 (#868): Enable support for read only transactions
  • [3779a86] #862: ENG-4728: Use libbacktrace only in debug builds and in single-threaded mode to avoid high memory consumption
  • [827ff02] [YSQL] Basic framework for cache consistency with multi-head DDLs #679
  • [299c718] Add pre- for leader election logging
  • [bfd2019] #895: Rate-limit the warning of unsupported YCQL consistency level.
  • [ff38980] Break the long test TestPgRegressPgTypes into two separate tests
  • [492c67c] Bump up master memory limit threshold for ASAN and TSAN
  • [abc6a2e] ENG-4680 (#873): Save postgres logs in tserver/logs directory
  • [ff9f36d] ENG-3648: #604: Preelections
  • [48411de] ENG-4713: Correctly handle openssl buffer overflow
  • [ec5cdf5] ENG-4522 (#769) : Support for ALTER TABLE command
  • [ce84a16] #821: Fix the race condition that causes the master to get stuck on aborting tasks.
  • [b7989a7] [ENG-4658] Work around for the "shuf: No such file or directory" error when trying to a pick worker host for remote compilation
  • [c87cc63] BUG: fixed memleak by destroying icu Timezone
  • [aefcb85] Using YugaByte BINARY internal type to store UUID
  • [99c7f6c] Fixed bugs when appending data of type CHAR(n) to tuple
  • [b75341a] [YSQL] ENG-4538, #711: DATE and TIME support
  • [95f0718] [#862] Upgrade to gperftools 2.7 to pick up tcmalloc bug fixes
  • [6444b19] [ENG-4666] [#862] Implement a global memory limit for the master and stop the master if exceeded
  • [edaa248] Improvements to yb_money, yb_numeric and yb_numeric_big tests
  • [5b73676] #798: Concurrent create-table calls can result in "Table Not Found"
  • [1305434] [YSQL] #865 Fix crash when copying heaptuple
  • [7f35bad] [#867]: Add deadline to DocDB lock manager
  • [3f939b4] ENG-4602: [YCQL] Add support for more timezone formats #806
  • [4e93a27] ENG-4646, #711: Enable support for MONEY datatype
  • [fb7333b] Add LLVM 6.0.1 back as an option in thirdparty
  • [61516d1] #843: Fix a bug in YBCInitializeRelCache that was causing the default values for tables to be loaded incorrectly
  • [e5edba5] ENG-4079: #673: Add serializable isolation support to YSQL API
  • [57ad672] ENG-4352: #662: Add more RPC related Metrics
  • [cf12af0] ENG-4605 (#813): Fix log warning for master replicas This
  • [0c8f53b] Move imported tests to yugabyte/java/yb-jedis-tests/src/test/java/redis/
  • [b9ac01b] [YQL] #807 Support for UUID Datatype
  • [8b86a63] (#641): Allow num_cpus to be explicitly passed for container environments; use std::thread::hardware_concurrency() as default
  • [b5464c9] ENG-4537: #711: Decimal/Numeric support
  • [fade5be] [YSQL] #715 Fixed "=" operator for char-based datatype.
  • [4066f1e] [YSQL] (#559) Support secondary index in DML
  • [8af2a9e] ENG-4617: Added yb_completed_tests folder to track finished Mac Jenkins tests
  • [a2850d7] ENG-3271: JSON secondary indexes on attributes (PART 2) #741
  • [41ef4e3] ENG-4614 (#820): UPSERT of JSONB column should not crash the server
  • [f9fc9b2] [YSQL] #711 Framework for DataType Support
  • [2aafd92] [ENG-4578] [#783] Create Memtable Arena Allocation Gflag
  • [e60e3ec] Import tests from YB-JEDIS into Yugabyte
  • [e1a073c] [#760] AtEOXact_Snapshot should set FirstXactSnapshot to false on exit in YugaByte-enabled mode
  • [676797a] [#839] Fix a bug in migration from old to new intent type set format
  • [823ff34] #822: ENG-4624: Do not retry at ql/executor layer on a timeout; timeout retries already happen at ybclient layer
  • [b56ee9e] #824: Only count votes from VOTERS in the active config for determining replicated OpIds
  • [a3fe830] [#828]: Disable transaction RPCs timeout in QLTransactionTest.ChangeLeader
  • [2b63f89] [YSQL] #714 Fix wrong syscatalog metadata after create table failure
  • [dba6d1f] ENG-4626: Improve TLS certificate verification failure reporting
  • [48b225c] ENG-4079: #673: Handle SELECT statements without a WHERE clause in a transaction with serializable isolation
  • [3aa3aff] [YSQL] (#761) Enable YugaByte distributed transactions in the PostgreSQL API by default
  • [1970e09] ENG-4619: During a partition, tservers can log ~250,000 "Leader Master has changed" messages per minute #823
  • [4ecd7dd] ENG-4593 (#826) Enable JSONB functions and add tests
  • [f68aaa8] [#825]: Decrease number of concurrent threads in SerializableTxnTest.Increment in sanitized mode
  • [0f8d64f] ENG-4581: #792: Disable compactions until tablet bootstrap is done
  • [0e53760] #750: ENG-4499: Changed status code names and phrasing to eliminate "SQL"
  • [ece81a5] #631: [YSQL] Enable not null constraints and check constraints
  • [b0c357b] ENG-4461: #723: Use consistent read snapshot for operations within a SQL statement
  • [ef8a651] #817: ENG-4595: Don't use consistent read point for single shard operations
  • [2712401] [YSQL] ENG-4572 #814 Clean up stray backends after tests.
  • [9d1e3a8] [#812]: Use yb::Thread in rocksdb ThreadPool
  • [c463104] ENG-4566: #795: Use DNS names when validating TLS certificate
  • [28d67dd] Splitting PgRegress testsuite to avoid TIMEOUT
  • [19fa40b] ENG-4496: Add hostname and username to YB diagnostic information (call-home)
  • [26d06db] ENG-743 (#793) : [YSQL] Support COPY statement
  • [11ed907] [#803]: Increase timing in RpcStubTest.TestDontHandleTimedOutCalls
  • [326501f] ENG-4351: #661: Show out of limit memory consumption in web interface
  • [268b438] [YSQL] #699 Enable Support for SET, SHOW, and RESET
  • [fab7fe4] ENG-4501 (#745): fix incorrect LOG line reporting flush/compaction stats
  • [55968b6] ENG-3271: JSON secondary indexes on attributes #741
  • [3f40371] ENG-4423 (#707) : added statement to logtrace for CQL and Redis
  • [fbb88ab] [#791]: Handle case when thread pool is not yet ready for intents cleanup
  • [e15bdb9] [YSQL] #712 Support COUNT(table.) and fix COUNT() for tables without primary key
  • [c8dc68c] ENG-4564: Added python@2 to brew install
  • [afc3cce] [#779]: Store postgres table metadata in all master replicas
  • [f8d1e03] [YSQL] #786 Raise error for indexing on column of unsupported type
  • [1aee234] [#784] Correctly set flushed frontier to the given value in a version edit
  • [ebedb36] #559: [YQL] Basic Postgres secondary index support
  • [c2651f8] [YSQL] #698 Support for CREATE TABLE AS SELECT
  • [766761e] (#710) [YSQL] support for pushdown of bind variables
  • [7457a46] [#781] Report elapsed time and timeout for synchronous leader master RPCs
  • [5860eb6] [#780] Flushed frontier should be allowed to go back when restoring a snapshot
  • [75045b6] ENG-4535: Fix for thirdparty libs build is failing on "include-what-you-use" for gcc builds
  • [66dd49b] [#773] [ENG-4531] Fix a bug with recovering flushed OpId and hybrid time on tablet server restart
  • [444e5dd] [#650]: Exit postgres when tserver is closed
  • [f9b8162] ENG-3501: Extended set of cassandra C++ driver test cases.
  • [41055f2] #766 Add a redis_version section to the redis INFO string
  • [8a30216] ENG-4479 #740 Use PostgreSQL process started by tablet server for Java-based PostgreSQL API tests
  • [70218be] [#767]: Don't use custom leader_failure_max_missed_heartbeat_periods in MasterFailoverTest
  • [d80be8f] Fix unstable TTL test TestRedisService.TestTtlPrimitive
  • [9ed0489] #632: [YSQL] Re-enable default values for columns in PostgreSQL tables
  • [a6a27e9] #700: A user must have the ALTER permission on a table to create an index on it
  • [cb025d6] ENG-3413: CreateTable should take into account blacklisted nodes #314
  • [718d6df] #577: Add leader status cache to avoid locking replica state
  • [64ad445] [#756]: Transaction with serializable isolation should not conflict with committed transaction
  • [34c576c] Enabling Clang static analyzer and add the include-what-you-use tool
  • [0fbc88e] Disable pg_wrapper-test in TSAN mode
  • [a0c220f] [#748]: Check if the leader is ready while updating propagated safe time
  • [3ff5e5e] Basic Tests Tracking Supported Features
  • [dc42ea0] #749 Persist co-located table metadata in tablet metadata
  • [69394aa] Cleanup PgStatement to return statement type via a virtual function
  • [13fa8cb] [#747] Fix TSAN issues in QLTransactionTest.WriteConflictsWithRestarts
  • [dc11643] Remove old unused cache and nvm_cache files and the nvml dependency
  • [cb5ea06] ENG-4350: #660: Don't replicate lease before replicating no op
  • [42eecc6] #746: Remove file block manager
  • [af08421] ENG-4079: #673: Add read intents for serializable isolation during write
  • [77bbc16] ENG-4340 #651 Reduce initdb log spew
  • [5835be4] #730 Fix flushed frontier going backwards during a compaction
  • [05659bc] Update boost source tarball checksum in thirdparty_src_checksums.txt
  • [72a5447] #704 Return a "Snapshot too old" error on attempts to read compacted-away data, e.g. during very long-running transactions
  • [9a60540] [YSQL] #658 RETURNING Support
  • [12a688d] ENG-4424: #708: Increase timeout waiting for transactions cleanup in tests
  • [77cbebd] ENG-4429: Fixed yb_release prefix handling for symlinks
  • [2007245] #709 Skip rebuilding PostgreSQL if it has not changed since previous build
  • [03c18ce] ENG-4383: Added bin/psql symlink into release
  • [5fe8101] [#706]: Treat not found transaction as aborted It does not check result of write operation, so it could try to
  • [3ed9810] [#705]: Hold mutex while updating obsolete_manifests_
  • [0f2a0c1] #653: Disable replica cache refresh by default.
  • [3d5122c] ENG-4079: #673: Replace intent type with intent type set
  • [986d014] [YSQL] #693 TRUNCATE Support
  • [8a4cdef] [#691]: Wait for tablet peer leadership in TabletServerTestBase
  • [f938535] [#690]: Increase allowed error for MacOS in TestMonoTime.TestCondition
  • [b395f07] [#689]: Use proxy_uuid in system.peers
  • [5d87393] [#688]: Use correct path for FSManager in TabletServerTest.TestRpcServerRPCFlag
  • [5398a99] [ENG-4172] [#569] Introduce Timeout to DocDB Range Scan Queries
  • [aad6e5e] ENG-4394: Fixed DBImpl::NotifyOnCompactionCompleted
  • [87ef37d] #687: Fixed QLTabletTest.GCLogWithRestartWithoutWrites/GCLogWithoutWrites TSAN issue
  • [250aaad] [ENG-3923] #336 Handle slow subscribers
  • [85d662d] #683: Pick better auto defaults for max_background_flushes
  • [d66fab0] Re-enable transactions in PostgreSQL
  • [c8a8f22] ENG-4027: Freezing Linuxbrew
  • [e989665] ENG-4353: #663: Release buffers when Socket is closed
  • [d18616c] ENG-4079: #673: Persist "read intents" for read operations done as part of serializable read-write transactions
  • [6fad9ea] ENG-4386: Postgres libs linking order is wrong
  • [9903b97] [#682]: Increase lifetime of Rpcs object used by heartbeater
  • [7aa6a27] #653: ENG-4341 - Refresh tablet replicas periodically for ConsistentPrefix reads
  • [3a276ba] Refreshed the README on github
  • [0b062e5] [#680]: Suppress dwarf_lookup_pc
  • [c611ff3] #677: ENG-4389: Master process can fail on startup if FQDN is not resolvable yet.
  • [0aa5d44] Disable a couple of tests on macOS after enabling test parallelization
  • [56b3bfe] #676: ENG-4388: psql wrapper script needs to be updated
  • [94ef7cf] [ENG-4233] Fix Failing Read Replicas
  • [4d6a913] #654 Allow for disabling load balancer for cql load tester
  • [d3c895e] #675 Fix broken table links in yb-master Web UI.
  • [c8cbfb9] Run tests in parallel using Spark on macOS (as we already do on Linux)
  • [3aee3ba] [YSQL] #565 UPDATE support
  • [c81947c] #674: Reduce duration of lock on catalog manager for Postgres sys catalog read and write operations.
  • [c7f8499] #639: Basic Postgres schema support
  • [f8de7c4] #243: Fix thread restriction under RemoteTablet::GetRemoteTabletServers().
  • [20afe12] [#672]: Remove cpp_redis and tacopie from 3rd party libraries
  • [067963c] #638: Support Postgres tables without user-specified primary key.
  • [78cddf4] #611: Support new Cassandra Java driver versions
  • [3832009] ENG-4079: #673: Extract base class and utility functions from QLTransactionTest
  • [ae39261] #670: Fix SEGV when setting paging state on uncovered index query.
  • [e001a9d] [#671]: Enable Thread Safety Analysis in Clang
  • [8729c04] #554 [YSQL] Move PostgreSQL catalog tables to YugaByte Master
  • [0edd204] ENG-4280 #615 ENG-4283 #618: Take prefix locks for rows with compound primary keys
  • [25d12de] #488: ENG-3917: Occasional delay in deletion of compacted files.
  • [0c3e33f] [YCQL] #589 Update YugaByte scanner to match BISON's new API
  • [89d4d63] Limit total size of mini-cluster logs in C++ and Java; fix remote_build.py
  • [98ebafb] [ENG-4263] #597: Implement Rename as a best effort mechanism.
  • [bcefe70] #516: Remove psql from the bin directory in the distribution package and remove psql third-party dependency
  • [4916c63] ENG-4334: #642: Fix double registration of write request
  • [fc2f99e] ENG-4335: #644: Fix TrimTrailingWhitespaceFromEveryLine
  • [f90bf4b] [YSQL] #645 Seg fault occurred when deleting from table
  • [6e798f9] #640: Use call_once for InitializeSystemInfo
  • [6e5454c] #632: [YSQL] Enable default values for columns in YSQL
  • [2d5e4a0] Fix a bug with local-command(s)/keys in a redis pipeline.
  • [f78748c] #552: DELETE statement support in PostgreSQL
  • [7e8a990] ENG-4129: Fix RateLimiter tests for MacOS
  • [6201f0b] Remove restriction on source addresses of PostgreSQL connections
  • [c9a6663] ENG-4291: #626: Handle term change correctly during sys catalog load
  • [da1abd3] #636: Changing yb-ctl to start PostgreSQL child process in each tablet server
  • [4c08e34] #596: ENG-4228: Remove failed non-voter replicas unconditionally
  • [a88cd0e] ENG-4282: #617: Fix IsRangeOperation for the case when range key is partially specified
  • [7577fc3] #630: Run PostgreSQL as a child process of tablet server
  • [f8162af] ENG-4281: #616: Fix order of release lock and MVCC manager notification
  • [f2a56d8] #608: ENG-3570 - Occasional/intermittent FATAL during remote bootstrap: flushed opID higher than in WAL
  • [1321035] ENG-4289: #625: Handle empty response in InetAddress::FromString
  • [5ca7b1f] #619: ENG-3994 - GRANT ALL should only grant the permissions supported by the resource
  • [0d5f3bb] #541 Support copying all tables from a namespace to another
  • [95802be] #593: Resources without permissions granted to them, still appear when querying system_auth.role_permissions
  • [793b4cd] [ENG-3921] #336 Forward pub-sub messages to other nodes in the cluster
  • [5cf0188] ENG-4123: #537: Ignore duplicate write requests
  • [6eaa1ec] ENG-4240: #613: Fix checking of tablet presence during transaction CLEANUP
  • [e609694] Restore sys catalog table name
  • [4c8c5a4] #606: ENG-4102 - A role shouldn't be able to delete itself
  • [117de1d] Support for System Columns
  • [b2b7630] [ENG-4239] Add Date and Time datatypes to yb-client
  • [f68dee5] #609: Add retry to InsertTestRowsRemote
  • [292cb1b] #610: Add retry to DeleteTableTest::GetLeaderUUID
  • [7f4912f] ENG-1333. Report errors in LoadBalancer.
  • [0d50c0c] ENG 3889: Only a SUPERUSER role can alter the SUPERUSER status of another role
  • [c908298] #538: ENG-3471: Fixed write-if handling in case leader is not ready to serve
  • [af34729] #543: Support creating Postgres sys catalog tables in yb-master
  • [c640994] [ENG-4131] Fix ClientTest_TestWriteTimeout
  • [ac8600a] #603: Propagate leader term
  • [9a3823e] #605: Change the /mem-trackers internal UI page to a tree-like view
  • [3072e66] #602: Skip calling Done in QLProcessor::Reschedule
  • [fac2acd] #512: ENG-3501: Added JSON fields checks into the C++ driver test.
  • [04f6b9f] #598: Add MemTrackers to RocksDB
  • [6617193] #600: ENG-4234: Handle failure in LookupTabletDone
  • [adf6560] ENG-3885: Only a SUPERUSER role should be able to drop another SUPERUSER role
  • [e1e717e] ENG-4224 Export BUILD_TYPE in yb_build.sh and reduce username-related failures of run_tests_on_spark.py
  • [ee3e45a] #585: Only a SUPERUSER role can create another SUPERUSER role
  • [09b2558] ENG-4232: Updated C++ Cassandra driver version to 2.9.0-yb-2. (Fixed RapidJson version.)
  • [058a1e4] Allowing the use of a custom (non-default location) Homebrew installation on macOS
  • [aad1797] Clean up handling of system tables in catalog manager.
  • [a1b53a6] #592: Fix syntax for ALTER ROLE
  • [ca97afb] ENG-4187: Fixed ASAN & TSAN issues in the cassandra driver test.
  • [0e68a53] (#573) ENG-4192 Implement minTimeUUID/maxTimeUUID in YCQL
  • [a63e1f9] #587: TABLE keyword should be optional in GRANT/REVOKE PERMISSION statements
  • [f8a4d19] #584: Avoid dropping COMMITTED record from logs and relax status handling in release mode
  • [9a51dd1] #591: Extended lifetime of RequestScope while writing provisional records
  • [2e73b50] #590: Use rescheduler for aborting transaction in ExecContext::Reset
  • [0d583ac] #581: Use correct check when QL executor should be rescheduled
  • [836fc6a] ENG-3601: #570: Use Prometheus compatible names for mem tracker metrics
  • [5ef6fe8] #567: Resolves fatal error when starting cluster in RF1 with PostgreSQL enabled.
  • [046c0f1] Remove the old PGSQL service and parser from our code base because we are now using PostgreSQL codebase directly
  • [4a3a8e4] #572: Change defaults of rpc threads/queues and tablets.
  • [9561c35] #41: ENG-4170: Implemented ALTER KEYSPACE statement.
  • [528b5d6] #580: Make ReplicaState::LogPrefix lockfree
  • [521c35e] #576: Use std::mutex in RaftConsensus instead of simple_spinlock
  • [91487ce] #575: Avoid calling CompleteWithStatus under lock
  • [ee65667] #564: Fix performance degradation with authorization enforcement
  • [33178f3] [ENG-4171] [#568] Fix ZRANGEBYSCORE SegFault.
  • [7acb661] ENG-3601: #570: Export memtracker stats as metrics
  • [f7829b0] #536 ENG-4128 Create transaction-status table on DDL path
  • [ab986ef] [#566] Enable TIMESTAMP type in PostgreSQL; fix float test on ASAN by disabling UBSAN on float.c
  • [960cb2c] Fixes #164: Select statement using uncovered index
  • [0325a4f] #512: ENG-3501: New Cassandra C++ driver test with YB ExternalMiniCluster.
  • [93cc7d8] Adding tests for float4 and float8 PostgreSQL types
  • [751c17c] Fix macOS packaging script so that it accepts @loader_path library paths
  • [0313d20] ENG-4125: A drop table/keyspace/role statement should automatically revoke all the permissions granted on the deleted resource
  • [7c1d28d] ENG-4151: Modify sample apps to accept username/password for Cassandra workloads
  • [adaee62] #512: ENG-3501: Added cassandra-cpp-driver to yugabyte/thirdparty.
  • [c98a687] Update minimum CMake version to 3.7.0 for FindICU module
  • [52fb4a4] yb_release.py: don't complain about the default YB_THIRDPARTY_DIR being different from that in the build descriptor file
  • [6c346c1] When creating a new resource (table/keyspace/role), the role creating it should get all the permissions on the resource
  • [47dc36b] Re-enable yb_int4.sql and yb_int8.sql tests, fix overflow handling in ASAN
  • [d3b589d] #526: Resolve query error after switching role in cqlsh
  • [fd1a4ec] YSQL Optimize SELECT with fully-specified multi-column primary key
  • [7e073ae] DELETE and UPDATE Basic Support
  • [28f4650] Fixes for Monitor/Config commands
  • [2cbd2dc] #73: ENG-2665 - Cassandra Authorization: Enforcement
  • [42e5e82] [#523] Enabling more data PostgreSQL types, adding a pg_regress based test suite, and fixing a bug with YBPgStatement lifecycle in the YB FDW
  • [cda791b] [ENG-3920] Pt. 1/3 Refactor existing monitor architecture to support local implementation of Publish/Subscribe
  • [50b527f] ENG-4124: Handle schedule failures in tests correctly
  • [286ea60] ENG-4101: Introduce intrusive MPSC queue and use it in preparer
  • [0451cc4] ENG-3397. Redis KEYS
  • [1441732] #522: Convert VERSION SysCatalog entry into flexible config entry.
  • [eed35bc] ENG-4118: Make all files within a release archive owner-writable to avoid errors during deployment
  • [9cf50c4] Fix LinkedListTest.TestLoadWhileOneServerDownAndVerify test
  • [89aa768] Enable some supported YB-Postgres features in parser
  • [8947924] SELECT with fully-specified multi-column primary key
  • [4952508] Update thirdparty version for Jenkins on Linux
  • [1ac4b39] #73: ENG-2665 - Cassandra Authorization: Permissions cache
  • [fa1b53f] Rename StringPiece to GStringPiece to un-break the build on Ubuntu 18.04
  • [1dc6d31] ENG-3861: #505: Fix check that reactor could accept new tasks
  • [b710c64] #507 Fix TestPgTransactions#testTransactionConflicts: read requests could get stuck if transaction status tablet leader changes
  • [b7fc14d] ENG-4037: Add ability to specify transaction start time externally
  • [f5bb57f] ENG-4032: Support date string literal before 1400 AD in YCQL.
  • [c753f84] #482: Raise error when the WHERE clause of a SELECT DISTINCT statement contains reference to a range or non-static columns.
  • [e71032d] Allow SET SESSION TRANSACTION and DROP DATABASE
  • [feec3e6] [ENG-3779] Expose JSONB type via yb-client
  • [f62a4ee] Report errors for unsupported PostgreSQL features in YugaByte
  • [20c37fe] ENG-4044 Fix how we handle PostgreSQL's text datum to unbreak SELECT with a text key specified
  • [95fcba2] Support for PostgreSQL in the macOS packaging script
  • [a3f2e87] Unset the LANG env var when launching PostgreSQL server
  • [8d14d9b] Decrease execution time of QLTransactionTest.RemoteBootstrap
  • [2715679] ENG-3949: #479: Decrease log flooding from distributed transactions
  • [ba68646] Misc extensions and fixes related to YB-PostgreSQL
  • [883f9a9] Fix how arguments are being passed to the constructor of the MacLibraryPackager class
  • [87f7429] Integrating PostgreSQL with YugaByte distributed transactions; adding PostgreSQL to yb-ctl and the release package
  • [438a60c] #493: Fix various issues with transactions and restarts
  • [d2b3c76] ENG-3898 Extend SELECT query support for YB-PostgreSQL
  • [f9a851f] Increase the default CQL queue limit to 5k
  • [d27d140] ENG-3836. Changed DCHECK to Status return in load balancer.
  • [90656c8] ENG-3987: Fixed GCC version handling
  • [ef610e4] ENG-3871: Test failure in TestMultipleLeaderRestarts
  • [4786d33] ENG-3965: Change timeout handling in RPC
  • [1ddc98b] #471: Implement YCQL date and time standard functions
  • [d7debca] Add seconds to Alive column.
  • [2b5fc7e] ENG-3874: Write empty batch with
  • [453b81c] Full scan support for Postgres API
  • [627a03f] ENG-3959. Add set_preferred_zones command to yb-admin.
  • [406a278] (#492) ENG-3416: Fix end-key calculation for range deletes
  • [86a103a] #490 (ENG-3981) Avoid creating checkpoints dir for system.transactions table
  • [1308465] #472: Support boolean datatype in YCQL primary key column
  • [2d2f378] #471: Support date and time datatypes in YCQL
  • [48adf81] #485 (ENG-3962) fix Tablet::AlterSchema() to flush tablet metadata when schema version changes because of index change also.
  • [54c2645] ENG-2326: #484 : Periodic sync for Remote bootstrap
  • [27c5898] ENG-3862: Add an intermediate state when scheduling a delay in RpcRetrier
  • [03bde5a] Avoid using ntp_adjtime in default clock
  • [d6e5118] Stabilize TestPgWrapper: correctly shut down backends on SIGTERM, and fix pointer corruption caused by implicit function declarations
  • [88e1d69] ENG-3949: #479: Fix removing from list of transactions that should be cleaned.
  • [c71ed15] PR #429: Adding SimpleQueryTest suite from Apache Cassandra to Yugabyte DB's YCQL testsuite:
  • [aba3ba1] ENG-3916. ZRangeByScore LIMIT option.
  • [749601b] #480 ENG-3954: Check if a transaction is aborted
  • [928220a] #369 Fix AlterTableRandomized.TestRandomSequence
  • [f81514d] ENG-3650: Patch old transactions status table type in sys catalog
  • [9904950] [#386] Switch ResettableHeartbeater with Periodic timer
  • [f2c3643] ENG-3896: Allow DESCRIBE permission on ROLE resource only
  • [ad371b9] ENG-3650: Remove the dependency of transaction coordinator on transaction participant
  • [f9f1fbb] ENG-3897: Remove calling MVCC w/o lease specified during intents cleanup
Assets 4

@ravimurthy ravimurthy released this Sep 19, 2018 · 632 commits to master since this release

Overview of Changes

  • Distributed multi-shard/multi-table ACID transactions!
  • Global & consistent secondary indexes
  • UNIQUE constraint on secondary indexes
  • Native JSON data type
  • Public/private IP bindings for multi/hybrid cloud or multi-region deployments
  • User authentication
  • Redis-compatibility features added to our YEDIS API:
    • “Read from followers” for tunable read consistency
    • Support for multiple databases or namespaces
    • Support for MONITOR, EXPIRE, TTL
  • Bug fixes and performance enhancements
  • Plus excellent progress on the Postgres-compatible SQL API!

Key Commits

  • [068f64f] Fixes #49 #111 #465 Efficiently support IN conditions for clustering columns
  • [92d442a] [#386] Switch ResettableHeartbeater with Periodic timer
  • [4ff9e7c] Remove TTL from init marker on TS type.
  • [7b28dd7] Cap max pre-allocated file size for compactions at 1G
  • [887820d] Fix hang during tserver shutdown
  • [afef8f2] Fixes #469: Optimize QLResultSet to serialize rows results directly
  • [2fdad35] #456: Update committed hybrid time from WAL records
  • [7e4108a] #467: Fix several issues with remote bootstrap
  • [0d9feb2] #338 Preliminary support for Redis Lists
  • [d902325] #334 Redis (P)ExpireAt Support
  • [d4865d4] #460: Add handling of controller status in GetMasterEntryForHosts
  • [19c9fed] Add memory limit to ram ratio (#458)
  • [c6538a8] Added broadcast address for stable pod IDs in K8s. This fixes master/tserver pod failure cases. (#464)
  • [cdbf60c] [#289] Cleanup intents for aborted transactions in the compaction path
  • [a0a2072] #459: Upgrading k8s yb cluster fails when bind address is private ip of pod.
  • [8438e9f] #418: Increase default network rate for remote bootstrap
  • [653871a] Prevent PeerMessageQueue from being deleted while Peer could use it
  • [6022cae] Add loadbalancer option (#450)
  • [154810c] (#418) Rate limiter for remote bootstrap
  • [e4a55b9] #451: Fix double reply in case of read from follower during startup
  • [5842ed6] Expose raw size in tablet-server metrics #439
  • [3bd4f37] #451: Check result of weak_ptr lock, before doing actual call in LocalYBInboundCall
  • [900a733] Fix CassandraSecondaryIndex degradation
  • [f8a83c2] Use scoped_refptr for pggate classes so we can get rid of maps from raw to shared pointers
  • [41b9a6e] #384 Rename ApplyAsync to ApplyOperation and get rid of the unused apply thread pool
  • [6cc4bfb] Reverse scan should handle intents.
  • [48a3d8f] Set in-memory master address list in tserver after a full move.
  • [5a460b8] #444: IntentAwareIterator is missing intents even during a forward scan
  • [d355997] Fixes #445 Invalid start_doc_key error with IN condition and paging state
  • [78c4c26] Split tables in YB /tables view by system/index/user
  • [e33a2bb] IntentAwareIterator could miss intents from the same transaction
  • [e940d61] #403 Support Deleting and Recreating a Redis Table online.
  • [2a4fb09] #432: Add "transactions" and "is_unique" columns to system_schema.indexes
  • [898e432] Integrate basic PostgreSQL statements from the external PostgreSQL process with the internal YugaByte PostgreSQL API
  • [82884f9] Skip cleanup when transaction is not loaded
  • [4be4c9b] Fixes #441: Return selected columns of "select * ..." with index in the same order of the table columns.
  • [b2164ff] Fixes #434: Deletes of specific range key incorrectly deleting all rows for that partition key
  • [b052e13] Fix to execute GetSegmentsSnapshot using lock
  • [1c6e237] #428: Support remote bootstrap for very large tablets
  • [eb04e90] Add file_macro to CCACHE_SLOPPINESS for better ccache hit ratio
  • [36953b4] #388: Support batch request of DMLs on table with secondary index
  • [d55bad6] Fix handling of write intents for aborted transaction
  • [73a5e9d] Ignore error when removing index from the indexed table while dropping an index.
  • [9d0085a] [#427] Change CQL wire protocol to move the JSONB type
  • [b5d66ea] Make master and tserver have their own overrides (#426)
  • [94be2b1] Fix UpdateTransaction call during cleanup
  • [03a3e47] #423: Truncate indexes when truncating a table
  • [7e82039] Remove intents of aborted transactions
  • [8758c9f] #343 Implement Select and CreateDB commands.
  • [026b582] Allow overriding the UUID to be used for new instance metadata for a fresh master or tserver
  • [03079e3] Allow remote compiler execution mode for Postgres build
  • [8765bab] #415: End a query always when LIMIT clause is hit.
  • [e65b6ed] #397: Added ability to specify multiple endpoints for master
  • [9bc7dc1] #410: Remove unused CQLRpcServerEnv
  • [7b925bd] Support for SELECT in Postgres API
  • [74751e8] #407: Refactor CQL write dependency check
  • [ee502c7] (#398) Make secondary index update RPC calls async
Assets 2

@rkarthik007 rkarthik007 released this Aug 17, 2018 · 768 commits to master since this release

Overview of changes

The following are some of the key fixes and features in this release:

  • Basic helm chart support to deploy Kubernetes clusters
  • Various bug fixes and performance improvements
  • YEDIS
    • Auth support for Redis
    • Safety flags to disable dropping data, for example using commands such as FLUSHALL
  • YCQL
    • Added support for "broadcast address" to enable public and private IP support
    • Ability to return fine-grained errors in CAS (compare and set) and batch statements
    • Support for weak secondary indexes for higher ingest performance

Key Commits

  • [76a3550] #401: Clean up YCQL batch execution
  • [ad4abee] #382: Unable to initialize catalog manager with docker.
  • [6db30ec] Updated the pom file to include new Cassandra driver changes.
  • [5d1774d] Clean up the use of SQL statement string in statement processor
  • [083465c] IsLoadBalanced check made per cluster.
  • [e9606c6] (#375) Optimize queries with IN list to execute in parallel when reasonable
  • [2048e79] #385: Add CassandraUniqueSecondaryIndex workload
  • [048e821] Fixed snapshot files collecting in the RemoteBootstrapSession and related test.
  • [0dcb7f0] #377: Move unique index duplicate-value check to index tablet.
  • [b5502ef] In TabletPeer, we should not access try to call log() before the log is initialized
  • [7cb3df0] Add batch update test for weak secondary index
  • [76aac82] Specify correct parent mem tracker for outbound connections
  • [189260f] Public/private IP support in YB
  • [059317c] #350 Avoid blocking in Peer::Close()
  • [d898ee1] A macro-based DSL for C-to-C++ APIs for PostgreSQL integration
  • [7e6fddd] [ENG-3454] Don't close Jedis after each pipeline batch.
  • [5a062dd] Improve performance when sending peer's requests (#350)
  • [a921f98] (#347) [YCQL] provide mechanism for DML operations to return finer-grained errors
  • [b2fa67b] ENG-3580: Fix abort scheduled async RPC task
  • [c0abc0f] #373: Persist index info in tablet metadata.
  • [eca8f18] [ENG-3484] #335 Implement Redis Monitor command
  • [28b4386] (#374) Add suport for partition_hash() builtin function.
  • [a5a78cd] [ENG-1735] #283 Implement Auth support for Redis
  • [93cd133] [ENG-3543] #357 Fix error messages for Redis
  • [0197104] ENG-3546: CQL TLS Encryption
  • [2848cd6] ENG-3326: Fixed reverse scan infinite loop issue.
  • [10ef4e2] [#145] Sum for Varint or decimal columns
  • [5985d82] ENG-3564: #361 Support for Alter Role
  • [b4f2b82] #366: Update README.md to have a section for running the C++ and Java tests.
  • [7e8e85e] (#349) Enhance YCQL grammar to support interchanging order of OFFSET and LIMIT clauses.
  • [a8dd107] #362: Cache index table metadata in tserver when updating an index
  • [f31cfc9] ENG-3472: #346: Clean committed transaction in TransactionParticipant
  • [16f1d6f] ENG-3529 #351: Refresh master leader when server queue is full
  • [cedfae3] ENG-3527: Implement weak secondary index
  • [b595fe5] #345 Fixed the return type of "PING" by adding a flag for returning a Simple String vs Bulk String
  • [2bd0312] #352: CREATE INDEX syntax changes
  • [81ac392] Clean up the C API for PostgreSQL integration with YB data fabric and use YBCStatus
  • [50fb6d2] ENG-3493: add ELSE ERROR to YCQL
  • [0f8ca9c] Remove some unnecessary functionality from yb_fdw
  • [5009728] (#339) Support YCQL offset clause in docdb and our paging state.
  • [a5719bd] Generate PostgreSQL catalog script for YugaByte DB
  • [9343596] (#339) Add grammar and semantic analysis for offset clause.
  • [6847aff] ENG-1756: Support Redis "CLUSTER SLOTS"
  • [708ea83] ENG-2929 : Shell master start should not allow master_addresses flag to be set.
  • [6716483] Set up a placeholder foreign data wrapper yb_fdw using file_fdw
  • [607173b] Allow specifying YB master addresses to use inside pggate with an env var
  • [160ab8e] Refactor PostgresSecondaryIndex workload.
  • [3aa2066] ENG-3453: Settings for TLS encryption
  • [8c229db] Add load-balancing test for query with secondary index
  • [d2366eb] ENG-3475: Remove compaction filter for provisional DB
  • [a062473] ENG-3481: added option to disable flushall in redis (#337)
Assets 2

@rkarthik007 rkarthik007 released this Jul 18, 2018 · 873 commits to master since this release

Overview of changes

Here are some key features and fixes:

  • Support for unique constraints to ensure that a column does not have duplicate values in YCQL.
  • Support fine grained selection and update of JSON attributes, built-in JSON operators in YCQL.
  • Support for AVERAGE, blob conversion functions in YCQL.
  • Ability to read from the local datacenter when using YEDIS.
  • Support ZSCORE in YEDIS.
  • Support bounded staleness for follower reads.
  • Presto integration.

Key Commits

  • [ab0611e] ENG-3453: Add TLS support to RPC layer
  • [70fcb1b] Add a workload to test secondary indexes in postgres.
  • [b06d6c3] Use thread pool to pickup status tablet when wait is disallowed
  • [979fbc8] Ensure we allow the same column multiple times in WHERE and IF clauses for json and collection columns
  • [aa3621e] Fix check failure when reading in future
  • [8e0bf34] #322: Always raise unique constraint error for unique secondary index violation
  • [bc58df3] Fix TSAN issue with sys_catalog in catalog manager
  • [43c1146] (#331) ENG-3407: Pick local transaction status tablet
  • [25177f6] Remove redundant delete from index table
  • [9879ada] #276: Bounded staleness for follower reads
  • [c6e0c4d] #322: Abort transaction on failed DML.
  • [b8ddceb] #322: Implement unique secondary index
  • [7ccda6f] #327: Fix missing null index column values.
  • [642e0e7] #325: Fix SEGV when creating an index with an unsupported index column datatype.
  • [97f32b2] #287: Fix occasional inconsistent read of data written in a distributed transaction
  • [405396a] #293: Use a separate RocksDB for provisional records
  • [d0779bd] #294: Tserver misses new master info in some cases.
  • [9582136] (#294) Fix tserver overwriting master address list in-memory.
  • [1c3a235] #311 Prefer local tserver, or tservers in the same zone when redis_allow_reads_from_followers is set
  • [a855a96] ENG-3378 cqlsh: "select * from system.transactions" crashes tserver.
  • [de1a8f9] (#303): Handle multiple updates to same column in SET expression appropriately.
  • [c846c6c] Support fine grained updates for JSON columns.
  • [41f4982] #310: Fixes for yb-ctl setup_redis wait and num_shards
  • [25c75f4] Add support of NTP clock to master
  • [6287d0a] ENG-3282: #313: Use HostPort in Proxy instead of Endpoint
  • [03dc761] ENG-2724 Strongly Typed Uuids
  • [3146d18] Implement transaction read restart
  • [c2e0f20] Add metric to track read requests requiring restart.
  • [57d40c6] #312 Fix handling of min value as token upper bound
  • [3ac9146] Add metric to track expired transactions.
  • [103f074] #307: Fix occasional "SetTransaction with non empty batcher" error when committing a distributed transaction
  • [7bb3972] ENG-3317: Fix issues with high concurrent transaction aborts because of write conflicts
  • [99f0ae1] #264 Implement blob conversion functions in YQL (simple data types).
  • [4b450cd] #238: Retry distributed transaction on conflicts
  • [eddf548] (#291) [YCQL] Aggregation functions headers
  • [0be501d] ENG-3293: implement secondary index workload
  • [40d75b7] Remove error logging and reduce pool overflow log rate.
  • [5b4b4b4] #296 Add the size_estimates system table to YCQL
  • [9aca5e0] Update Spark and Cassandra driver versions
  • [4f1d848] (#26) Implement CAST(expr as Typename) builtin function (especially for json)
  • [2dfc031] (#290): Add support for avg in YCQL
  • [30c425f] ENG-3213: Implement IsLeaderLoadBalanced RPC
  • [9a13bde] Remove ExecContext::Error overloaded APIs
  • [1c011ea] #277: Add metric to track distribute transaction conflicts
  • [71f1d82] Ensure json scalar strings don't require quotes.
  • [28845be] Fix some sample apps to honor the user specified table name.
  • [49edf46] Add custom numeric opcode to builtin function directory.
  • [deb2fc5] #282: Enhance master change config to remove a dead peer.
  • [56803e3] ENG-3184: Two read buffers per connection should not count towards the read buffer limit
  • [5dae084] Reduce rpc timeout defaults for redis
  • [f619ccd] ENG-3239. Remove repeated log.
  • [94d19c0] (#175) ENG 3108: Implement ZSCORE
  • [5bfb330] ENG-3217: Introduce interval for update consumption of root mem tracker
  • [92ce6d0] #260 Accept lowercase options expire_in expire_at. Improve the error messages when the absolute timestamp is incorrect
  • [eb5b0f8] #259 Fix inconsistencies in YCQL keyword list
  • [5d48691] #237: Fix transaction status leadership change
  • [7bf8cbb] (#163) Support fine grained selection of json attributes in CQL.
  • [5cc9357] (#162) Add support in docdb to correctly evaluate json operators.
Assets 2

@rkarthik007 rkarthik007 released this May 4, 2018 · 987 commits to master since this release

YugaByte DB 1.0 release 🔥🍾

Overview of changes

Big payload!

  • PostgreSQL API beta support
  • Cassandra API enhancements (now called YugaByte Cloud Query Language or YCQL):
    • Introduced a JSONB data type for storing documents
    • Added a personalization and a timeseries workload to the sample apps
  • Redis API enhancements (now called YugaByte Dictionary Service or YEDIS):
    • Introduced follower reads
    • Added more command coverage (like ZRANGE, TSLastN, etc)
  • Distributed transactions enhancements
    • Increased transactions perf
    • Fixed some corner case error conditions
  • Lots of performance improvements
    • Improved core decoding logic to boost perf on 2-core machines
    • Better handling during huge spikes (backpressure)
    • Handle large network buffers better when getting large memory-buffer requests

Key Commits

  • [0d07add] #239 Callback should keep a reference to Synchronizer in UpdateReplica
  • [10338ff] Consensus: Use periodic timers for failure detection
  • [f1e12eb] Loose-ends for PostgreSQL support
  • [e9f3735] Table splits should be created based on primary cluster tserver count.
  • [193cd4c] (#162) Add grammar and parsing for basic json where clause filtering.
  • [0bc9d69] #198 Show only Cassandra keyspaces and tables in cqlsh
  • [39ef45d] Fix bug in Postgres SSL negotiation handling
  • [95aa22e] Read replicas should not show up as LEARNERSs
  • [a5bbca4] Avoid using spinlocks in the ts_tablet_manager.
  • [ec4ae20] Fix value validation in CassandraTransactionalKeyValue load-tester.
  • [c19b14f] Fix intermittent "Pool is CLOSING" exception in the write-only setup phase of Java load-tester
  • [c560765] Adding java support for shared thirdparty
  • [c52c55e] Automatically set network buffer limits based on available memory
  • [7ba423c] LogCache improvements: SpaceUsed outside lock and fix race on next_sequential_op_index_
  • [7892a2a] Limit max wait in SetPermanentUuidForRemotePeer
  • [9bf8fce] Use atomic variable for state in TabletPeer
  • [43cda3b] Further reduce writers for batch CQL workloads.
  • [7458489] Removed filenames from error messages
  • [e0b4b5d] Add links for Cassandra, Redis and PGSQL API RPCs in progress in tserver.
  • [aad076c] Reduce default logging from reactor.
  • [be18db9] Fix TSAN version of TestRedisService.TestHMGetTiming
  • [bb204e4] #236: Fixed TestPathUtil.TestODirectFileCreationInDir
  • [26552c5] Linting error for common due to new lib yb_bfpg
  • [e86a336] #197: Fix DoсDB compaction filter to skip transaction metadata
  • [222341b] Switch RaftConsensus to shared_ptr
  • [3cdeecf] Load balancer does not respect replication factor
  • [a9fcc08] #196: Fix "Commit of expired transaction" error under high-load distributed transactions
  • [9b6df54] Harden CreateTable validation in CatalogManager
  • [5dc30fc] Improve DocDB and IntentAwareIterator performance
  • [92ac433] Keep pending operation counter disabled on tablet shutdown
  • [49c9d4a] Postgresql Support
  • [7b12cce] Allow integration of distributed txns with NTP clocks such as AWS time sync service
  • [d6d130b] #194: Some java client wait APIs could incorrectly exit due to overflow.
  • [b125942] Fix postgres port in java mini cluster
  • [7482468] #170: New TsRevRangeByTimestamp redis command
  • [8797e39] Implement ZRANGE for Sorted Sets.
  • [158def7] Improve performance of IntentAwareIterator
  • [91e6e02] #159: Added psql 3rd party
  • [9ba8436] Async DNS resolution of peers
  • [a7d1447] Removed exclusive index bounds in ZRevRange.
  • [d0679af] #23: Use cassandra driver's DNS resolution
  • [b67e03c] #82 Do not tombstone a tablet even if new tserver doesn't get promoted
  • [d537ed2] #87 Expire based on time spent in the queue
  • [1796740] (#141) DocDB filtering on minor compactions
  • [08762c7] #148: Fixed clqsh wrapper
  • [be2d598] Basic jsonb implementation with support at C++ SQL processor level.
  • [c9d74a2] #110: Allow redis reads from followers
  • [d78db0f] Remove transaction_table_num_replicas flag.
  • [e640ac7] Add support for arrays in our jsonb serialization format.
  • [05d100c] Additional data types for jsonb serialization.
  • [907243a] (#109) Support TSCard for redis time series.
  • [17c3081] Make the appenders use a thread pool
  • [c2555ed] Allow YBClient callback(s) to run in a separate threadpool
  • [9cd6a2f] Limit the total amount of memory consumed by network read buffers
  • [8f2f390] add gflag to enable/disable compression of sstables
  • [29058a0] #109 Implement TSLastN functionality for redis time series.
  • [89a0269] Change defaults for Kernel Watchdog
  • [4c5813a] Detecting if the data/log/wals allow file creation with O_DIRECT.
  • [409704d] Increase performance of FastDecodeSignedVarInt.
  • [74536a0] PrimitiveValue write time is being accessed without being initialized
  • [95f3092] Create yugabyte-statefulset-local-ssd-gke.yaml
  • [ee306e0] Add a Cassandra batch timeseries app.
  • [37f9c6e] Fix failed task transition kStateWaiting -> kStateRunning
  • [6d981be] Fix batch/transaction inter-statement dependency analysis for multiple writes.
  • [561b811] Avoid overhead of encoding read_time_.global_limit in IntentAwareIterator::SeekForward().
  • [57946b9] Cassandra Personalization Workload
  • [1192f57] Avoid overhead of decoding DocHyridTime in IntentAwareIterator::SkipFutureRecords.
Assets 2
Pre-release

@rkarthik007 rkarthik007 released this Mar 21, 2018 · 1119 commits to master since this release

Overview of changes

  • Basic jsonb serialization and de-serialization to support a document datatype
  • Lots of performance improvements
    • Improved performance when running on 2 core machines with auto-tuning of various gflags
    • Reduced lock contention
    • Eliminated unnecessary memory copies
  • YugaByte DB can now run on ZFS filesystem
  • Initial secondary index read/write support
  • Handling large requests in Redis and memory protection on violating max command size

Key commits

  • [cdf48e3] Fix hybrid time going backwards because of logical component overflow
  • [0bf401e] Add RPC read buffers to MemTracker
  • [451b7f8] Misc client template simplifications
  • [e27d429] #103 Do not crash due to block size mismatch on ZFS
  • [218ddcd] Remove log spew from TabletServiceImpl::Read
  • [956deb2] #101 pick better auto-defaults for max number of compactions
  • [28ca802] Keep java client API to pass in number of tablets for redis table creation.
  • [e321311] Secondary index read/write
  • [e463718] Fix macOS build issue in physical_time.cc
  • [f4a1c89] Fix build in pg_server_main.cc
  • [5a91590] #1 Enable building on gcc 7 on Ubuntu 17.10
  • [6f9bdef] Add version info to log file including build timestamp and git
  • [f9bafcc] New clock abstraction layer
  • [2826097] Change defaults on server side for some flags.
  • [0440305] Fix missing comma in ybctl for pgsql service display.
  • [a18d393] Remove number of tablets from redis table creation client APIs.
  • [26f909e] RPC support for processing commands from Postgresql clients.
  • [40786a4] #97: Fix for Fatal failure on scaling down kubernetes cluster
  • [a60c81f] Report the stored log entry header CRC in the CRC mismatch error message.
  • [4c6baeb] Remove semi-colon requirement after "begin transaction" plus misc code cleanup.
  • [5104ad2] #98: Remove pointer to TableReader from MultiLevelIndexReader
  • [4327930] Quick fix of SegFault during reads.
  • [44640b0] Avoid copying of QLValue and QLValuePB.
  • [960f19f] Do not print a failure message when remote bootstrap is started successfully
  • [5b16cf1] Send num shards parameter to master from yb-ctl.
  • [c311ad3] Master removal needs to remove yb-data contents for shell mode.
  • [2726eac] Resolving bug where the placement info that gets allocated to a master/tserver is off by one respect to the command line argument.
  • [280fdc0] Fix ScopedPendingOperation to return Status::Busy error when unavailable.
  • [d16bc28] Backoff exponentially for server too busy errors
  • [e0b78dc] Set upperbound for intent iterator in IntentAwareIterator.
  • [7147469] Miscellaneous test improvements, and better diagnostics before a fatal error in MvccManager::AddPending
  • [6be69a8] Handle case when read buffer is full and command is not fully transferred yet
  • [72de5cb] Put ValueType::kIntentPrefix's before all non-intent ValueTypes.
  • [ed45995] Handle dependencies for libraries in mac_library_packager.py
  • [556f710] TSAN Race Condition between Register and Load Balancer.
  • [e2f4b9b] Fix the increment of SQLProcessor_ExecuteRequest metric.
  • [2871820] #93 Fix trying to read ahead of the log while preparing peer request
  • [b1d4d7f] #90: Reduce lock contention in TabletInvoker::IsLocalCall
  • [0250c13] #88: Fix return value from XX and NX options
  • [63363df] Basic jsonb serialization and de-serialization code.
  • [897a925] #86: Accept lowercase options for SET
  • [679b57b] Fix bug timezone parsing for timestamp literals
  • [4902d06] Allow flushing all tablets of a given table
Assets 4
Pre-release

@rkarthik007 rkarthik007 released this Mar 21, 2018 · 1176 commits to master since this release

Overview of changes

  • Enables high data density by turning multi-level indexing on by default. This allows efficient queries even where there is a lot of data per node.
  • Core performance fixes to run better on lightweight (2 vCPU) machines
  • Better handling of huge io spikes. The scenario is having many more requests from the clients than what the database cluster can handle.
  • Tuning the Redis query layer performance

Key commits

  • [a4498b4] #88: Ensure we check TTL when we look up the type of a subdoc
  • [beae94d] Remove Scoped watch stack from outbound calls. Gflags for consensus log
  • [abe90b1] Make the appenders use a thread pool
  • [6b33e8d] Bundle yb-prof.py in the release package.
  • [e94574c] Change defaults for the number of concurrent tserver-to-master RPCs and Redis buffer size
  • [e96545a] Allow std::atomic usage with clang and libstdc++
  • [13724f5] Remove unnecessary encoding and decoding from DocDB
  • [b36c90e] Spawn a shell master when adding a new master.
  • [702d528] Use multi-level indexes by default
  • [72ffc21] Fixing reactor-test: DelayedTask::AbortTask should still call the callback for tasks that have not been scheduled
  • [c0c4c7b] Add paging to table iterator to fix RaftConsensusITest.TestAutoCreateReplica
Assets 2
Pre-release

@rkarthik007 rkarthik007 released this Mar 9, 2018 · 1189 commits to master since this release

Overview of changes

  • Support for using Kubernetes style cnames in YugaByte CQL client natively! The sample applications can easily be run as a Kubernetes service now.
  • Lots of perf fixes for running YugaByte on machines with large number of CPUs.
  • Added built-in time functions such as now(), totimestamp(), tounixtimestamp() and unixtimestampof().
  • Support for high-performance batch transaction codepath with intra-batch dependencies.

Key commits

  • [fadfd84] Use a separate thread pool for consensus service
  • [71fc829] Initialize being_compacted to false
  • [db55a6f] Support Kubernetes style cname in YugaByteDB clients
  • [c27f929] Correct apply of repeated keys in txn
  • [8ceaeb9] Enable creating multiple outstanding pipelines per IO thread
  • [699a58e] Handle intra-dependency in batch request / transaction
  • [39100f4] Replace upgrade lock with shared_lock and unique_lock
  • [f17b2a8] Implementing now(), totimestamp(), tounixtimestamp() and unixtimestampof() functions for YQL.
  • [024730e] Use circular buffer for receive
  • [c619be7] Reserve the number of subkeys from ParseCollection
  • [de6a268] Handle intra-dependency in batch request / transaction
  • [a26f8d2] Need to remove the special tserver service queue sizes on "low-cpu" machines.
  • [3857155] Change distributed transaction test cases to test writing to the same hash / primary key in the same transaction.
  • [99d4adb] Group lookup tablet requests
  • [0bd8878] Set iterate_upper_bound to avoid seeking over deleted keys in RocksDB.
  • [88cf84e] Introduce limits for queued commands and don't drop connection on read buffer overflow
  • [2250acb] #59: Set GIT_DIR to /tmp when building AWS SDK so that it does not pick up YugaByte DB version as its own
  • [b4a9b8e] Reduce log spew for unavailable peer
  • [0261099] Make Redis read buffer size and number of concurrent master lookups in meta cache configurable
  • [f3e9672] Gflag for block cache num shards
  • [b0553d9] Porting nonredundant functionalities of local_cluster_ctl to yb-ctl.
  • [dc6e2e2] Log spew during tablet startup under load.
  • [4ad88df] system_redis.redis should not be counted as an user table
  • [7302b1f] Correct handling of failure to retry RPC command
  • [f90a83f] Fix TransactionManager destruction
Assets 4
Pre-release

@rkarthik007 rkarthik007 released this Feb 22, 2018

Overview of changes

  • Support for natively running YugaByte DB using Kubernetes StatefulSets! YugaByte DB now works in GKE as well.
  • Distributed transactions supported at isolation level SNAPSHOT ISOLATION.
  • HINCRBY support in Redis API.
  • Defaults for CQL client driver switched to strong consistency.
  • Lots of performance enhancements and fixes. Notably - lock free hybric clock implementation. New YugaByte perf report uploaded.
  • Improved debuggability - added slow query logging for Redis.
  • Overhauled the third party build system to make it more readable and easy to maintain.

Thanks

Key commits

  • [2150bee] Added AWS C++ SDK 3rd party.
  • [8d8030f] Fix TransactionManager destruction
  • [9546e7e] Fix counter column datatype in system_schema.columns.
  • [3074f56] Allow multiple writes of the same row within a batch request.
  • [8ee1288] Update 3rd party version for mac
  • [cfd54dd] Periodic timers
  • [2292067] Implement HIncrBy
  • [7a7421f] Fix a CHECK failure due to handling of slow responses when looking for leader master
  • [bed88c9] Create cloud-machine-types-analysis.md
  • [90ae07b] Update yb-perf-0.9.5rc-Feb-13.md
  • [ab9deaa] Set defaults for read_pool_* flags
  • [eb32648] #51: Kubernetes cname only tries the first ip address on GKE (#53)
  • [2dbd7e8] Update yugabyte-statefulset.yaml
  • [0c87387] Fix master UI to show only running tables
  • [c294304] Capture by reference to avoid copies during HMGet
  • [c6d2635] High CPU usage in CleanWaitingConnections
  • [d4fd22c] Print slow queries for Redis. Similar to CQL
  • [6488bfd] Set GLOG flag so that we do not sync for WARNING messages
  • [e230091] Fix parsing of tcmalloc max thread cache bytes.
  • [54121c0] Added tests for ImportSnapshotMeta.
  • [072595a] Update Cassandra Java driver versions to use default YB_STRONG consistency level.
  • [c62d02a] Make HybridClock Now/Update lock-free
  • [4fa3118] Parallelize reads for Redis ops in the same TabletService::Read request
  • [8cd931c] Increase tcmalloc local thread cache size for tservers.
  • [3ef0019] Reuse iterator for Redis Operations
  • [dfb16c9] Prevent core dumps for LOG(FATAL) and CHECK.
  • [07ffec9] Fix QLTabletTest.SkewedClocks and other safe time logic in MvccManager
  • [58c97d3] Change YQL transaction isolation level to SNAPSHOT_ISOLATON.
Assets 4
You can’t perform that action at this time.