Skip to content

@rao-vasireddy rao-vasireddy released this Oct 28, 2019 · 110 commits to master since this release

Release highlight

YSQL is now generally available and is ready for production deployments. Details available in announcing YugabyteDB 2.0 GA: Jepsen Tested, High-Performance Distributed SQL](https://blog.yugabyte.com/announcing-yugabyte-db-2-0-ga:-jepsen-tested,-high-performance-distributed-sql/ ).

YSQL changes

  • [YSQL] Build YBTupleID value with respect to DocDB column order. #2438
  • [YSQL] Support TLS Server-Server Encryption through YSQL API. #1845
  • [YSQL] Check constraints when doing UPDATE. #2361
  • [YSQL] Add default password to yugabyte user for ysqlsh. #2594
  • [YSQL] authentication and authorization. #2610
  • [YSQL] client authentication update. #2614
  • [YSQL] Enable FOR SHARE and FOR KEY SHARE row locking in YSQL SELECT statements. DMLs on tables with foreign key references work regardless of isolation level (previously only worked in SERIALIZABLE isolation)#1199
  • [YSQL] Avoid namespace id resolving for import_namespace yb-admin command. #2582
  • [YSQL] SERIALIZABLE READ ONLY DEFERRABLE. #2161

YCQL changes

  • [YCQL] Fix for SIGBUS TS crash on invalid statement. #2476
  • [YCQL] A role shouldn't be able to modify the superuser status of itself or of a role granted to it. #2505
  • [YCQL] Fix issue with permissions inheritance. #2550
  • [YCQL] Mangling the column name in IndexTable for index selecting processing. #741
  • [YCQL] Wrong port sent for NEW_NODE topology change events. #2607
  • [YCQL] Fixed: Cannot CREATE TYPE when user has CREATE permissions for keyspace. #2556
  • [YCQL] Add support for IF clause in SELECT statement. #2640
  • [YCQL] Add description for JSONB INDEX and IF clause in SELECT. #2651
  • [YCQL] Not reserving keyword USER and WHEN. #1661
  • [YCQL] EXPLAIN SELECT causes tserver to crash. #2616

System improvements

  • Added missing extractor checks into version_edit, so ldb doesn't crash. #2510
  • Fix race in Batcher. #2501
  • [2DC] Update checkpoint for replicated records, ignore tablet status while looking up peer. #2487
  • Reduce log level while skipping over display of stats for tablets with state RaftGroupStatePB::SHUTDOWN. #1488
  • [2DC] Send from_checkpoint only when consumer knows the last checkpoint. #2522
  • Remove may exist state from YBTransaction. #2501
  • [CDC] Setup Java Console Connector for CDC. #2081
  • Fix operation memory double tracking. #1436
  • Reinit total replicas to move when master leader fails during tserver blacklisting. #2453
  • Display num sst files for tablets in yb_admin list_tablets_for_tablet_server. #2536
  • Use boost::amotic for op_id_copy_ member (128 bits structure). #2545
  • Derive proxy rpc addresses. #1023
  • Update consensus meta term during local bootstrap. #2570
  • Add metrics for thread count in each category. #1785
  • Cleanup queue on timeout. #2592
  • Do not call LogReader::Open() twice when skip_wal_rewrite is enabled. #2124
  • Display num log segments for tablets in yb_admin list_tablets_for_tablet_server. #2555
  • Fix Bug in YEDIS Keys Command. #2381
  • Fix num tablets per table. #2630
  • modify_placement_info clears read replica config. #2628
  • Fix use of load balancer concurrent action limits. #2624
  • Node should be able to transition from master to shell mode and back to master. #2007
  • Use new linuxbrew (20191015T202549). #2608
  • Remove in_memory_erase option and use SingleDelete instead of Delete. #2569
  • Fixed invalid read-time for RF==1 case. #2394
  • Move Rpcs to CDC consumer from CDC poller. #2549
  • [CDC] Make redirection to tablet leaders for GetChanges asynchronous. #2503
Assets 2
Oct 28, 2019
Add v2.0.2 tag

@rao-vasireddy rao-vasireddy released this Oct 9, 2019 · 240 commits to master since this release

Release highlight

YSQL is now generally available and is ready for production deployments. Details available in Announcing YugabyteDB 2.0 GA: Jepsen Tested, High-Performance Distributed SQL.

YSQL changes

  • [YSQL] Allow ALTER COLUMN add/set/drop identity. #2226
  • [YSQL] PostgreSQL error code handling to avoid string-based error comparisons. #1289
  • [YSQL] Fix duplicate key error being suppressed in COPY. #2289
  • [YSQL] VERIFY_RESULT creates undesirable copy of Result<T&>'s value. #2298
  • [YSQL] Warn on non-txn COPY with incompatible relation. #2278
  • [YSQL] Allow ALTER COLUMN add/set/drop identity. #2226
  • [YSQL] PostgreSQL error code handling to avoid string-based error comparisons. #1289
  • [YSQL] Aggregate pushdowns in YSQL. #1851
  • [YSQL] Skip dropped columns when iterating through the columns for UPDATE statement. #1969
  • [YSQL] Use keyspace id instead of name in requests from yb-admin to catalog_manager. #1476
  • [YSQL] Fix insert into temp table in function body assertion error. #1541
  • [YSQL] Fix memory leak in psql. #2388
  • [YSQL] Add default password for yugabyte user. #2465

YCQL changes

  • [YCQL] Fixed permission check for DROP TYPE. #2373
  • [YCQL] Fix for 'value too long' error code mapping. #2116
  • [YCQL] 'expression not yet supported' error for old-format index update. #2470

System improvements

  • Do not retry flush/compact async tasks. #2229
  • Make yb-admin log an error message instead of fatal-ing on failed init. #2141
  • Propagate timeout to transaction. #2334
  • Warn on non-txn COPY with incompatible relation. #2278
  • [2DC] Add Exponential Backoff to CDC Consumer. #2362
  • Add support for cotable id in EnumerateIntents (needed for transactional DDL). #1404
  • TServer blacklisting for leadership role. #1748
  • Avoid unnecessary RocksDB lookup when we have multiple ops to one tablet. #2186
  • [2DC] Reduce checkpoint frequency, avoid self looping in CDC producer, apply polling delay correctly. #2364
  • Request intents cleanup even on failed replicas. #2411
  • Add number of sst files to tables and tablets pages for tserver. #2230
  • Fix role and term cache. #2267
  • yb-admin add RF to add_read_replicas. #2395
  • [2DC] Fix transaction status and record handling in CDC producer/consumer. #2386
  • Handle concurrent shutdown and remote bootstrap. #2428
  • [CDC] Fix bug in condition check for tablet leader. #2449
  • Fix remote bootstrap for encryption at rest. #2462
  • Implement dynamic priorities for compaction tasks. #2004
  • [2DC] Update checkpoint for replicated records, ignore tablet status while looking up peer. #2487
  • [2DC] yb-admin command to pause/resume replication. #2367
Assets 2
Oct 8, 2019
Add v1.3.2 tag

@rao-vasireddy rao-vasireddy released this Sep 17, 2019 · 393 commits to master since this release

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v2.0.0. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v2.0.0 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v2.0.0, create a new cluster and then load the previously saved data.

YSQL Changes

  • [YSQL] Add CREATE TABLE tablet splits to grammar.#1486
  • [YSQL] Add catch-all ysql configuration gflags.#1140
  • [YSQL] Allow ALTER TABLE ALTER COLUMN SET|DROP NOT NULL. #1200
  • [YSQL] Add retry logic to pgbench. #2058
  • [YSQL] Optimize single-row UPDATE/DELETE statements. #1028
  • [YSQL] IN predicate pushdown. #1554
  • [YSQL] Add top-level ysql configuration gflags. #1140
  • [YSQL] Add support for various ALTER TABLE ADD COLUMN variants. #2055
  • [YSQL] Support UDTs, records. #1676
  • [YSQL] Order range columns with respect to primary key descripttion. #1250
  • [YSQL] Support ALTER TABLE ADD COLUMN with default value. #2055
  • [YSQL] Enable ANALYZE, VACUUM in grammar. #2217
  • [YSQL] Non-transactional COPY in YSQL. #1926
  • [YSQL] CREATE/DROP AGGREGATE, CAST, OPERATOR, OPERATOR CLASS. #1981
  • [YSQL] Create yugabyte, system_platform as default database. #2277

YCQL Changes

  • [YCQL] Fixed TS crash if a built-in function argument is a column. #1944

System Improvements

  • Propagate intents apply failure to client.#1974
  • Check shutdown state while scheduling compaction.#2100
  • Add leader check on master TS Write interface. #1995
  • Correctly handle duplicate key in postgres layer. #2012
  • Remove StopWatch from RocksDB Seek. #2136
  • Avoid picking read time for the serializable transactions. #2125
  • CDC: Producer and consumer master handshake to setup 2DC. #1977
  • Optimized Drop Table Locking. #2056
  • Reduce number of sys_catalog writes: skip no change heartbeats. #1966
  • CDC: Consumer changes. #1481
  • CDC: Add master master support. #1824
  • Increase WAL retention time when CDC stream is setup on table. #2050
  • Apply changes received from producer universe. #2153
  • CDC: Delete for DROP DATABASE. #2114
  • Show Load Balancer state in Master UI. #646
  • Add support for error code categories to Status. #2200
  • [2DC] Handle rpc controller errors and CDC consumer shutdown. #2231
  • Don't apply transactions on tablets without intents. #2070
  • Allow dynamic priorities in PriorityThreadPool. #2004
  • Added rocksdb_universal_compaction_always_include_size_threshold. #2085
  • Do full table name match while finding matching tables in 2DC. #2250
  • Fix cleaning Raft log after transactional writes are stopped. #2221
  • Fix handling of calls expired in queue. #2269
  • Load balancer should not be considered idle when tablet remote bootstrap. #2193
  • [YSQ]: Fix number of tablets created for system tables. #2293
  • Reduce fan-out and use tablet invoker for writing data.
  • SerializableReadWriteConflict needs better success probability.
Assets 3

@rao-vasireddy rao-vasireddy released this Aug 8, 2019 · 669 commits to master since this release

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.3.1. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.3.1 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.3.1, create a new cluster and then load the previously saved data.

YSQL Changes

  • [YSQL] Fix error in nextval_internal retry.#1783
  • [YSQL] Support bulk inserts for COPY FROM command.#1590
  • [YSQL] Support for Triggers and Extensions.#1156
  • [YSQL] Simplify gflags enabling YSQL.#1691
  • [YSQL] Support NULLS LAST for SELECT.#1734

System Improvements

  • Extra cleanup of intents of aborted transactions to prevent repeated log messages.#1722
  • Cleanup checkpoints on startup.#498
  • Unified yb::RandomAccessFile and rocksdb::RandomAccessFile.#1661
  • Clear Pending Delete State on RPC Failure.#424
  • Do not crash yb-master process if we fail to find a namespace when listing tables.#1716
  • Allow consecutive master restarts to succeed.#1074
  • Change soft memory limit handling.#1437
  • Use log prefix in MvccManager.#1835
  • Fix handling timeout of a local call in the service.#1866
  • Fixed TS crash due to NULL name in the PTBindVar object.#1821
  • Add /tables view to tablet server ui.#1830
  • Add ability to avoid using the compaction thread pool for flushes.#1874
  • Tablet leader tracking at catalog manager.#1881
  • Allow in memory erase of intents.#1240
  • Fix assignment order of propagated_safe_time and propagated.#1889
  • Memory monitor should not flush tablet that is already flushing.#1880
  • Fixed potential race condition in memory-based flush.#1859
Assets 2

@rao-vasireddy rao-vasireddy released this Jul 16, 2019 · 2483 commits to master since this release

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.3.0. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.3.0 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.3.0, create a new cluster and then load the previously saved data.

YSQL Changes

  • [YSQL] Enable DO statements (a.k.a anonymous blocks).#1656
  • [YSQL] Fix transaction read restart logic causing stale reads and lost writes.#1646
  • [YSQL] Enable beta-features by default.#1639
  • [YSQL] Fix value overwrite check in REPEATABLE READ isolation mode in YSQL.#1577
  • [YSQL] Allow SELECT from sequences.#999
  • [YSQL] Avoid auto-retrying non-DML requests.#1333
  • [YSQL] Set the read point correctly at REPEATABLE READ isolation when turning a read-only transaction into a read-write one.
  • [YSQL] Add new key column ybindexkeysuffix to YSQL index table [format change].#1058
  • [YSQL] Add support for setval().#1003
  • [YSQL] YSQL catalog cache consistency checking via tablet server shared memory.#1457
  • [YSQL] Fix persistence issue with catalog version.#1629

YCQL Changes

  • Added built-in function toJson() for YCQL SELECT statement.#992

System Improvements

  • Add Master UI page for tracking async tasks.#1145
  • Use rw_spinlock in meta cache to decrease lock contention.#1667
  • Change default maximum per-tablet RPC batch size to 32mb.#1440
  • Fix for flushing old tablets based on memory usage.#1672
  • Replaced OS_LINUX with linux for ifdefs.#1711
  • Unified yb::SequentialFile and rocksdb::SequentialFile.#1661
  • Add a compact_table command to yb-admin.#1735
  • Automatically filter out YSQL sys catalog tables from YBClient calls.#1537
  • Add TLS support to java yb-client.
  • Allow pausing a compaction when another compaction with a higher priority is scheduled.#1314
  • Fixed Tablet::Truncate to re-enable auto compactions.1765
  • Fix race condition in PriorityThreadPool.#1775
Assets 2

@rao-vasireddy rao-vasireddy released this Jul 10, 2019 · 2697 commits to master since this release

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.2.12. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.12 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.12, create a new cluster and then load the previously saved data.

YSQL Changes

  • [YSQL] Fix scan operator when range key is not fully-specified.#1576
  • [YSQL] Fix ORDER BY range column of a secondary index.#1591
  • [YSQL] Add YSQL RPCs to tserver utilz page.#1446

System Improvements

  • Resolve ABA problem in LockFreeStackResolve ABA problem in LockFreeStack.#1561
  • Fix recursive locking in Batcher.#1584
  • Use boost::atomic for 16-bytes structures.#1189
  • Fix integer overflow during Raft log read path.#1606
  • Fix concurrent access to primary table info in RaftGroupMetadata.#1631
  • A new priority-aware thread pool implementation for compactions and flushes.#1314
  • Create mem trackers that reflects TC Malloc stats.#1633
  • Make sure we don't read an invalid replicate index from the footer.#1625

Performance Benchmarks

This section gives an overview of the performance of this release. The benchmarks were done on the following setup:

  • 3 nodes with 16-vCPUs
  • Replication factor = 3
  • Each key-value is ~64 bytes (combined)
  • See setup details for more info

Summary

API Workload Type Throughput Latency Avg CPU on DB node
YCQL KeyValue 100% Reads 218K 1.17ms 48%
YCQL KeyValue 100% Writes 73K 3.5ms 44%
YCQL Batched Writes 100% Writes 247K 26ms 52%
YCQL Secondary Index 100% Reads 187K 1.4ms 44%
YCQL Secondary Index 100% Writes 5K 12.9ms 51%

Details

YCQL KeyValue

	Workload CassandraKeyValue with 256 writers and 0 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Read latency, ms/op            | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Write throughput, ops/sec      | 73037.05  | 68884.95  | 74102.43  | 1211.67   | 73480.55  | 68884.95  | 68884.95  | 71993.35  | 74051.57  | 74102.43  | 74102.43  |
| Write latency, ms/op           | 3.50      | 3.45      | 3.72      | 0.06      | 3.48      | 3.45      | 3.45      | 3.45      | 3.55      | 3.72      | 3.72      |
| Load tester CPU, user, %       | 16.91     | 14.90     | 18.10     | 0.88      | 17.30     | 14.90     | 14.91     | 15.18     | 17.70     | 18.08     | 18.10     |
| Load tester CPU, system, %     | 9.20      | 7.60      | 10.10     | 0.83      | 9.65      | 7.60      | 7.60      | 7.72      | 9.90      | 10.09     | 10.10     |
| Cluster node CPU, user, %      | 32.93     | 23.80     | 38        | 3.45      | 32.45     | 23.80     | 24.31     | 30.13     | 37.20     | 37.49     | 38        |
| Cluster node CPU, system, %    | 15.43     | 12.70     | 16.60     | 0.82      | 15.40     | 12.70     | 13.22     | 14.43     | 16.30     | 16.50     | 16.60     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 192 writers and 16 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 18493.93  | 17156.62  | 19190.41  | 587.15    | 18606.10  | 17156.62  | 17156.62  | 17440.30  | 19151.33  | 19190.41  | 19190.41  |
| Read latency, ms/op            | 0.87      | 0.83      | 0.93      | 0.03      | 0.86      | 0.83      | 0.83      | 0.83      | 0.92      | 0.93      | 0.93      |
| Write throughput, ops/sec      | 60571.82  | 58544.09  | 62509.31  | 983.23    | 60550.72  | 58544.09  | 58544.09  | 59035.93  | 62380.73  | 62509.31  | 62509.31  |
| Write latency, ms/op           | 3.17      | 3.07      | 3.28      | 0.05      | 3.17      | 3.07      | 3.07      | 3.08      | 3.25      | 3.28      | 3.28      |
| Load tester CPU, user, %       | 17.68     | 16        | 20.70     | 0.87      | 17.55     | 16        | 16.06     | 17.20     | 18.39     | 20.58     | 20.70     |
| Load tester CPU, system, %     | 8.73      | 8.10      | 10.40     | 0.54      | 8.50      | 8.10      | 8.11      | 8.22      | 9.58      | 10.36     | 10.40     |
| Cluster node CPU, user, %      | 35.62     | 32.90     | 38.80     | 1.71      | 34.95     | 32.90     | 33.41     | 33.72     | 38.10     | 38.39     | 38.80     |
| Cluster node CPU, system, %    | 15.38     | 14.90     | 15.80     | 0.19      | 15.40     | 14.90     | 15.01     | 15.10     | 15.60     | 15.70     | 15.80     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 128 writers and 64 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 71301.76  | 64468.86  | 78149.80  | 6074.55   | 69364.13  | 64468.86  | 64468.86  | 64563.72  | 78080.94  | 78149.80  | 78149.80  |
| Read latency, ms/op            | 0.90      | 0.82      | 0.99      | 0.08      | 0.92      | 0.82      | 0.82      | 0.82      | 0.99      | 0.99      | 0.99      |
| Write throughput, ops/sec      | 42974.61  | 40464.12  | 45473.25  | 1883.46   | 42860     | 40464.12  | 40464.12  | 40927.76  | 45053.88  | 45473.25  | 45473.25  |
| Write latency, ms/op           | 2.98      | 2.82      | 3.16      | 0.13      | 2.98      | 2.82      | 2.82      | 2.84      | 3.13      | 3.16      | 3.16      |
| Load tester CPU, user, %       | 21.46     | 19.10     | 23.30     | 1.28      | 21.60     | 19.10     | 19.15     | 20        | 23.08     | 23.29     | 23.30     |
| Load tester CPU, system, %     | 10.17     | 9.20      | 10.90     | 0.56      | 10.30     | 9.20      | 9.21      | 9.31      | 10.80     | 10.89     | 10.90     |
| Cluster node CPU, user, %      | 38.31     | 28.30     | 41.60     | 3.01      | 39.80     | 28.30     | 34.10     | 34.34     | 41.09     | 41.20     | 41.60     |
| Cluster node CPU, system, %    | 13.11     | 11.10     | 14        | 0.51      | 13.15     | 11.10     | 11.66     | 12.80     | 13.50     | 13.60     | 14        |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 64 writers and 128 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 113115.50 | 111032.59 | 114690.37 | 1211.91   | 113593.04 | 111032.59 | 111032.59 | 111280.88 | 114639.78 | 114690.37 | 114690.37 |
| Read latency, ms/op            | 1.13      | 1.11      | 1.15      | 0.01      | 1.13      | 1.11      | 1.11      | 1.12      | 1.15      | 1.15      | 1.15      |
| Write throughput, ops/sec      | 21349.23  | 20717.49  | 21753.92  | 264.27    | 21335.10  | 20717.49  | 20717.49  | 20915.45  | 21699.52  | 21753.92  | 21753.92  |
| Write latency, ms/op           | 3.00      | 2.94      | 3.09      | 0.04      | 3         | 2.94      | 2.94      | 2.95      | 3.06      | 3.09      | 3.09      |
| Load tester CPU, user, %       | 23.54     | 21.80     | 27.90     | 1.24      | 23.50     | 21.80     | 21.81     | 22.03     | 24.36     | 27.72     | 27.90     |
| Load tester CPU, system, %     | 10.43     | 9.60      | 12.10     | 0.53      | 10.45     | 9.60      | 9.61      | 9.71      | 10.79     | 12.03     | 12.10     |
| Cluster node CPU, user, %      | 38.33     | 34.30     | 42.90     | 3.03      | 37.40     | 34.30     | 34.62     | 35.21     | 42.60     | 42.89     | 42.90     |
| Cluster node CPU, system, %    | 11.41     | 10.30     | 12.40     | 0.52      | 11.30     | 10.30     | 10.60     | 10.80     | 12.10     | 12.20     | 12.40     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 0 writers and 256 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 218803.58 | 209206.20 | 225392    | 4068.91   | 218849.97 | 209206.20 | 209206.20 | 213317.10 | 224702.67 | 225392    | 225392    |
| Read latency, ms/op            | 1.17      | 1.14      | 1.22      | 0.02      | 1.17      | 1.14      | 1.14      | 1.14      | 1.20      | 1.22      | 1.22      |
| Write throughput, ops/sec      | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Write latency, ms/op           | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Load tester CPU, user, %       | 31.19     | 13        | 34.70     | 4.52      | 32.20     | 13        | 13        | 30.10     | 33.10     | 34.70     | 34.70     |
| Load tester CPU, system, %     | 13.48     | 5.70      | 16.20     | 2.08      | 13.70     | 5.70      | 5.70      | 12.80     | 15.70     | 16.20     | 16.20     |
| Cluster node CPU, user, %      | 36.82     | 17.10     | 46.40     | 6.92      | 34.90     | 17.10     | 20.36     | 31.92     | 45.70     | 45.90     | 46.40     |
| Cluster node CPU, system, %    | 7.30      | 5.60      | 9.70      | 1.54      | 6.50      | 5.60      | 5.70      | 5.70      | 9.50      | 9.50      | 9.70      |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

YCQL Secondary Index

	Workload CassandraSecondaryIndex with 64 writers and 0 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Read latency, ms/op            | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Write throughput, ops/sec      | 4957.63   | 4654.57   | 5466.63   | 190.85    | 4935.75   | 4654.57   | 4654.57   | 4701.88   | 5270.11   | 5466.63   | 5466.63   |
| Write latency, ms/op           | 12.92     | 11.71     | 13.75     | 0.48      | 12.96     | 11.71     | 11.71     | 12.15     | 13.61     | 13.75     | 13.75     |
| Load tester CPU, user, %       | 2.23      | 2         | 3.20      | 0.28      | 2.10      | 2         | 2         | 2         | 2.64      | 3.20      | 3.20      |
| Load tester CPU, system, %     | 1.55      | 1.40      | 1.80      | 0.11      | 1.50      | 1.40      | 1.40      | 1.40      | 1.80      | 1.80      | 1.80      |
| Cluster node CPU, user, %      | 32.02     | 29.60     | 33.70     | 0.96      | 32.20     | 29.60     | 29.70     | 30.62     | 33.26     | 33.38     | 33.70     |
| Cluster node CPU, system, %    | 19.25     | 17.80     | 20        | 0.52      | 19.30     | 17.80     | 18.06     | 18.42     | 19.88     | 19.94     | 20        |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraSecondaryIndex with 32 writers and 64 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 59031.00  | 57817.73  | 62446.65  | 1109.89   | 58745.42  | 57817.73  | 57817.73  | 58043.71  | 60768.14  | 62446.65  | 62446.65  |
| Read latency, ms/op            | 1.08      | 1.02      | 1.11      | 0.02      | 1.09      | 1.02      | 1.02      | 1.06      | 1.10      | 1.11      | 1.11      |
| Write throughput, ops/sec      | 3131.73   | 3076.68   | 3206.08   | 31.21     | 3127.82   | 3076.68   | 3076.68   | 3089.73   | 3176.39   | 3206.08   | 3206.08   |
| Write latency, ms/op           | 10.22     | 9.98      | 10.40     | 0.10      | 10.23     | 9.98      | 9.98      | 10.08     | 10.36     | 10.40     | 10.40     |
| Load tester CPU, user, %       | 14.97     | 14.20     | 15.70     | 0.41      | 15        | 14.20     | 14.20     | 14.52     | 15.62     | 15.70     | 15.70     |
| Load tester CPU, system, %     | 8.25      | 7.20      | 8.60      | 0.34      | 8.30      | 7.20      | 7.20      | 7.60      | 8.60      | 8.60      | 8.60      |
| Cluster node CPU, user, %      | 37.01     | 34.70     | 40.10     | 1.55      | 36.50     | 34.70     | 34.92     | 35.22     | 39.18     | 39.32     | 40.10     |
| Cluster node CPU, system, %    | 14.41     | 13.80     | 14.80     | 0.26      | 14.50     | 13.80     | 13.90     | 14.02     | 14.80     | 14.80     | 14.80     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraSecondaryIndex with 16 writers and 128 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 104821.67 | 99481.21  | 120241.63 | 7320.25   | 101330.40 | 99481.21  | 99481.21  | 99546.21  | 119939.73 | 120241.63 | 120241.63 |
| Read latency, ms/op            | 1.23      | 1.06      | 1.29      | 0.08      | 1.26      | 1.06      | 1.06      | 1.07      | 1.29      | 1.29      | 1.29      |
| Write throughput, ops/sec      | 1949.49   | 1929.03   | 1990.20   | 17.51     | 1943.61   | 1929.03   | 1929.03   | 1931.10   | 1978.04   | 1990.20   | 1990.20   |
| Write latency, ms/op           | 8.21      | 8.04      | 8.29      | 0.07      | 8.23      | 8.04      | 8.04      | 8.09      | 8.28      | 8.29      | 8.29      |
| Load tester CPU, user, %       | 19.35     | 18.10     | 22.90     | 1.18      | 19        | 18.10     | 18.10     | 18.26     | 21.06     | 22.90     | 22.90     |
| Load tester CPU, system, %     | 8.89      | 8.40      | 10.30     | 0.59      | 8.60      | 8.40      | 8.40      | 8.40      | 10.06     | 10.30     | 10.30     |
| Cluster node CPU, user, %      | 40.96     | 36.50     | 44.20     | 2.31      | 41.90     | 36.50     | 36.72     | 37.20     | 43.18     | 43.62     | 44.20     |
| Cluster node CPU, system, %    | 11.82     | 11.10     | 12.30     | 0.33      | 11.90     | 11.10     | 11.10     | 11.20     | 12.20     | 12.24     | 12.30     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraSecondaryIndex with 0 writers and 256 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 187239.42 | 185195.56 | 188133.96 | 804.53    | 187287.10 | 185195.56 | 185195.56 | 185805.85 | 188109.85 | 188133.96 | 188133.96 |
| Read latency, ms/op            | 1.37      | 1.36      | 1.38      | 0.01      | 1.37      | 1.36      | 1.36      | 1.36      | 1.38      | 1.38      | 1.38      |
| Write throughput, ops/sec      | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Write latency, ms/op           | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Load tester CPU, user, %       | 19.78     | 0.90      | 29.40     | 11.76     | 26.90     | 0.90      | 0.99      | 1.50      | 27.87     | 29.17     | 29.40     |
| Load tester CPU, system, %     | 8.55      | 0.70      | 14        | 4.88      | 11.45     | 0.70      | 0.74      | 1         | 11.80     | 13.67     | 14        |
| Cluster node CPU, user, %      | 34.88     | 12.30     | 48.50     | 9.85      | 36.65     | 12.30     | 16.62     | 22.21     | 48.20     | 48.36     | 48.50     |
| Cluster node CPU, system, %    | 8.70      | 4.30      | 17.20     | 4.21      | 7.70      | 4.30      | 4.50      | 4.67      | 16.56     | 16.93     | 17.20     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

YCSB

+--------------+----------------------+--------------------+--------------------+--------------------+
| Workload     | Throughput (ops/sec) | Op Type            | Avg Latency (us)   | 99th% Latency (us) |
| workloada    | 75054.10             | READ               | 2527.53            | 20895              |
|              |                      | UPDATE             | 4032.19            | 25391              |
|              |                      |                    |                    |                    |
| workloadb    | 89653.56             | READ               | 2711.76            | 7967               |
|              |                      | UPDATE             | 3391.83            | 8983               |
|              |                      |                    |                    |                    |
| workloadc    | 89456.49             | READ               | 2753.47            | 7939               |
|              |                      |                    |                    |                    |
| workloadd    | 87712.48             | READ               | 2765.26            | 8327               |
|              |                      | INSERT             | 3598.85            | 9671               |
|              |                      |                    |                    |                    |
| workloade    | 16485.62             | INSERT             | 14684.71           | 70847              |
|              |                      | SCAN               | 14963.57           | 70271              |
|              |                      |                    |                    |                    |
| workloadf    | 49282.21             | READ               | 2956.77            | 20815              |
|              |                      | READ-MODIFY-WRITE  | 7039.35            | 29743              |
|              |                      | UPDATE             | 4074.65            | 23871              |
|              |                      |                    |                    |                    |
+--------------+----------------------+--------------------+--------------------+--------------------+
Assets 2

@rkarthik007 rkarthik007 released this Apr 17, 2019 · 3606 commits to master since this release

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.2.4. There are two ways to work around this problem:

  • If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.4 and create a new cluster.
  • If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.4, create a new cluster and then load the previously saved data.

YSQL Changes

  • Added support for primary keys. It is now possible to create a table with a primary key. #955
  • Added support for temporary tables. #805
  • Added support for CYCLE option in CREATE SEQUENCE statement. #993
  • Enabled SET TRANSACTION READ WRITE statement. #1081
  • Support RelationType in TableInfo to distinguish between system tables, user tables and user indexes. #906

Bug Fixes

  • Fixed a crash in tserver that is observed while inserting null value into an indexed column. #945
  • Fixed a bug where indexes on system catalog tables were not being updated during DDL. #1083
  • Fixed detection of single row transactions on tables that have primary keys.
  • Fixed MiniMaster startup race condition in test infrastructure. #1040

System Improvements

  • Reduce per-connection overhead for cache preloading. #1090 #1091
  • Faster release of sending buffers to reduce memory consumption when leader sends large updates to followers who are lagging behind. #1079
  • Parallelize DNS resolution while filling system.peers and system.local tables #1000
  • Script for post-processing test results and generating JSON-based result files.
  • Improve stability of QLTransactionTest.IntentsCleanupAfterRestart. #1102

Key Commits

  • [f2ecaa1] [YSQL] Fix single-row-txn detection after primary key change
  • [90680ca] [#1102]: Improve stability of QLTransactionTest.IntentsCleanupAfterRestart
  • [a4c614f] JSON-based test reports
  • [33835b0] [#1079]: Release sending buffers as soon as possible
  • [1bc77da] ENG-4682 (#906): Support RelationType (user table, index, system table) in TableInfo
  • [1dedcd1] ENG-4943 #945 [YSQL] Correct handling for NULL value in indexed column
  • [465b6d6] [YSQL] #1090 #1091 Reduce per-connection overhead for cache preloading
  • [382b89c] [YSQL] ENG-4601 (#805): Add support for temporary tables
  • [5165f01] #1081 [YSQL] Enable READ WRITE transaction
  • [9f31c41] ENG-4927: #1040: Fix MiniMaster startup race condition
  • [be0908b] [YSQL] #955: Handle primary key as an index
  • [4b48e25] ENG-4872: #1000: Parallel DNS resolution while filling system.peers and system.local tables
  • [4096f59] ENG-4900: Add external mini cluster test with TLS support
  • [34ee0fc] [YSQL] #1083: Handle sys catalog index updates
  • [ac52116] #993: [YSQL] Add support for CYCLE option in CREATE SEQUENCE
  • [3149f36] ENG-4363, #668: Support JSON operator on INSERT (Part 1 - Refactoring)
  • [a3ea483]
  • [b3a60ee] [yb-docker-ctl] Save initdb log (#1087)
  • [b131fbc] ENG-4888: #1031: Make YCQL index name optional
  • [9d333f7] #1064: Fix data race with YQL permissions cache
  • [4a56fe1] #1038: Change FLAGS_remote_boostrap_rate_limit_bytes_per_sec to FLAGS_remote_bootstrap_rate_limit_bytes_per_sec
  • [56a6029] #752: Add TLS encryption support to yb-admin.
  • [d9b6385] [YSQL] #1066: Fix memory leak in ResourceOwner
  • [fb4c1a2] Added yb-docker-ctl to CE (#1077)
  • [046b4b6] ENG-4855: #977: Handle restart read in YSQL
  • [30725bf] ENG-4343: #655: Pick transaction read time at tablet server when possible
  • [0341b29] [YSQL] #1067: Buffer create index write operations only for initdb
  • [78ca75a] [#1050]: Add Tablet Server capabilities
  • [9d724d5] ENG-4790: Removing most of yb-loadtester from repo due to migration to standalone yb-sample-apps repo
  • [bbcc67b] ENG-4891: Removing the SQL RPCs button.
  • [14279cc] [YCQL] #1042: Fix ASAN issue in YCQL executor restart
  • [cc3584f] ENG-4800: #952: Use call buffer for receive from socket
  • [8715be3] [YSQL] #1051: Bulk insert in initdb
  • [987abc6] ENG-4877 (#1009): yb-ctl enhancements
  • [0f261a9] [YSQL] 1049 ASAN reported memory leaks in Postgres
  • [9eaf15b] Add a new pattern for NFS directories: /Volumes/net
  • [3a61698] [#1041]: Skip dumping of finished call
  • [a322d95] [YSQL] #711 Add support for all datatypes
  • [7e2c89c] [#1030]: Add transactional retry writes tests 2) Second
  • [request]licated. Fixed by returning AlreadyPresent status and handling it in
  • [callbac] not handled correctly. For instance when request cannot be processed because of service shutdown, we could stop retrying to apply the operation. Fixed by correct handling of such errors.
  • [509c5a2] #921: Fixed: [YCQL] improve error messages related to index DDL
  • [e48b7da] ENG-4468: Always perform only one server rpc call for readiness check.
  • [0f8fdd9] [YSQL] ENG-4831 (#969): Enable drop cascade
  • [85836b5] Allow spaces around equality sign in @RunWith(value=SomeClass) annotation
  • [e533efc] ENG-4844, #949: Treat TUPLE as a keyword
  • [40dd31f] [#984]: Allow running C++ test by specifying only gtest_filter
  • [9cb4f10] ENG-4746: #925: Implement change_blacklist & get_universe_config commands in yb-admin
  • [9e28eee] [YSQL] #995: Raise error on unsupported WHERE condition with secondary index
  • [9fd097e] [#965] Reduce the number of concurrent connections to YB masters and tservers from PostgreSQL backend processes
  • [7ea3cd0] [YSQL] ENG-4839 (#976): Handle postgres session termination correctly in YB mode
  • [f55f1c0] #991: Fixed TestRpc.TestDisconnect
  • [aa8c4ec] #999: [YSQL] SELECTing directly from a sequence throws an error
  • [8d226ce] ENG-4852: Do not pass Datum data types to ybc_pgate
  • [f702d72] [YSQL] #979: Fix ORDER BY query using secondary index
  • [66a94f3] ENG-4840 (#987): Fix incorrect count for system tablet leader count
  • [c847562] #964: Change compaction and remote bootstrap rate defaults.
  • [5ed041c] #788 Optimize select statements where a clustering-key needs to be in a given range
  • [0c00885] [#933]: Leader lease revocation
  • [03b996d] ENG-4746: #925: Migrate missing java yb-cli functionality to yb-admin (Preparation)
Assets 2
You can’t perform that action at this time.