From a452a5472e169d620d2d05c70be84fbd56b6cc7c Mon Sep 17 00:00:00 2001 From: Sid Choudhury Date: Mon, 27 May 2019 15:00:35 -0700 Subject: [PATCH] misc updates to intro, compatibility, yb-ctl and ysql-spring --- content/latest/admin/yb-ctl.md | 174 ++++++++++++++---- content/latest/admin/yb-docker-ctl.md | 13 +- content/latest/admin/ysqlsh.md | 2 +- .../latest/develop/build-apps/java/ycql.md | 2 +- .../develop/build-apps/java/ysql-jdbc.md | 2 +- .../build-apps/java/ysql-spring-data.md | 25 ++- content/latest/faq/compatibility.md | 11 +- content/latest/introduction.md | 2 +- .../latest/manage/diagnostics-reporting.md | 2 +- 9 files changed, 168 insertions(+), 65 deletions(-) diff --git a/content/latest/admin/yb-ctl.md b/content/latest/admin/yb-ctl.md index ec320072..d9013bb9 100644 --- a/content/latest/admin/yb-ctl.md +++ b/content/latest/admin/yb-ctl.md @@ -20,6 +20,52 @@ Use the **-\-help** option to see all the commands supported. ```sh $ ./bin/yb-ctl --help ``` +```sh +usage: yb-ctl [-h] [--binary_dir BINARY_DIR] [--data_dir DATA_DIR] + [--replication_factor REPLICATION_FACTOR] + [--num_shards_per_tserver NUM_SHARDS_PER_TSERVER] + [--timeout TIMEOUT] [--verbose] [--install-if-needed] + {create,start,stop,destroy,restart,wipe_restart,add_node,remove_node,start_node,stop_node,restart_node,status,setup_redis} + ... + +positional arguments: + {create,start,stop,destroy,restart,wipe_restart,add_node,remove_node,start_node,stop_node,restart_node,status,setup_redis} + create Create a new cluster + start Create a new cluster, or start existing cluster if it + already exists. + stop Stops the cluster + destroy Destroy the current cluster + restart Restart the current cluster all at once + wipe_restart Stop the cluster, wipe all data files and start the + cluster asbefore. Will lose all the flags though. + add_node Add a new node to the current cluster + remove_node Stop a particular node in the cluster. + start_node Start a particular node with flags. + stop_node Stop a particular node in the cluster + restart_node Restart the node specified. + status Get info on the current cluster processes + setup_redis Setup YugaByte to support Redis API + +optional arguments: + -h, --help show this help message and exit + --binary_dir BINARY_DIR + Specify a custom directory in which to find the + yugabyte binaries. + --data_dir DATA_DIR Specify a custom directory where to store data. + --replication_factor REPLICATION_FACTOR, --rf REPLICATION_FACTOR + Replication factor for the cluster as well as default + number of masters. + --num_shards_per_tserver NUM_SHARDS_PER_TSERVER + Number of shards (tablets) to start per tablet server + for each table. + --timeout TIMEOUT Timeout in seconds for operations that wait on the + cluster + --verbose If specified, will log internal debug messages to + stderr. + --install-if-needed With this option, if YugaByte DB is not yet installed + on the system, the latest version will be downloaded + and installed automatically. +``` Here are the default values for all the optional arguments. @@ -31,11 +77,12 @@ Optional Argument | Default | Description `--require_clock_sync`| `false` | Tells YugaByte DB whether to depend on clock synchronization between the nodes in the cluster `--num_shards_per_tserver`| `2` | Number of shards (tablets) per tablet server for each table -## Creating a cluster + +## Create a cluster The `create` cluster command is used to create a cluster. -The number of nodes created with the initial create command is always equal to the replication factor in order to ensure that all the replicas for a given tablet can be placed on different nodes. Use the [add_node](#adding-nodes) and [remove_node](#stopping-removing-nodes) commands to expand or shrink the cluster. +The number of nodes created with the initial create command is always equal to the replication factor in order to ensure that all the replicas for a given tablet can be placed on different nodes. Use the [add_node](#add-nodes) and [remove_node](#stop-remove-nodes) commands to expand or shrink the cluster. Each of these initial nodes run a `yb-tserver` process and a `yb-master` process. Note that the number of yb-masters in a cluster has to equal the replication factor for the cluster to be considered operating normally. @@ -61,15 +108,70 @@ Add a node to make it a 4 node cluster. $ ./bin/yb-ctl add_node ``` - - Creating a 5 node cluster with replication factor 5. ```sh $ ./bin/yb-ctl --rf 5 create ``` +## Default cluster configuration + +### Data directory + +Cluster data is installed in `$HOME/yugabyte-data/`. + +#### Node directories + +`yugabyte-data/node-#/` directory created for node #. + +This directory contains the following. + +```sh +yugabyte-data/node-#/disk-#/ +initdb.log +cluster_config.json +``` + +#### Disk directories + +`yugabyte-data/node-#/disk-#/` directory created for each disk. + +This directory contains the following. + +```sh +yugabyte-data/node-#/disk-#/pg_data/ +yugabyte-data/node-#/disk-#/yb-data/ +``` + +### Logs + +yb-master logs are located at + +```sh +yugabyte-data/node-#/disk-#/yb-data/master.out +yugabyte-data/node-#/disk-#/yb-data/master/logs +``` + +yb-tserver logs are located at + +```sh +yugabyte-data/node-#/disk-#/yb-data/tserver.out +yugabyte-data/node-#/disk-#/yb-data/tserver/logs +``` + +## Start and stop an existing cluster + +Create a new cluster, or start an existing cluster if it already exists. +```sh +$ ./bin/yb-ctl start +``` + +Stop a cluster so that you can start it later. +```sh +$ ./bin/yb-ctl stop +``` -## Checking cluster status +## Check cluster status You can get the status of the local cluster including the URLs for the admin UIs for the YB-Master and YB-TServer using the `status` command. @@ -77,19 +179,17 @@ You can get the status of the local cluster including the URLs for the admin UIs $ ./bin/yb-ctl status ``` +## Initialize the YEDIS API -## Initializing the YEDIS API - -The `setup_redis` command to initialize YugaByte's Redis-compatible YEDIS API. +The `setup_redis` command to initialize YugaByte DB's Redis-compatible YEDIS API. ```sh $ ./bin/yb-ctl setup_redis ``` +## Add and remove nodes -## Adding and removing nodes - -### Adding nodes +### Add nodes - Adding a new node to the cluster. This will start a new yb-tserver process and give it a new `node_id` for tracking purposes. @@ -97,22 +197,19 @@ $ ./bin/yb-ctl setup_redis $ ./bin/yb-ctl add_node ``` +### Stop/remove nodes -### Stopping/removing nodes - -We can Stop a cluster node by executing the `stop` command. The command takes the node id of the node -that has to be removed as input. Stop node command expects a node id which denotes the index of the server that -needs to be stopped. It also takes an optional flag `--master` which denotes that the server is a +We can stop a node by executing the `stop` command. The command takes the node id of the node +that has to be removed as input. Stop node command expects a node id which denotes the index of the server that needs to be stopped. It also takes an optional flag `--master` which denotes that the server is a master. ```sh $ ./bin/yb-ctl stop_node 4 ``` - At this point of time `remove_node` and `stop_node` do the same thing. So they can be used interchangeably. -## Destroying a cluster +## Destroy a cluster You can use the `destroy` command to destroy a cluster. This command stops all the nodes and deletes the data directory of the cluster. @@ -121,9 +218,11 @@ deletes the data directory of the cluster. $ ./bin/yb-ctl destroy ``` + + ## Advanced commands -### Creating a local cluster across multiple zones, regions and clouds +### Create a cluster across multiple zones, regions and clouds You can pass the placement information for nodes in a cluster from the command line. The placement information is provided as a set of (cloud, region, zone) tuples separated by commas. Each cloud, region and zone entry is separated by dots. @@ -139,7 +238,7 @@ To add a node: $ ./bin/yb-ctl add_node --placement_info "cloud1.region1.zone1" ``` -### Creating a local cluster with custom flags +### Create a cluster with custom flags You can also pass custom flags to the masters and tservers. ```sh @@ -158,7 +257,7 @@ To add a node with custom master flags: $ ./bin/yb-ctl add_node --master_flags "log_cache_size_limit_mb=128,log_min_seconds_to_retain=20" ``` -### Restarting a cluster +### Restart a cluster The `restart` command can be used to restart a cluster. Please note that if you restart the cluster, all custom defined flags and placement information will be lost. Nevertheless, you can pass the @@ -168,43 +267,39 @@ placement information and custom flags in the same way as they are passed in the $ ./bin/yb-ctl restart ``` -Restarting with cloud, region and zone flags: +- Restart with cloud, region and zone flags ```sh -$ ./bin/yb-ctl wipe_restart --placement_info "cloud1.region1.zone1" +$ ./bin/yb-ctl restart --placement_info "cloud1.region1.zone1" ``` -Restarting with custom flags: +- Restart with custom flags ```sh -$ ./bin/yb-ctl wipe_restart --master_flags "log_cache_size_limit_mb=128,log_min_seconds_to_retain=20,master_backup_svc_queue_length=70" --tserver_flags "log_inject_latency=false,log_segment_size_mb=128,raft_heartbeat_interval_ms=1000" +$ ./bin/yb-ctl restart --master_flags "log_cache_size_limit_mb=128,log_min_seconds_to_retain=20,master_backup_svc_queue_length=70" --tserver_flags "log_inject_latency=false,log_segment_size_mb=128,raft_heartbeat_interval_ms=1000" ``` -### Restarting a node +### Restart a node -The `restart` first stops the node and then starts it again(essentially restarting it). At this point of time the node is not decommissioned from the cluster. -Thus one of the primary advantages of this command is that it can be used to wipe out old flags and pass in new ones. Just like -create, you can pass the cloud/region/zone and custom flags in the `restart` command. +The `restart` first stops the node and then starts it again. At this point of time the node is not decommissioned from the cluster. Thus one of the primary advantages of this command is that it can be used to wipe out old flags and pass in new ones. Just like create, you can pass the cloud/region/zone and custom flags in the `restart` command. ```sh $ ./bin/yb-ctl restart_node 2 ``` -- Restarting node with placement info: +- Restart node with placement info ```sh $ ./bin/yb-ctl restart_node 2 --placement_info "cloud1.region1.zone1" ``` - -- Restarting master node: +- Restart master node ```sh $ ./bin/yb-ctl restart_node 2 --master ``` - -- Restarting node with flags: +- Restart node with flags ```sh $ ./bin/yb-ctl restart_node 2 --master --master_flags "log_cache_size_limit_mb=128,log_min_seconds_to_retain=20" @@ -212,25 +307,24 @@ $ ./bin/yb-ctl restart_node 2 --master --master_flags "log_cache_size_limit_mb=1 ### Wipe and restart a cluster -We can use the `wipe_restart` command for this. This command stops all the nodes, removes the underlying data directort, then starts back the same -number of nodes that you had in your previous configuration. +This command stops all the nodes, removes the underlying data directories, then starts back the same number of nodes that you had in your previous configuration. -Just like the `restart` command the custom defined flags and placement information will be lost during `wipe_restart`, -though you can pass placement information and custom flags in the same way as they are passed in the -`create` command. +Just like the `restart` command the custom defined flags and placement information will be lost during `wipe_restart`, though you can pass placement information and custom flags in the same way as they are passed in the `create` command. ```sh $ ./bin/yb-ctl wipe_restart ``` -Wipe and restart with placement info flags: +- Wipe and restart with placement info flags ```sh $ ./bin/yb-ctl wipe_restart --placement_info "cloud1.region1.zone1" ``` -Wipe and restart with custom flags: +- Wipe and restart with custom flags ```sh $ ./bin/yb-ctl wipe_restart --master_flags "log_cache_size_limit_mb=128,log_min_seconds_to_retain=20,master_backup_svc_queue_length=70" --tserver_flags "log_inject_latency=false,log_segment_size_mb=128,raft_heartbeat_interval_ms=1000" ``` + + diff --git a/content/latest/admin/yb-docker-ctl.md b/content/latest/admin/yb-docker-ctl.md index 748767be..58b54ac4 100644 --- a/content/latest/admin/yb-docker-ctl.md +++ b/content/latest/admin/yb-docker-ctl.md @@ -25,7 +25,6 @@ $ mkdir ~/yugabyte && cd ~/yugabyte $ wget https://downloads.yugabyte.com/yb-docker-ctl && chmod +x yb-docker-ctl ``` - ## Help command Use the **-\-help** option to see all the commands supported. @@ -35,17 +34,23 @@ $ ./yb-docker-ctl -h ``` ``` -usage: yb-docker-ctl [-h] {create,status,destroy,add_node,remove_node} ... +usage: yb-docker-ctl [-h] + {create,add_node,status,destroy,stop_node,start_node,stop,start,remove_node} + ... YugaByte Docker Container Control positional arguments: - {create,status,destroy,add_node,remove_node} + {create,add_node,status,destroy,stop_node,start_node,stop,start,remove_node} Commands create Create YugaByte Cluster + add_node Add a new YugaByte Cluster Node status Check YugaByte Cluster status destroy Destroy YugaByte Cluster - add_node Add a new YugaByte Cluster Node + stop_node Stop a YugaByte Cluster Node + start_node Start a YugaByte Cluster Node + stop Stop YugaByte Cluster so that it can be started later + start Start YugaByte Cluster if one already exists remove_node Stop a YugaByte Cluster Node optional arguments: diff --git a/content/latest/admin/ysqlsh.md b/content/latest/admin/ysqlsh.md index 4ad26772..5875def7 100644 --- a/content/latest/admin/ysqlsh.md +++ b/content/latest/admin/ysqlsh.md @@ -25,7 +25,7 @@ ysqlsh is installed as part of YugaByte DB and is located in the bin directory o ## Example ```sh -$ ./bin/ysqlsh --echo-queries +$ ./bin/ysqlsh ``` ``` diff --git a/content/latest/develop/build-apps/java/ycql.md b/content/latest/develop/build-apps/java/ycql.md index c7ee5275..d46c5c7e 100644 --- a/content/latest/develop/build-apps/java/ycql.md +++ b/content/latest/develop/build-apps/java/ycql.md @@ -23,7 +23,7 @@ showAsideToc: true
  • - YSQL - Spring Data with JPA/Hibernate + YSQL - Spring Data JPA with Hibernate
  • diff --git a/content/latest/develop/build-apps/java/ysql-jdbc.md b/content/latest/develop/build-apps/java/ysql-jdbc.md index 311975a1..ebff1cfe 100644 --- a/content/latest/develop/build-apps/java/ysql-jdbc.md +++ b/content/latest/develop/build-apps/java/ysql-jdbc.md @@ -28,7 +28,7 @@ showAsideToc: true
  • - YSQL - Spring Data with JPA/Hibernate + YSQL - Spring Data JPA with Hibernate
  • diff --git a/content/latest/develop/build-apps/java/ysql-spring-data.md b/content/latest/develop/build-apps/java/ysql-spring-data.md index 5064aee0..5fb39ffe 100644 --- a/content/latest/develop/build-apps/java/ysql-spring-data.md +++ b/content/latest/develop/build-apps/java/ysql-spring-data.md @@ -24,7 +24,7 @@ showAsideToc: true
  • - YSQL - Spring Data with JPA/Hibernate + YSQL - Spring Data JPA with Hibernate
  • @@ -120,7 +120,7 @@ $ curl \ ### Using the YSQL shell ```sh -$ ./bin/ysqlsh -h 127.0.0.1 -p 5433 -U postgres --echo-queries +$ ./bin/ysqlsh ``` ``` ysqlsh (11.2) @@ -132,7 +132,6 @@ postgres=# postgres=> SELECT count(*) FROM users; ``` ``` -SELECT count(*) FROM users; count ------- 2 @@ -143,7 +142,6 @@ SELECT count(*) FROM users; postgres=> SELECT count(*) FROM products; ``` ``` -SELECT count(*) FROM products; count ------- 2 @@ -154,14 +152,13 @@ SELECT count(*) FROM products; postgres=> SELECT count(*) FROM orders; ``` ``` -SELECT count(*) FROM orders; count ------- 2 (1 row) ``` -### Using the REST API endpoint +### Using the REST API ```sh $ curl http://localhost:8080/users @@ -170,13 +167,13 @@ $ curl http://localhost:8080/users { "content": [ { - "userId": 3, + "userId": 2, "firstName": "Tom", "lastName": "Stewart", "email": "tstewart@yb.com" }, { - "userId": 2, + "userId": 1, "firstName": "John", "lastName": "Smith", "email": "jsmith@yb.com" @@ -221,9 +218,9 @@ $ curl http://localhost:8080/orders "orderId": "999ae272-f2f4-46a1-bede-5ab765bb27fe", "user": { "userId": 2, - "firstName": "John", - "lastName": "Smith", - "email": "jsmith@yb.com" + "firstName": "Tom", + "lastName": "Stewart", + "email": "tstewart@yb.com" }, "userId": null, "orderTotal": 25, @@ -234,9 +231,9 @@ $ curl http://localhost:8080/orders "orderId": "1598c8d4-1857-4725-a9ab-14deb089ab4e", "user": { "userId": 2, - "firstName": "John", - "lastName": "Smith", - "email": "jsmith@yb.com" + "firstName": "Tom", + "lastName": "Stewart", + "email": "tstewart@yb.com" }, "userId": null, "orderTotal": 15, diff --git a/content/latest/faq/compatibility.md b/content/latest/faq/compatibility.md index a8810d83..a1e2cc81 100644 --- a/content/latest/faq/compatibility.md +++ b/content/latest/faq/compatibility.md @@ -13,11 +13,18 @@ isTocNested: true showAsideToc: true --- +## What does API compatibility exactly mean? + +API compatibility refers to the fact that the database APIs offered by YugaByte DB servers implement the same wire protocol and modeling/query language as that of an existing database. Since client drivers, command line shells, IDE integrations and other ecosystem integrations of the existing database rely on this wire protocol and modeling/query language, they are expected to work with YugaByte DB without major modifications. + +For example, the [YSQL](../../api/ysql) API is compatible with PostgreSQL. This means PostgreSQL client drivers, psql command line shell, IDE integrations such as TablePlus and DBWeaver and more can be used with YugaByte DB. The same concept applies to [YCQL](../../api/ycql) in the context of the Apache Cassandra Query Language. + ## Why are YugaByte DB APIs compatible with popular DB languages? - YugaByte DB's API compatibility is aimed at accelerating developer onboarding. By integrating well with the existing ecosystem, YugaByte DB ensures that developers can get started easily using a language they are already comfortable with. -- YugaByte DB's API compatibility is not aimed at lift-and-shift porting of existing applications written for the original language. This is because existing applications are most likely not written to take advantage of the distributed SQL APIs provided by YugaByte DB. For such existing applications, developers should expect to modify their previously monolithic PostgreSQL and/or non-transactional Cassandra data access logic as they look to migrate to YugaByte DB. +- YugaByte DB's API compatibility is not aimed at lift-and-shift porting of existing applications written for the original language. This is because existing applications are not written to take advantage of the distributed SQL APIs provided by YugaByte DB. For such existing applications, developers should expect to modify their previously monolithic PostgreSQL and/or non-transactional Cassandra data access logic as they look to migrate to YugaByte DB. + ## YSQL Compatibility with PostgreSQL @@ -34,7 +41,7 @@ As highlighted in [Distributed PostgreSQL on a Google Spanner Architecture – Q YugaByte DB's goal is to remain as compatible with PostgreSQL as possible. If you see a feature currently missing, please file a [GitHub issue](https://github.com/YugaByte/yugabyte-db/issues) for us. -## YCQL Compatibility with Apache Cassandra +## YCQL Compatibility with Apache Cassandra QL ### Do INSERTs do “upserts” by default? How do I insert data only if it is absent? diff --git a/content/latest/introduction.md b/content/latest/introduction.md index 0caf5dbd..79f49cde 100644 --- a/content/latest/introduction.md +++ b/content/latest/introduction.md @@ -22,7 +22,7 @@ showAsideToc: true {{< vimeo 305074082 >}} -YugaByte DB is a high-performance SQL database for powering internet-scale, globally-distributed applications. Built using a unique combination of high-performance document store, auto sharding, per-shard distributed consensus replication and multi-shard ACID transactions (inspired by Google Spanner), YugaByte DB serves both scale-out RDBMS and internet-scale OLTP workloads with low query latency, extreme resilience against failures and global data distribution. As a cloud native database, it can be deployed across public and private clouds as well as in Kubernetes environments with ease. +YugaByte DB is a high-performance distributed SQL database for powering global, internet-scale applications. Built using a unique combination of high-performance document store, auto sharding, per-shard distributed consensus replication and multi-shard ACID transactions (inspired by Google Spanner), YugaByte DB serves both scale-out RDBMS and internet-scale OLTP workloads with low query latency, extreme resilience against failures and global data distribution. As a cloud native database, it can be deployed across public and private clouds as well as in Kubernetes environments with ease. YugaByte DB Community Edition is developed and distributed as an [Apache 2.0 open source project](https://github.com/YugaByte/yugabyte-db/). diff --git a/content/latest/manage/diagnostics-reporting.md b/content/latest/manage/diagnostics-reporting.md index 10bd2896..230336ab 100644 --- a/content/latest/manage/diagnostics-reporting.md +++ b/content/latest/manage/diagnostics-reporting.md @@ -113,7 +113,7 @@ You can add the following flags while starting the [yb-master](../../admin/yb-ma Flag | Default | Description ----------------------|---------|------------------------ -`--callhome_collection_level` | `low` | Collection level with possible values of `low`, `medium`, or `high` +`--callhome_collection_level` | `medium` | Collection level with possible values of `low`, `medium`, or `high` `--callhome_interval_secs` | 3600 | Collection interval in seconds `--callhome_url ` | `http://diagnostics.yugabyte.com` | Endpoint where diagnostics information is reported `--callhome_enabled` | `true` | Controls whether diagnostics information is collected and reported. Set to `false` to disable collection.