From 7dde36edfa5f25bba14ea5694e7d84da51bcf564 Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 24 Nov 2015 14:41:34 +0900 Subject: [PATCH 01/12] Don't use dash --- tajo-docs/src/main/sphinx/table_management/tablespaces.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst index 79ea65f1f5..00077a0984 100644 --- a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst +++ b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst @@ -29,7 +29,7 @@ The following is an example for two tablespaces for hbase and hdfs: { "spaces": { - "hbase-cluster1": { + "hbase_cluster1": { "uri": "hbase://quorum1:port,quorum2:port/" }, From 559ab3e44ca6f5f71aef11c5128fbcbd33906ce8 Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 24 Nov 2015 15:01:03 +0900 Subject: [PATCH 02/12] storage-site.json document is added --- tajo-docs/src/main/sphinx/configuration.rst | 7 ++- .../configuration/storage-site-json.rst | 57 +++++++++++++++++++ .../sphinx/table_management/tablespaces.rst | 5 +- 3 files changed, 64 insertions(+), 5 deletions(-) create mode 100644 tajo-docs/src/main/sphinx/configuration/storage-site-json.rst diff --git a/tajo-docs/src/main/sphinx/configuration.rst b/tajo-docs/src/main/sphinx/configuration.rst index 5214ea8cb6..8b0b8a2e66 100644 --- a/tajo-docs/src/main/sphinx/configuration.rst +++ b/tajo-docs/src/main/sphinx/configuration.rst @@ -1,6 +1,6 @@ -************************ +************* Configuration -************************ +************* .. toctree:: :maxdepth: 1 @@ -13,4 +13,5 @@ Configuration configuration/ha_configuration configuration/service_config_defaults configuration/tajo-site-xml - configuration/catalog-site-xml \ No newline at end of file + configuration/catalog-site-xml + configuration/storage-site-json diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst new file mode 100644 index 0000000000..39f4e560ad --- /dev/null +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -0,0 +1,57 @@ +************************** +The storage-site.json File +************************** + +This file is for configuring :doc:`table spaces`. +The syntax of storage-site.json is like this basically: + +.. code:: json + + { + "spaces": { + , , ... + }, + "storages": { + , , ... + } + } + +------ +Spaces +------ + +This is a section for registering table spaces. Some space config example is here: + +.. code:: json + + "spaces": { + "hbase_cluster1": { + "uri": "hbase://quorum1:port,quorum2:port/", + "config": { ... } + }, + ... + } + +* space name : Your own table space name which indicates a specific table space. Alpha-numeric characters and underscore(_) are permitted. +* uri : An URI address of a table space +* config : It is optional. You can specify it as JSON object to pass to each table space handler. + +If you specify a table space, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_. + +-------- +Storages +-------- + +This is for registering storage format and custom storage handler class. +Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to use ``storages`` configuration. +Anyway for your knowledge, here is one storage conf example(it's quoted from Tajo internal hdfs storage configuration which is already being used). + +.. code:: json + + "storages": { + "hdfs": { + "handler": "org.apache.tajo.storage.FileTablespace", + "default-format": "text" + }, + ... + } diff --git a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst index 00077a0984..eb0a7ebd15 100644 --- a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst +++ b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst @@ -1,9 +1,10 @@ -************************************* +*********** Tablespaces -************************************* +*********** Tablespaces in Tajo allow users to define locations in the storage system where the files or data objects representing database objects can be stored. Once defined, a tablespace can be referred to by name when creating a database or a table. Especially, it is very useful when a Tajo cluster instance should use heterogeneous storage systems such as HDFS, MySQL, and Oracle. +============= Configuration ============= From 715f19fb9f5bf918107f662001fe66045539be6a Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 24 Nov 2015 15:05:18 +0900 Subject: [PATCH 03/12] Refined --- .../src/main/sphinx/configuration/storage-site-json.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index 39f4e560ad..0155a738ce 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -32,11 +32,11 @@ This is a section for registering table spaces. Some space config example is her ... } -* space name : Your own table space name which indicates a specific table space. Alpha-numeric characters and underscore(_) are permitted. -* uri : An URI address of a table space -* config : It is optional. You can specify it as JSON object to pass to each table space handler. +* **space name** : Your own table space name which indicates a specific table space. Alpha-numeric characters and underscore(_) are permitted. +* **uri** : An URI address of a table space +* **config** : It is optional. You can specify it as JSON object to pass to each table space handler. -If you specify a table space, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_. +After you specify a table space, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_. -------- Storages From d50ec1e69b970ef00d71d1d0c77c73d4250f49ac Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 24 Nov 2015 15:06:08 +0900 Subject: [PATCH 04/12] refined more --- tajo-docs/src/main/sphinx/configuration/storage-site-json.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index 0155a738ce..014ff9501b 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -3,7 +3,7 @@ The storage-site.json File ************************** This file is for configuring :doc:`table spaces`. -The syntax of storage-site.json is like this basically: +The syntax of ``storage-site.json`` is like this basically: .. code:: json From 5fb14e4fe52de7a38f48bf506b50af78986e9f5b Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 24 Nov 2015 16:05:58 +0900 Subject: [PATCH 05/12] I think copyright should be current year. --- tajo-docs/src/main/sphinx/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/conf.py b/tajo-docs/src/main/sphinx/conf.py index f6a2d968e6..dbd9e14061 100644 --- a/tajo-docs/src/main/sphinx/conf.py +++ b/tajo-docs/src/main/sphinx/conf.py @@ -58,7 +58,7 @@ # General information about the project. project = u'Apache Tajo' -copyright = u'2014, Apache Tajo Team' +copyright = u'2015, Apache Tajo Team' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the From a0138e80f5e34467b838c6e59bf9401cb6f31943 Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Wed, 25 Nov 2015 10:39:39 +0900 Subject: [PATCH 06/12] Link to storage-site.json doc is added --- .../src/main/sphinx/table_management/tablespaces.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst index eb0a7ebd15..f521162525 100644 --- a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst +++ b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst @@ -8,9 +8,11 @@ Tablespaces in Tajo allow users to define locations in the storage system where Configuration ============= -By default, Tajo use in ``${tajo.rootdir}/warehouse`` in ``conf/tajo-site.xml`` as a default tablespace. It also allows users to register additional tablespaces. +By default, Tajo use in ``${tajo.rootdir}/warehouse`` in :doc:`conf/tajo-site.xml` as a default tablespace. It also allows users to register additional tablespaces like below. -``conf/storage-site.json`` file. +--------------------------- +conf/storage-site.json file +--------------------------- The configuration file has the following struct: @@ -40,6 +42,8 @@ The following is an example for two tablespaces for hbase and hdfs: } } +For more tails, see :doc:`conf/storage-site.json`. + .. note:: From 8109c365a24a5c2d14e741ddbb58b0259d3f6a8e Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Wed, 25 Nov 2015 10:42:26 +0900 Subject: [PATCH 07/12] some words added --- tajo-docs/src/main/sphinx/table_management/tablespaces.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst index f521162525..af0abfa771 100644 --- a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst +++ b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst @@ -8,7 +8,7 @@ Tablespaces in Tajo allow users to define locations in the storage system where Configuration ============= -By default, Tajo use in ``${tajo.rootdir}/warehouse`` in :doc:`conf/tajo-site.xml` as a default tablespace. It also allows users to register additional tablespaces like below. +By default, Tajo use in ``${tajo.rootdir}/warehouse`` in :doc:`conf/tajo-site.xml` as a default tablespace. It also allows users to register additional tablespaces using ``storage-site.json`` file like below. --------------------------- conf/storage-site.json file From 3fc21d8c880f1cc42e2ea025704d76f653678c10 Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Thu, 26 Nov 2015 12:14:59 +0900 Subject: [PATCH 08/12] * example is fixed * typo is fixed --- .../configuration/storage-site-json.rst | 24 ++++++++++++------- .../sphinx/table_management/tablespaces.rst | 4 ++-- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index 014ff9501b..cf208763d2 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -2,7 +2,7 @@ The storage-site.json File ************************** -This file is for configuring :doc:`table spaces`. +This file is for configuring :doc:`/table_management/tablespaces`. The syntax of ``storage-site.json`` is like this basically: .. code:: json @@ -20,23 +20,29 @@ The syntax of ``storage-site.json`` is like this basically: Spaces ------ -This is a section for registering table spaces. Some space config example is here: +This is a section for registering tablespaces. Some space config example is here: .. code:: json "spaces": { - "hbase_cluster1": { - "uri": "hbase://quorum1:port,quorum2:port/", - "config": { ... } + "jdbc_mysql": { + "uri": "jdbc:mysql://127.0.0.1:3306", + "configs": { + "mapped_database":"mydb", + "connection_properties": { + "username":"tajouser", + "password":"123456" + } + } }, ... } -* **space name** : Your own table space name which indicates a specific table space. Alpha-numeric characters and underscore(_) are permitted. -* **uri** : An URI address of a table space -* **config** : It is optional. You can specify it as JSON object to pass to each table space handler. +* **space name** : Your own tablespace name which indicates a specific tablespace. Alpha-numeric characters and underscore(_) are permitted. +* **uri** : An URI address of a tablespace +* **configs** : It is optional. You can specify it as JSON object to pass to each tablespace handler. -After you specify a table space, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_. +After you specify a tablespace, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_. -------- Storages diff --git a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst index af0abfa771..a60fd2c69d 100644 --- a/tajo-docs/src/main/sphinx/table_management/tablespaces.rst +++ b/tajo-docs/src/main/sphinx/table_management/tablespaces.rst @@ -20,7 +20,7 @@ The configuration file has the following struct: { "spaces": { - "${table_space_name}": { + "${tablespace_name}": { "uri": "hbase://quorum1:port,quorum2:port/" } } @@ -42,7 +42,7 @@ The following is an example for two tablespaces for hbase and hdfs: } } -For more tails, see :doc:`conf/storage-site.json`. +For more details, see :doc:`conf/storage-site.json`. .. note:: From 534bdae8ce7e9343a358c6bb77a705c5e214c590 Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Fri, 27 Nov 2015 12:33:43 +0900 Subject: [PATCH 09/12] Example changed to pgsql --- .../src/main/sphinx/configuration/storage-site-json.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index cf208763d2..23aa5a7313 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -25,12 +25,12 @@ This is a section for registering tablespaces. Some space config example is here .. code:: json "spaces": { - "jdbc_mysql": { - "uri": "jdbc:mysql://127.0.0.1:3306", + "jdbc_pgsql": { + "uri": "jdbc:postgresql://127.0.0.1:5740/origin", "configs": { - "mapped_database":"mydb", + "mapped_database":"tajo_pgsql_origin", "connection_properties": { - "username":"tajouser", + "user":"tajouser", "password":"123456" } } From 3364135c316bd3ae12c10ac750c8ae4a750ecd3d Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 1 Dec 2015 12:02:56 +0900 Subject: [PATCH 10/12] Comments applied --- .../src/main/sphinx/configuration/storage-site-json.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index 23aa5a7313..e61fc3f3cb 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -49,8 +49,9 @@ Storages -------- This is for registering storage format and custom storage handler class. -Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to use ``storages`` configuration. -Anyway for your knowledge, here is one storage conf example(it's quoted from Tajo internal hdfs storage configuration which is already being used). +Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to add ``storages`` configuration. +However, if you want to use your custom storage as one of Tajo's data source, you need to add a configuration for your storage. Here is an example of HDFS storage. +See `:doc:storage_plugins/overview` for more information. .. code:: json From 673dfc68f015e8fe8f4bf9321b5820750501a57e Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 1 Dec 2015 12:03:35 +0900 Subject: [PATCH 11/12] missing word added --- tajo-docs/src/main/sphinx/configuration/storage-site-json.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index e61fc3f3cb..8f3f4d0cc0 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -49,7 +49,7 @@ Storages -------- This is for registering storage format and custom storage handler class. -Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to add ``storages`` configuration. +Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to add any ``storages`` configuration. However, if you want to use your custom storage as one of Tajo's data source, you need to add a configuration for your storage. Here is an example of HDFS storage. See `:doc:storage_plugins/overview` for more information. From 880512d7de7b221ce6246066d178157f87190f2a Mon Sep 17 00:00:00 2001 From: Jongyoung Park Date: Tue, 1 Dec 2015 12:24:15 +0900 Subject: [PATCH 12/12] typo fixed --- tajo-docs/src/main/sphinx/configuration/storage-site-json.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst index 8f3f4d0cc0..d37755181d 100644 --- a/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst +++ b/tajo-docs/src/main/sphinx/configuration/storage-site-json.rst @@ -51,7 +51,7 @@ Storages This is for registering storage format and custom storage handler class. Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to add any ``storages`` configuration. However, if you want to use your custom storage as one of Tajo's data source, you need to add a configuration for your storage. Here is an example of HDFS storage. -See `:doc:storage_plugins/overview` for more information. +See :doc:`storage_plugins/overview` for more information. .. code:: json