Skip to content

Commit

Permalink
Release 9.0 (#884)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jonels-msft
jonels-msft committed Dec 2, 2019
1 parent b9455cc commit 8ea8d6e
Show file tree
Hide file tree
Showing 15 changed files with 123 additions and 254 deletions.
138 changes: 21 additions & 117 deletions admin_guide/upgrading_citus.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,21 @@ Upgrading the Citus version requires first obtaining the new Citus extension and
Patch Version Upgrade
---------------------

To upgrade a Citus version to its latest patch, issue a standard upgrade command for your package manager. Assuming version 8.3 is currently installed on Postgres 11:
To upgrade a Citus version to its latest patch, issue a standard upgrade command for your package manager. Assuming version 9.0 is currently installed on Postgres 11:

**Ubuntu or Debian**

.. code-block:: bash
sudo apt-get update
sudo apt-get install --only-upgrade postgresql-11-citus-8.3
sudo apt-get install --only-upgrade postgresql-11-citus-9.0
sudo service postgresql restart
**Fedora, CentOS, or Red Hat**

.. code-block:: bash
sudo yum update citus82_11
sudo yum update citus90_11
sudo service postgresql-11.0 restart
.. _major_minor_upgrade:
Expand All @@ -48,22 +48,22 @@ Each major and minor version of Citus is published as a package with a separate
Step 1. Update Citus Package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When doing a **major** version upgrade instead, be sure to upgrade the Citus extension first, and the PostgreSQL version second (see :ref:`upgrading_postgres`). Here is how to do a **minor** upgrade from 8.2 to 8.3:
If upgrading both Citus and Postgres, always be sure to upgrade the Citus extension first, and the PostgreSQL version second (see :ref:`upgrading_postgres`). Here is how to do a Citus upgrade from 8.3 to 9.0:

**Ubuntu or Debian**

.. code-block:: bash
sudo apt-get update
sudo apt-get install postgresql-11-citus-8.3
sudo apt-get install postgresql-11-citus-9.0
sudo service postgresql restart
**Fedora, CentOS, or Red Hat**

.. code-block:: bash
# Fedora, CentOS, or Red Hat
sudo yum swap citus82_11 citus83_11
sudo yum swap citus83_11 citus90_11
sudo service postgresql-11 restart
Step 2. Apply Update in DB
Expand Down Expand Up @@ -94,52 +94,37 @@ After installing the new package and restarting the database, run the extension

.. _upgrading_postgres:

Upgrading PostgreSQL version from 10 to 11
Upgrading PostgreSQL version from 11 to 12
##########################################

.. note::

Do not attempt to upgrade *both* Citus and Postgres versions at once. If both upgrades are desired, upgrade Citus first.

Also **Citus 7.x is not compatible with Postgres 11.** Before upgrading Postgres 10 to 11, be sure to follow the above steps to upgrade from Citus 7.x to 8.3.

Record the following paths before you start (your actual paths may be different than those below):

Existing data directory (e.g. /opt/pgsql/10/data)
:code:`export OLD_PG_DATA=/opt/pgsql/10/data`
:code:`export OLD_PG_DATA=/opt/pgsql/11/data`

Existing PostgreSQL installation path (e.g. /usr/pgsql-10)
:code:`export OLD_PG_PATH=/usr/pgsql-10`
:code:`export OLD_PG_PATH=/usr/pgsql-11`

New data directory after upgrade
:code:`export NEW_PG_DATA=/opt/pgsql/11/data`
:code:`export NEW_PG_DATA=/opt/pgsql/12/data`

New PostgreSQL installation path
:code:`export NEW_PG_PATH=/usr/pgsql-11`
:code:`export NEW_PG_PATH=/usr/pgsql-12`

On Every Node (Coordinator and workers)
---------------------------------------
For Every Node
--------------

1. Back up Citus metadata in the old server.
1. Back up Citus metadata in the old coordinator node.

.. code-block:: postgres
CREATE TABLE public.pg_dist_partition AS
SELECT * FROM pg_catalog.pg_dist_partition;
CREATE TABLE public.pg_dist_shard AS
SELECT * FROM pg_catalog.pg_dist_shard;
CREATE TABLE public.pg_dist_placement AS
SELECT * FROM pg_catalog.pg_dist_placement;
CREATE TABLE public.pg_dist_node_metadata AS
SELECT * FROM pg_catalog.pg_dist_node_metadata;
CREATE TABLE public.pg_dist_node AS
SELECT * FROM pg_catalog.pg_dist_node;
CREATE TABLE public.pg_dist_local_group AS
SELECT * FROM pg_catalog.pg_dist_local_group;
CREATE TABLE public.pg_dist_transaction AS
SELECT * FROM pg_catalog.pg_dist_transaction;
CREATE TABLE public.pg_dist_colocation AS
SELECT * FROM pg_catalog.pg_dist_colocation;
-- this step for the coordinator node only, not workers
SELECT citus_prepare_pg_upgrade();
2. Configure the new database instance to use Citus.

Expand All @@ -149,7 +134,7 @@ On Every Node (Coordinator and workers)
shared_preload_libraries = 'citus'
* **DO NOT CREATE** Citus extension yet
* **DO NOT CREATE** Citus extension

* **DO NOT** start the new server

Expand Down Expand Up @@ -178,91 +163,10 @@ On Every Node (Coordinator and workers)

* **DO NOT** run any query before running the queries given in the next step

7. Restore metadata.

.. code-block:: postgres
INSERT INTO pg_catalog.pg_dist_partition
SELECT * FROM public.pg_dist_partition;
INSERT INTO pg_catalog.pg_dist_shard
SELECT * FROM public.pg_dist_shard;
INSERT INTO pg_catalog.pg_dist_placement
SELECT * FROM public.pg_dist_placement;
INSERT INTO pg_catalog.pg_dist_node_metadata
SELECT * FROM public.pg_dist_node_metadata;
INSERT INTO pg_catalog.pg_dist_node
SELECT * FROM public.pg_dist_node;
TRUNCATE TABLE pg_catalog.pg_dist_local_group;
INSERT INTO pg_catalog.pg_dist_local_group
SELECT * FROM public.pg_dist_local_group;
INSERT INTO pg_catalog.pg_dist_transaction
SELECT * FROM public.pg_dist_transaction;
INSERT INTO pg_catalog.pg_dist_colocation
SELECT * FROM public.pg_dist_colocation;
8. Drop temporary metadata tables.

.. code-block:: postgres
DROP TABLE public.pg_dist_partition;
DROP TABLE public.pg_dist_shard;
DROP TABLE public.pg_dist_placement;
DROP TABLE public.pg_dist_node_metadata;
DROP TABLE public.pg_dist_node;
DROP TABLE public.pg_dist_local_group;
DROP TABLE public.pg_dist_transaction;
DROP TABLE public.pg_dist_colocation;
9. Restart sequences.
7. Restore metadata on new coordinator node.

.. code-block:: postgres
SELECT setval('pg_catalog.pg_dist_shardid_seq', (SELECT MAX(shardid)+1 AS max_shard_id FROM pg_dist_shard), false);
SELECT setval('pg_catalog.pg_dist_placement_placementid_seq', (SELECT MAX(placementid)+1 AS max_placement_id FROM pg_dist_placement), false);
SELECT setval('pg_catalog.pg_dist_groupid_seq', (SELECT MAX(groupid)+1 AS max_group_id FROM pg_dist_node), false);
SELECT setval('pg_catalog.pg_dist_node_nodeid_seq', (SELECT MAX(nodeid)+1 AS max_node_id FROM pg_dist_node), false);
SELECT setval('pg_catalog.pg_dist_colocationid_seq', (SELECT MAX(colocationid)+1 AS max_colocation_id FROM pg_dist_colocation), false);
10. Register triggers.

.. code-block:: postgres
CREATE OR REPLACE FUNCTION create_truncate_trigger(table_name regclass) RETURNS void LANGUAGE plpgsql as $$
DECLARE
command text;
trigger_name text;
BEGIN
trigger_name := 'truncate_trigger_' || table_name::oid;
command := 'create trigger ' || trigger_name || ' after truncate on ' || table_name || ' execute procedure pg_catalog.citus_truncate_trigger()';
execute command;
command := 'update pg_trigger set tgisinternal = true where tgname
= ' || quote_literal(trigger_name);
execute command;
END;
$$;
SELECT create_truncate_trigger(logicalrelid) FROM pg_dist_partition ;
DROP FUNCTION create_truncate_trigger(regclass);
11. Set dependencies.

.. code-block:: postgres
-- this step for the coordinator node only, not workers
INSERT INTO
pg_depend
SELECT
'pg_class'::regclass::oid as classid,
p.logicalrelid::regclass::oid as objid,
0 as objsubid,
'pg_extension'::regclass::oid as refclassid,
(select oid from pg_extension where extname = 'citus') as refobjid,
0 as refobjsubid ,
'n' as deptype
FROM
pg_dist_partition p;
SELECT citus_finish_pg_upgrade();
6 changes: 3 additions & 3 deletions conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,9 @@
# built documents.
#
# The short X.Y version.
version = '8.3'
version = '9.0'
# The full version, including alpha/beta/rc tags.
release = '8.3'
release = '9.0'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand All @@ -77,7 +77,7 @@

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
exclude_patterns = ['.venv/*']

# The reST default role (used for this markup: `text`) to use for all
# documents.
Expand Down
7 changes: 5 additions & 2 deletions develop/api_guc.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -298,12 +298,15 @@ Sets the frequency at which Citus checks for statuses of jobs managed by the tas
citus.task_executor_type (enum)
*****************************************

Citus has two executor types for running distributed SELECT queries. The desired executor can be selected by setting this configuration parameter. The accepted values for this parameter are:
Citus has three executor types for running distributed SELECT queries. The desired executor can be selected by setting this configuration parameter. The accepted values for this parameter are:

* **real-time:** The real-time executor is the default executor and is optimal when you require fast responses to queries that involve aggregations and co-located joins spanning across multiple shards.
* **adaptive:** The default. It is optimal for fast responses to queries that involve aggregations and co-located joins spanning across multiple shards.

* **task-tracker:** The task-tracker executor is well suited for long running, complex queries which require shuffling of data across worker nodes and efficient resource management.

* **real-time:** (deprecated) Serves a similar purpose as the adaptive executor, but is less flexible and can cause more connection pressure on worker nodes.


This parameter can be set at run-time and is effective on the coordinator. For more details about the executors, you can visit the :ref:`distributed_query_executor` section of our documentation.

.. _multi_task_logging:
Expand Down
2 changes: 1 addition & 1 deletion develop/api_metadata.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ Here's an example of how ``create_distributed_function()`` adds entries to the
-- will have two rows, one for the TYPE and one for the FUNCTION
TABLE citus.pg_dist_object;
.. code-block::
.. code-block:: text
-[ RECORD 1 ]---------------+------
classid | 1247
Expand Down
35 changes: 16 additions & 19 deletions develop/api_udf.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -453,19 +453,17 @@ servers, relevant only for streaming replication. Default 0
Return Value
******************************

A tuple which represents a row from :ref:`pg_dist_node
<pg_dist_node>` table.

The nodeid column from the newly inserted row in :ref:`pg_dist_node <pg_dist_node>`.

Example
***********************

.. code-block:: postgresql
select * from master_add_node('new-node', 12345);
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive | groupid | noderole | nodecluster
--------+---------+----------+----------+----------+-------------+----------+---------+----------+ ------------
7 | 7 | new-node | 12345 | default | f | t | 0 | primary | default
master_add_node
-----------------
7
(1 row)
.. _master_update_node:
Expand Down Expand Up @@ -523,17 +521,17 @@ servers, relevant only for streaming replication. Default 0
Return Value
******************************

A tuple which represents a row from :ref:`pg_dist_node <pg_dist_node>` table.
The nodeid column from the newly inserted row in :ref:`pg_dist_node <pg_dist_node>`.

Example
***********************

.. code-block:: postgresql
select * from master_add_inactive_node('new-node', 12345);
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive | groupid | noderole | nodecluster
--------+---------+----------+----------+----------+-------------+----------+---------+----------+ -------------
7 | 7 | new-node | 12345 | default | f | f | 0 | primary | default
master_add_inactive_node
--------------------------
7
(1 row)
master_activate_node
Expand All @@ -553,18 +551,17 @@ Arguments
Return Value
******************************

A tuple which represents a row from :ref:`pg_dist_node
<pg_dist_node>` table.
The nodeid column from the newly inserted row in :ref:`pg_dist_node <pg_dist_node>`.

Example
***********************

.. code-block:: postgresql
select * from master_activate_node('new-node', 12345);
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive| noderole | nodecluster
--------+---------+----------+----------+----------+-------------+---------+----------+ -------------
7 | 7 | new-node | 12345 | default | f | t | primary | default
master_activate_node
----------------------
7
(1 row)
master_disable_node
Expand Down Expand Up @@ -621,17 +618,17 @@ Arguments
Return Value
******************************

A tuple which represents a row from :ref:`pg_dist_node <pg_dist_node>` table.
The nodeid column for the secondary node, inserted row in :ref:`pg_dist_node <pg_dist_node>`.

Example
***********************

.. code-block:: postgresql
select * from master_add_secondary_node('new-node', 12345, 'primary-node', 12345);
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster
--------+---------+----------+----------+----------+-------------+----------+-----------+-------------
7 | 7 | new-node | 12345 | default | f | t | secondary | default
master_add_secondary_node
---------------------------
7
(1 row)
Expand Down
11 changes: 6 additions & 5 deletions develop/reference_ddl.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ Here is a reference of the categories of DDL statements which propagate. Note th
Adding/Modifying Columns
~~~~~~~~~~~~~~~~~~~~~~~~

Citus propagates most `ALTER TABLE <https://www.postgresql.org/docs/current/static/ddl-alter.html>`_ commands automatically. Adding columns or changing their default values work as they would in a single-machine PostgreSQL database:
Citus propagates most `ALTER TABLE <https://www.postgresql.org/docs/current/static/ddl-alter.html>`__ commands automatically. Adding columns or changing their default values work as they would in a single-machine PostgreSQL database:

.. code-block:: postgresql
Expand Down Expand Up @@ -232,15 +232,16 @@ Adding/Removing Constraints

Using Citus allows you to continue to enjoy the safety of a relational database, including database constraints (see the PostgreSQL `docs <https://www.postgresql.org/docs/current/static/ddl-constraints.html>`_). Due to the nature of distributed systems, Citus will not cross-reference uniqueness constraints or referential integrity between worker nodes.

To set up a foreign key between colocated distributed tables, always include the distribution column in the key. This may involve making the key compound.

Foreign keys may be created in these situations:

* between two local (non-distributed) tables,
* between two reference tables,
* between two :ref:`colocated <colocation>` distributed tables when the key includes the distribution column, or
* as a distributed table referencing a :ref:`reference table <reference_tables>`

Reference tables are not supported as the *referencing* table of a foreign key constraint, i.e. keys from reference to reference and reference to distributed are not supported.

To set up a foreign key between colocated distributed tables, always include the distribution column in the key. This may involve making the key compound.
Foreign keys from reference tables to distributed tables are not supported.

.. note::

Expand Down Expand Up @@ -348,7 +349,7 @@ Later, during non-peak hours, a database administrator can attempt to fix the ba
ALTER TABLE users
VALIDATE CONSTRAINT syntactic_email;
The PostgreSQL documentation has more information about NOT VALID and VALIDATE CONSTRAINT in the `ALTER TABLE <https://www.postgresql.org/docs/current/sql-altertable.html>`_ section.
The PostgreSQL documentation has more information about NOT VALID and VALIDATE CONSTRAINT in the `ALTER TABLE <https://www.postgresql.org/docs/current/sql-altertable.html>`__ section.

Adding/Removing Indices
~~~~~~~~~~~~~~~~~~~~~~~
Expand Down

0 comments on commit 8ea8d6e

Please sign in to comment.