Implement Citus support for pg_auto_failover (#933)

* Add a PgInstanceKind guard in the Keeper FSM.

This guard allows to choose a transition function specialized in a given
node kind.

* Implement Citus support for pg_auto_failover.

* Implement make CITUS=1 cluster.

* Allow CI testing of the Citus support.

* Add an architecture diagram covering Citus formations.

* Improve all diagrams.

Keep the concept of the first diagram with the kind of a circle with four
nodes (application, primary, secondary, monitor) and build on-top of the
same idea for the others diagrams.

Every diagram builds by adding some level of complexity on-top of the
previous one. By keeping a central theme it makes it obvious and much easier
to re-use the understanding from the previous diagram in the next.

* Fix the FSM with missing transition that got lost in the merge.

* Add support for quorum and priority to make compose.

* When using Citus, the default dbname is now "citus".

* Fix docs, Citus is now included in the Open Source version.

* Add support for --pg-hba-lan for create coordinator and worker.

* Add better support for Citus in the dev env (Dockerfile, make compose).

* Given Citus workers, allow for greater NODE_ARRAY_MAX_COUNT.

* Fix computing Citus nodes group/seq and names.

When given W workers, we create N Nodes per worker, plus N coordinators.

* Improve default node names for a Citus cluster.

* Rework Citus Cluster Tutorial using docker-compose.

* Refactor CLI environment variable support code.

* Fix monitor installcheck expected file, and automated naming.

* Update Python imports for Citus test files.

* Docs improvements for Citus coverage.

* Update Citus version matrix and compatibility with PGVERSIONs.

Also we need to drop support for Postgres 10, for which I failed to find a
version of Citus that would be compatible. Fortunately, it's the time when
we also add support for Postgres 15, Postgres 10 is soon to be unsupported
anyway.

* Fix Citus tests to use the "citus" database.

* Increate tests timeout.

* Separate building the docker image and running tests in CI steps.

* Skip building the docker image for TEST=linting.

* Improve workflow for linting / non-linting tests.

* Update Citus tests for Citus 11.1.

In Citus 11.1 we need to make sure the nodes metadata is in sync before some
cluster wide operations such as DROP TABLE. This is best done by calling a
citus test only function: wait_until_metadata_sync.

The function needs to first be created/exposed at the SQL level.

* Another round of Citus test fixes.

  - Fix the tests to be consistent about using the "citus" dbname
  - Fix the timeout when calling walt_until_metadata_sync in some tests

* Tests: install faster metadata_sync_interval for Citus coordinators.

* Update docs/citus.rst

Co-authored-by: Jelte Fennema <github-tech@jeltef.nl>

.github/workflows/build-package.yml → .github/workflows/run-tests.yml

@@ -1,7 +1,5 @@
-name: Build Package
+name: Run Tests
 
-env:
-  TRAVIS: "true"
 on:
   push:
     branches:
@@ -13,14 +11,13 @@ on:
   workflow_dispatch:
 
 jobs:
-  build_package:
+  run_tests:
     name: Run test
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         PGVERSION:
-          - 10
           - 11
           - 12
           - 13
@@ -31,6 +28,7 @@ jobs:
           - single
           - monitor
           - ssl
+          - citus
         include:
           - PGVERSION: 14
             TEST: tablespaces
@@ -48,6 +46,7 @@ jobs:
             echo "TRAVIS_BUILD_DIR=$(pwd)" >> $GITHUB_ENV
 
       - name: Clone and install linting tools
+        if: ${{ env.TEST == 'linting' }}
         run: |
           sudo apt-get install python3-pip
           pip3 install --user black
@@ -58,7 +57,18 @@ jobs:
           install_uncrustify
           rm -rf uncrustify*
 
+      - name: Check code formatting and banned function
+        if: ${{ env.TEST == 'linting' }}
+        run: |
+          make lint
+
+      - name: Build Docker Test Image
+        if: ${{ env.TEST != 'linting' }}
+        run: |
+          make build-test-image
+
       - name: Run Test
+        if: ${{ env.TEST != 'linting' }}
         timeout-minutes: 15
         run: |
           make ci-test

.travis.yml

@@ -22,6 +22,9 @@ matrix:
     - env: PGVERSION=12 TEST=single
     - env: PGVERSION=13 TEST=single
     - env: PGVERSION=14 TEST=single
+    - env: PGVERSION=11 CITUSVERSION=10.0 TEST=citus
+    - env: PGVERSION=12 CITUSVERSION=10.1 TEST=citus
+    - env: PGVERSION=13 CITUSVERSION=10.1 TEST=citus
     - env: PGVERSION=10 TEST=monitor
     - env: PGVERSION=11 TEST=monitor
     - env: PGVERSION=12 TEST=monitor
@@ -37,6 +40,7 @@ matrix:
 before_install:
   - git clone -b v0.7.18 --depth 1 https://github.com/citusdata/tools.git
   - sudo make -C tools install
+  - 'if [ -z "$LINTING" ]; then curl https://install.citusdata.com/community/deb.sh | sudo bash; fi'
   - 'if [ -z "$LINTING" ]; then setup_apt; fi'
   - 'if [ -z "$LINTING" ]; then nuke_pg; fi'
   - python --version
@@ -56,6 +60,8 @@ install:
   - 'if [ -n "$LINTING" ]; then rm -rf uncrustify*; fi'
   - 'if [ -z "$LINTING" ]; then install_pg; fi'
   - 'if [ -z "$LINTING" ]; then install_custom_pg; fi'
+  - 'if [ -z "$LINTING" ]; then sudo apt-get install postgresql-${PGVERSION}-citus-${CITUSVERSION}; fi'
+  - PIPENV_PIPFILE="${TRAVIS_BUILD_DIR}"/tests/Pipfile pipenv install --system --deploy
   - env
   - 'if [ -z "$LINTING" ]; then pg_config; fi'
   - 'if [ -z "$LINTING" ]; then PATH=`pg_config --bindir`:$PATH which pg_ctl; fi'

Dockerfile

@@ -3,14 +3,14 @@
 # target version of Postgres. In the Makefile, we use that to our advantage
 # and tag test images such as pg_auto_failover_test:pg14.
 #
-ARG PGVERSION=10
+ARG PGVERSION=14
 
 #
 # Define a base image with all our build dependencies.
 #
 # This base image contains all our target Postgres versions.
 #
-FROM debian:buster-slim as base
+FROM debian:bullseye-slim as base
 
 ARG PGVERSION
 
@@ -21,21 +21,28 @@ RUN apt-get update \
 	curl \
 	gnupg \
 	git \
+	gawk \
+	flex \
+	bison \
     iproute2 \
-    libicu-dev \
-    libkrb5-dev \
-    libssl-dev \
+	libcurl4-gnutls-dev \
+	libicu-dev \
+	libncurses-dev \
+	libxml2-dev \
+	zlib1g-dev \
     libedit-dev \
-    libreadline-dev \
-    libpam-dev \
-    zlib1g-dev \
+    libkrb5-dev \
     liblz4-dev \
-	libxml2-dev \
-    libxslt1-dev \
-    libselinux1-dev \
-	libncurses-dev \
     libncurses6 \
-    make \
+    libpam-dev \
+    libreadline-dev \
+    libselinux1-dev \
+    libssl-dev \
+    libxslt1-dev \
+    libzstd-dev \
+    uuid-dev \
+	make \
+	autoconf \
     openssl \
     pipenv \
     python3-nose \
@@ -53,7 +60,7 @@ RUN apt-get update \
 	&& rm -rf /var/lib/apt/lists/*
 
 RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
-RUN echo "deb http://apt.postgresql.org/pub/repos/apt buster-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
+RUN echo "deb http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
 
 # bypass initdb of a "main" cluster
 RUN echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf
@@ -70,12 +77,25 @@ RUN adduser docker sudo
 RUN adduser docker postgres
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
+FROM base as citus
+
+ARG PGVERSION
+ARG CITUSTAG=v11.1.2
+
+ENV PG_CONFIG /usr/lib/postgresql/${PGVERSION}/bin/pg_config
+
+RUN git clone -b ${CITUSTAG} --depth 1 https://github.com/citusdata/citus.git /usr/src/citus
+WORKDIR /usr/src/citus
+
+RUN ./configure
+RUN make -s clean && make -s -j8 install
+
 #
 # On-top of the base build-dependencies image, now we can build
 # pg_auto_failover for a given --build-arg PGVERSION target version of
 # Postgres.
 #
-FROM base as build
+FROM citus as build
 
 ARG PGVERSION
 
@@ -109,7 +129,7 @@ ENV PATH /usr/lib/postgresql/${PGVERSION}/bin:/usr/local/sbin:/usr/local/bin:/us
 #
 # And finally our "run" images with the bare minimum for run-time.
 #
-FROM debian:buster-slim as run
+FROM debian:bullseye-slim as run
 
 ARG PGVERSION
 
@@ -127,12 +147,14 @@ RUN apt-get update \
     psutils \
     dnsutils \
     bind9-host \
-    postgresql-common \
+	libcurl4-gnutls-dev \
+    libzstd-dev \
+	postgresql-common \
     libpq-dev \
 	&& rm -rf /var/lib/apt/lists/*
 
 RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
-RUN echo "deb http://apt.postgresql.org/pub/repos/apt buster-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
+RUN echo "deb http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
 
 # bypass initdb of a "main" cluster
 RUN echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf
@@ -145,6 +167,9 @@ RUN adduser docker sudo
 RUN adduser docker postgres
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
+COPY --from=build /usr/lib/postgresql/${PGVERSION}/lib/citus.so /usr/lib/postgresql/${PGVERSION}/lib
+COPY --from=build /usr/share/postgresql/${PGVERSION}/extension/citus* /usr/share/postgresql/${PGVERSION}/extension/
+
 COPY --from=build /usr/lib/postgresql/${PGVERSION}/lib/pgautofailover.so /usr/lib/postgresql/${PGVERSION}/lib
 COPY --from=build /usr/share/postgresql/${PGVERSION}/extension/pgautofailover* /usr/share/postgresql/${PGVERSION}/extension/
 COPY --from=build /usr/lib/postgresql/${PGVERSION}/bin/pg_autoctl /usr/local/bin