SingleNode deployment #206

Ray-Eldath · 2023-09-19T07:54:49Z

closes #69, closes #53, closes #100, closes #123

Change logs

This PR implements singlenode deployment. All code, scripts and tests are incorporated.

Please see the full design proposal: https://github.com/orgs/cloudberrydb/discussions/188#discussion-5593615 which has already been discussed with and approved by reviewers.

~~Commit messages're still in a mess. I'll squash them into a nice write-up after we finish reviewing.~~

This PR is heavily based on @wfnuser's previous work (#77).

Why are the changes needed?

Please refer to the Motivation section of the design proposal.

CLAassistant · 2023-09-19T07:55:01Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ wfnuser
✅ Ray-Eldath
❌ tglsfdc
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

avamingli · 2023-09-19T12:51:17Z

Commit messages're still in a mess. I'll squash them into a nice write-up after we finish reviewing.

Hi, thanks for your contribution.
Commit messages are also part of review process, please squash commits as three kinds: our CBDB codes change/All tests added/PG commits cherry-picked one-by-one.
Review by commits is helpful for reviewers as this pr have so many test files.

avamingli

Most codes look good.

src/backend/optimizer/plan/planner.c

src/backend/optimizer/prep/prepunion.c

src/backend/optimizer/path/allpaths.c

src/backend/commands/vacuum.c

src/backend/commands/tablecmds.c

src/test/regress/pg_regress.c

src/include/executor/executor.h

src/backend/utils/misc/guc_gp.c

src/backend/postmaster/postmaster.c

src/backend/executor/execCurrent.c

src/backend/access/table/table.c

Ray-Eldath · 2023-10-10T11:05:54Z

really can't replicate the failure of https://github.com/cloudberrydb/cloudberrydb/actions/runs/6467257609/job/17557480521?pr=206 on my machine even i've already enabled xml support with --with-libxml

already fixed.

apart from this, all reviewer comments are fixed or replied. could you please check whether my response addressed all your concerns? @avamingli

yjhjstz · 2023-10-26T09:01:22Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

gpMgmt/bin/gppylib/gparray.py

src/test/isolation/specs/udf-insert-deadlock.spec

Ray-Eldath · 2023-10-30T03:23:38Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

I've tried with proper postmaster.conf using env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby/log/startup.log -w -t 600 -o " -p 7001" start and env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility" start can successfully create a singlenode deployment. but the recommended approach is still to use gpinitsystem --singlenodeMode to start one coordinator and one coordinator standby.

yjhjstz · 2023-10-31T23:18:51Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

I've tried with proper postmaster.conf using env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby/log/startup.log -w -t 600 -o " -p 7001" start and env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility" start can successfully create a singlenode deployment. but the recommended approach is still to use gpinitsystem --singlenodeMode to start one coordinator and one coordinator standby.

can we change default datadirs/qddir to make singlenode dir ?

Ray-Eldath · 2023-11-01T03:39:26Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

I've tried with proper postmaster.conf using env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby/log/startup.log -w -t 600 -o " -p 7001" start and env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility" start can successfully create a singlenode deployment. but the recommended approach is still to use gpinitsystem --singlenodeMode to start one coordinator and one coordinator standby.

can we change default datadirs/qddir to make singlenode dir ?

we can arbitrarily change dir of the coordinator and the coordinator standby. when using gpinitsystem, env COORDINATOR_DATADIR can be set, and when using pg_ctl, just change -D argument.

yjhjstz · 2023-11-01T03:50:33Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

I've tried with proper postmaster.conf using env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby/log/startup.log -w -t 600 -o " -p 7001" start and env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility" start can successfully create a singlenode deployment. but the recommended approach is still to use gpinitsystem --singlenodeMode to start one coordinator and one coordinator standby.

can we change default datadirs/qddir to make singlenode dir ?

we can arbitrarily change dir of the coordinator and the coordinator standby. when using gpinitsystem, env COORDINATOR_DATADIR can be set, and when using pg_ctl, just change -D argument.

COORDINATOR_DATADIR can only change datadirs, but I want to rename qddir to singlenode dir.

Ray-Eldath · 2023-11-02T02:28:43Z

how to install single node with rpm package ? think about install like pg style: initdb, then pg_ctl start @Ray-Eldath

I've tried with proper postmaster.conf using env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/standby/log/startup.log -w -t 600 -o " -p 7001" start and env GPSESSID=0000000000 GPERA=f6a2f666f34494b3_230907184019 $GPHOME/bin/pg_ctl -D /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/gpadmin/cbdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility" start can successfully create a singlenode deployment. but the recommended approach is still to use gpinitsystem --singlenodeMode to start one coordinator and one coordinator standby.

can we change default datadirs/qddir to make singlenode dir ?

we can arbitrarily change dir of the coordinator and the coordinator standby. when using gpinitsystem, env COORDINATOR_DATADIR can be set, and when using pg_ctl, just change -D argument.

COORDINATOR_DATADIR can only change datadirs, but I want to rename qddir to singlenode dir.

gpinitsystem accepts a clusterConfigFile where you can arbitrarily set COORDINATOR_DIRECTORY and DATA_DIRECTORY to put cluster data anywhere you want. this file (and the paths) is generated by demo_cluster.sh.

If we want singlenode deployment to have gpAux/gpdemo/datadirs/singlenodedir instead of gpAux/gpdemo/datadirs/qddir, we need to take care of this in demo_cluster.sh. Is this what you want? I think it's actually a great idea.

yjhjstz · 2023-11-02T06:12:26Z

gpinitsystem accepts a clusterConfigFile where you can arbitrarily set COORDINATOR_DIRECTORY and DATA_DIRECTORY to put cluster data anywhere you want. this file (and the paths) is generated by demo_cluster.sh.

If we want singlenode deployment to have gpAux/gpdemo/datadirs/singlenodedir instead of gpAux/gpdemo/datadirs/qddir, we need to take care of this in demo_cluster.sh. Is this what you want? I think it's actually a great idea.

yes, make different dir.

Ray-Eldath · 2023-11-02T08:57:29Z

yes, make different dir.

fixed in

cloudberrydb/gpAux/gpdemo/demo_cluster.sh

Line 30 in c2d7dda

QDDIR=$DATADIRS/singlenodedir

check_agg_arguments_walker() supposed that it needn't descend into the arguments of a lower-level aggregate function, but this is just wrong in the presence of multiple levels of sub-select. The oversight would lead to executor failures on queries that should be rejected. (Prior to v11, they actually were rejected, thanks to a "redundant" execution-time check.) Per bug #17835 from Anban Company. Back-patch to all supported branches. Discussion: https://postgr.es/m/17835-4f29f3098b2d0ba4@postgresql.org

Commit 3e310d837 taught isAssignmentIndirectionExpr() to look through CoerceToDomain nodes. That's not sufficient, because since commit 04fe805 it's been possible for the planner to simplify CoerceToDomain to RelabelType when the domain has no constraints to enforce. So we need to look through RelabelType too. Per bug #17897 from Alexander Lakhin. Although 3e310d837 was back-patched to v11, it seems sufficient to apply this change to v12 and later, since 04fe805 came in in v12. Dmitry Dolgov Discussion: https://postgr.es/m/17897-4216c546c3874044@postgresql.org

This commit is a squash of the following commits: - Fix all isolation2_schedule cases in single node mode - Pass all the selected isolation2_schedule tests in single node mode. - Copy tests from test/isolation2 for isolation2 schedule in single-node. - Fix all greenplum_schedule tests diffs in single node mode. - Fix all parallel_schedule tests diff in single node mode - Fix test singlenode_regress/xml (only failed on CI)

Since CBDB isn't fully compatible with Postgres, many features and grammars have been dependent upon. In a scenario where only one node is needed but CBDB is already relied by business logic, a singlenode mode can be used. Utility mode is different from singlenode because it still requires segments be created, it just connects to one of them in a special mode designated by PGOPTIONS. On the contrary, singlenode mode is fully supported by all deploying scripts, user can create a running node that have no segment and directly connect to it as if it were a normal cluster. Plus, there're some functionalities that aren't enabled in utility mode but should be in singlenode. This new mode is supported by adding a new GUC gp_internal_is_singlenode. We've settled to leverage the already-written code to support GP_ROLE_UTILITY by running singlenode deployment in GP_ROLE_UTILITY. This way seems leads to the minimum code change. There're several places where special care are needed: code that should run only under normal utility mode but not in singlenode are taken care of by IS_UTILITY_BUT_NOT_SINGLENODE() macro, and, conversely, code that should only run in singlenode are handled by Gp_role == GP_ROLE_DISPATCH || IS_SINGLENODE(). For the remaining code changes, they're all caused by several global replacements. Scripts has been changed to support singlenode deployment. When creating demo cluster, use NUM_PRIMARY_MIRROR_PAIRS=0 to create singlenode deployment. It'll write the GUC in postgresql.conf and only start QD and one standby. gpstart and gpstop behaves normally because singlenode mode is controlled by a GUC that only initialized in gpinitsystem. We delibrately pick a long name because we don't want user to accidentally change this after cluster initialization. See: Discussion#188 <https://github.com/orgs/cloudberrydb/discussions/188>

This PR adds a new GitHub Actions job, ic-singlenode-test, to build workflow. It has been verified that the build procedure and log uploading both work as expected.

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 2 times, most recently from f5a1f51 to 2651b1c Compare September 19, 2023 07:58

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 8 times, most recently from 2706ccb to ff14705 Compare September 25, 2023 03:14

avamingli reviewed Sep 25, 2023

View reviewed changes

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 5 times, most recently from 27757d7 to cf7f33c Compare September 26, 2023 10:40

Ray-Eldath commented Oct 8, 2023

View reviewed changes

src/backend/access/table/table.c Show resolved Hide resolved

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 4 times, most recently from e64d6c6 to d724550 Compare October 10, 2023 09:17

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 3 times, most recently from 4363279 to 1daf639 Compare October 11, 2023 02:47

Ray-Eldath mentioned this pull request Oct 11, 2023

Limit CI concurrency to one per PR #233

Merged

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 2 times, most recently from 60e4396 to 70421cd Compare October 11, 2023 03:25

yjhjstz reviewed Oct 27, 2023

View reviewed changes

gpMgmt/bin/gppylib/gparray.py Show resolved Hide resolved

yjhjstz reviewed Oct 27, 2023

View reviewed changes

src/test/isolation/specs/udf-insert-deadlock.spec Outdated Show resolved Hide resolved

Ray-Eldath dismissed avamingli’s stale review via 8d04121 October 27, 2023 02:05

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 2 times, most recently from 8d04121 to 4d3f71f Compare October 30, 2023 03:23

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch from 4d3f71f to 223997d Compare October 30, 2023 07:12

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 2 times, most recently from 229bcab to 2f8bdf3 Compare November 2, 2023 08:56

yjhjstz approved these changes Nov 2, 2023

View reviewed changes

Ray-Eldath requested a review from avamingli November 3, 2023 01:39

avamingli approved these changes Nov 3, 2023

View reviewed changes

Ray-Eldath force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch 3 times, most recently from 4fbc2ba to 4534e4c Compare November 6, 2023 02:36

tglsfdc and others added 5 commits November 8, 2023 10:55

Add CI for singlenode mode

a3c6f7e

This PR adds a new GitHub Actions job, ic-singlenode-test, to build workflow. It has been verified that the build procedure and log uploading both work as expected.

avamingli force-pushed the feat/feat-singlenode/utility-fix-plan-1 branch from 4534e4c to a3c6f7e Compare November 8, 2023 02:55

avamingli approved these changes Nov 8, 2023

View reviewed changes

avamingli merged commit 6658030 into cloudberrydb:main Nov 8, 2023
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SingleNode deployment #206

SingleNode deployment #206

Ray-Eldath commented Sep 19, 2023 •

edited

CLAassistant commented Sep 19, 2023 •

edited

avamingli commented Sep 19, 2023 •

edited

avamingli left a comment •

edited

Ray-Eldath commented Oct 10, 2023 •

edited

yjhjstz commented Oct 26, 2023 •

edited

Ray-Eldath commented Oct 30, 2023

yjhjstz commented Oct 31, 2023

Ray-Eldath commented Nov 1, 2023

yjhjstz commented Nov 1, 2023

Ray-Eldath commented Nov 2, 2023 •

edited

yjhjstz commented Nov 2, 2023

Ray-Eldath commented Nov 2, 2023 •

edited

SingleNode deployment #206

SingleNode deployment #206

Conversation

Ray-Eldath commented Sep 19, 2023 • edited

Change logs

Why are the changes needed?

CLAassistant commented Sep 19, 2023 • edited

avamingli commented Sep 19, 2023 • edited

avamingli left a comment • edited

Choose a reason for hiding this comment

Ray-Eldath commented Oct 10, 2023 • edited

yjhjstz commented Oct 26, 2023 • edited

Ray-Eldath commented Oct 30, 2023

yjhjstz commented Oct 31, 2023

Ray-Eldath commented Nov 1, 2023

yjhjstz commented Nov 1, 2023

Ray-Eldath commented Nov 2, 2023 • edited

yjhjstz commented Nov 2, 2023

Ray-Eldath commented Nov 2, 2023 • edited

Ray-Eldath commented Sep 19, 2023 •

edited

CLAassistant commented Sep 19, 2023 •

edited

avamingli commented Sep 19, 2023 •

edited

avamingli left a comment •

edited

Ray-Eldath commented Oct 10, 2023 •

edited

yjhjstz commented Oct 26, 2023 •

edited

Ray-Eldath commented Nov 2, 2023 •

edited

Ray-Eldath commented Nov 2, 2023 •

edited