[Bug] Expand fails to recognize user-defined tablespaces

[nix-oss]

Apache Cloudberry version

2.1.0-incubating

What happened

When user-defined tablespaces exist in the cluster, gpexpand does not create the newTableSpaceInfo.json file in the new segment template when running with a config file (non-interactive mode).

This leads to broken symlinks for tablespaces and causes failures during the prepare_schema() stage.

The problem is present in two methods within the same file:

1. read_tablespace_file() in gpMgmt/bin/gpexpand (lines 1219–1285)
2. generate_tablespace_inputfile() in gpMgmt/bin/gpexpand (lines 1114–1148)

• read_tablespace_file() (line 1244):

tblspc_oids = os.listdir(coordinator_tblspc_dir)
tblspc_oid_names = self.get_tablespace_oid_names()
flag = False
for oid in tblspc_oids:
    if oid in tblspc_oid_names:  <---
        flag = True
if not flag:
    return None  

• generate_tablespace_inputfile() (line 1124)

tblspc_oid_names = self.get_tablespace_oid_names()
tblspc_info = {}

for oid in tblspc_oids:
    if oid not in tblspc_oid_names:  <---
        continue
    location = os.path.dirname(os.readlink(os.path.join(coordinator_tblspc_dir,
                                                        oid)))
    tblspc_info[oid] = {"location": location,
                        "name": tblspc_oid_names[int(oid)]}

Root cause: type mismatch between string values returned by os.listdir() and integer keys returned by the SQL query SELECT oid, spcname FROM pg_tablespace.

When attaching to the process and debugging, the types differ, causing this issue:

• type mismatch

• newTableSpaceInfo is None

Impact:

• newTableSpaceInfo is always None.
• _handle_tablespace_template() is never invoked.
• newTableSpaceInfo.json is not included in the template tar.
• gpconfigurenewsegment on new hosts does not fix tablespace symlinks.
• new segments start with broken symlinks.

What you think should happen instead

To fix the issue, unify the data types in both methods:

• In generate_tablespace_inputfile():

for oid in tblspc_oids:
  if int(oid) not in tblspc_oid_names:

• In read_tablespace_file():

tblspc_oids = os.listdir(coordinator_tblspc_dir)
tblspc_oid_names = self.get_tablespace_oid_names()
flag = False
for oid in tblspc_oids:
  if int(oid) in tblspc_oid_names:

After this change, newTableSpaceInfo is no longer None:

How to reproduce

1. Connect to the database:

gpadmin@cbdb-mdw:~$ psql warehouse

2. Create a user-defined tablespace:

warehouse=# CREATE TABLESPACE ts_stage_logs LOCATION '/tblspc_stage_logs';

warehouse=# SELECT * FROM pg_tablespace;
  oid  |    spcname    | spcowner | spcacl | spcoptions | spcfilehandlersrc | spcfilehandlerbin 
-------+---------------+----------+--------+------------+-------------------+-------------------
  1663 | pg_default    |       10 |        |            |                   | 
  1664 | pg_global     |       10 |        |            |                   | 
 17019 | ts_stage_logs |       10 |        |            |                   | 
(3 rows)

3. Create an append-optimized columnar table and populate it with test data:

 warehouse=# CREATE TABLE logs_aot (
     id              BIGSERIAL,
     log_timestamp   TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
     log_level       VARCHAR(10)
 )
 WITH (
     APPENDONLY = TRUE,
     ORIENTATION = COLUMN
 )
 TABLESPACE ts_stage_logs
 DISTRIBUTED BY (id);

 warehouse=# INSERT INTO logs_aot (
     log_timestamp,
     log_level
 )
 SELECT
     NOW() - (random() * INTERVAL '90 days'),
     (ARRAY['INFO', 'WARN', 'ERROR', 'DEBUG'])[floor(random() * 4 + 1)]
 FROM generate_series(1, 8000);

4. Prepare the expansion configuration file for adding 2 new hosts to the cluster:

sdw3-new|sdw3-new|6000|/primary_data/gpseg4|10|4|p
sdw4-new|sdw4-new|7000|/mirror_data/gpseg4|11|4|m
sdw3-new|sdw3-new|6001|/primary_data/gpseg5|12|5|p
sdw4-new|sdw4-new|7001|/mirror_data/gpseg5|13|5|m
sdw4-new|sdw4-new|6000|/primary_data/gpseg6|14|6|p
sdw3-new|sdw3-new|7000|/mirror_data/gpseg6|15|6|m
sdw4-new|sdw4-new|6001|/primary_data/gpseg7|16|7|p
sdw3-new|sdw3-new|7001|/mirror_data/gpseg7|17|7|m

5. Start the expansion process (non-interactive mode):

gpadmin@cbdb-mdw:~$ gpexpand -i expand.cfg

6. The following error appears in the logs (log truncated for readability.)

20260805:18:33:04:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Heap checksum setting consistent across cluster
20260805:18:33:04:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Syncing Apache Cloudberry extensions
20260805:18:33:04:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Locking catalog
20260805:18:33:05:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Locked catalog
20260805:18:33:06:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Creating segment template
20260805:18:33:07:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Copying postgresql.conf from existing segment into template
20260805:18:33:08:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Copying pg_hba.conf from existing segment into template
20260805:18:33:09:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Creating schema tar file
...
20260805:18:33:26:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data from database postgres
20260805:18:33:27:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data from database warehouse
20260805:18:33:27:052024 gpexpand:cbdb-mdw:gpadmin-[ERROR]:-gpexpand failed: ERROR:  could not open file "pg_tblspc/17019/GPDB_3_302606111/17018/16386": No such file or directory  (seg6 203.0.113.5:6000 pid=21449)

Exiting...
20260805:18:33:27:052024 gpexpand:cbdb-mdw:gpadmin-[ERROR]:-gpexpand is past the point of rollback. Any remaining issues must be addressed outside of gpexpand.
20260805:18:33:27:052024 gpexpand:cbdb-mdw:gpadmin-[INFO]:-Shutting down gpexpand...

Hope this analysis is helpful.

Operating System

Ubuntu 22.04

Anything else

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.