Support cloudberry #627

zhangwenchao-123 · 2025-10-21T09:05:42Z

Add the module name, JIRA# to PR/commit and description.
Add tests for the change.

The following two operator delete functions doesn't lookup in madlib library. Because it's not added in the library script file. void operator delete (void *ptr, std::size_t sz) noexcept; void operator delete[](void *ptr, std::size_t sz) noexcept; The two functions are missing previously.

tuhaihe · 2025-10-21T09:30:36Z

src/ports/cloudberry/cmake/FindCloudberry.cmake

+set(_PG_CONFIG_VERSION_MACRO "GP_VERSION")
+set(_SEARCH_PATH_HINTS
+    "/usr/local/cloudberry-db-devel/bin"
+    "/usr/local/cloudberry-db/bin"


Is there a need to add /usr/local/cbdb/bin?

Have we used this path?

Have added path /usr/local/cloudberry/bin

I believe the line just after this "$ENV{GPHOME}/bin" will help catch most scenarios. Users will be sourcing cloudberry-env.sh (Cloudberry 3+) or greenplum_path.sh (Cloudberry 2 ).

tuhaihe · 2025-10-21T09:32:44Z

src/ports/cloudberry/cmake/FindCloudberry.cmake

+    if(_PG_CONFIG_HEADER_CONTENTS MATCHES "#define SERVERLESS 1")
+        message("-- Detected Hashdata Cloud (Cloudberry Serverless)")
+        set(CLOUDBERRY_SERVERLESS TRUE PARENT_SCOPE)


Remove these lines?

tuhaihe · 2025-10-21T09:34:56Z

src/ports/cloudberry/3/CMakeLists.txt

There is still no Cloudberry 3.0 yet. So can remove this file?

Apache MADlib should be able to build against both the REL_2_STABLE and main (3.0.0) branches. I believe it is better to keep support for Cloudberry 3.0. As main (3.0) has not released, maybe support for 3.0 can be labelled as experimental.

That makes sense. Thanks!

Agree with ed.

tuhaihe · 2025-10-21T09:47:36Z

src/ports/cloudberry/cmake/FindCloudberry.cmake

We also need to add the standard Apache license header to the new files, including FindCloudberry.cmake, and FindCloudberry_1.cmake and other files.

edespino

❌ What's Missing (Critical Issues)

1. No main CMakeLists.txt for Cloudberry:
   - src/ports/cloudberry/CMakeLists.txt doesn't exist
   - This file should mirror the structure of
     src/ports/greenplum/CMakeLists.txt (13KB, ~300 lines)
   - Should define: port configuration, source files, SQL handling,
     build functions, and version management

2. Not integrated into the build system:
   - src/ports/CMakeLists.txt only contains:
       add_subdirectory(postgres)
       add_subdirectory(greenplum)
   - Missing: add_subdirectory(cloudberry)
3. No CloudberryUtils.cmake:
   - Greenplum has GreenplumUtils.cmake with utility functions
   - May need similar utilities for Cloudberry-specific features

🔍 Current State

CMake configuration completed but:
- Cloudberry was NOT detected (the FindCloudberry code was never executed)
- Only PostgreSQL and Greenplum detection ran
- Build directory shows only postgres/ and greenplum/ subdirectories

However, there IS a Cloudberry installation:
- Location: /usr/local/cloudberry/
- Version: Based on PostgreSQL 14.4 with GP_VERSION_NUM 30000 (Cloudberry v3.0.0)
- This matches the src/ports/cloudberry/3/ directory structure

📊 Summary

The Cloudberry port is partially implemented. The detection logic and
version-specific configs exist, but they're not wired into the build
system.

To complete the implementation, you would need:

1. Create src/ports/cloudberry/CMakeLists.txt (modeled after Greenplum's)
2. Add add_subdirectory(cloudberry) to src/ports/CMakeLists.txt
3. Potentially create CloudberryUtils.cmake for Cloudberry-specific features
4. Test the full build process with Cloudberry detection

edespino · 2025-10-22T01:44:14Z

Have you looked at the website updates (https://madlib.apache.org - https://github.com/apache/madlib-site) other source documentation files? We will need to review these as well.

edespino · 2025-10-22T01:48:36Z

As @tuhaihe mentioned about ASF headers, when I ran the Apache Release Audit Tool (RAT), the following is seen (run the following in the root of the MADlib source: mvn apache-rat:check):

❯ head -30 target/rat.txt

*****************************************************
Summary
-------
Generated at: 2025-10-21T18:46:08-07:00
Notes: 4
Binaries: 5
Archives: 0
Standards: 311

Apache Licensed: 307
Generated Documents: 0

JavaDocs are generated and so license header is optional
Generated files do not required license headers

4 Unknown Licenses

*******************************

Unapproved licenses:

  src/ports/cloudberry/cmake/FindCloudberry.cmake
  src/ports/cloudberry/cmake/FindCloudberry_1.cmake
  src/ports/cloudberry/cmake/FindCloudberry_2.cmake
  src/ports/cloudberry/cmake/FindCloudberry_3.cmake

*******************************

zhangwenchao-123 · 2025-10-22T02:23:12Z

Have you looked at the website updates (https://madlib.apache.org - https://github.com/apache/madlib-site) other source documentation files? We will need to review these as well.

No, have not. Should we update this website?

tuhaihe · 2025-10-22T02:45:18Z

Have you looked at the website updates (https://madlib.apache.org - https://github.com/apache/madlib-site) other source documentation files? We will need to review these as well.

No, have not. Should we update this website?

Yes, we should update the related description on the website. I’d like to help with this.

zhangwenchao-123 · 2025-10-22T02:48:53Z

Have you looked at the website updates (https://madlib.apache.org - https://github.com/apache/madlib-site) other source documentation files? We will need to review these as well.

No, have not. Should we update this website?

Yes, we should update the related description on the website. I’d like to help with this.

Nice!

zhangwenchao-123 · 2025-10-22T02:57:50Z

❌ What's Missing (Critical Issues)

1. No main CMakeLists.txt for Cloudberry:
   - src/ports/cloudberry/CMakeLists.txt doesn't exist
   - This file should mirror the structure of
     src/ports/greenplum/CMakeLists.txt (13KB, ~300 lines)
   - Should define: port configuration, source files, SQL handling,
     build functions, and version management

2. Not integrated into the build system:
   - src/ports/CMakeLists.txt only contains:
       add_subdirectory(postgres)
       add_subdirectory(greenplum)
   - Missing: add_subdirectory(cloudberry)
3. No CloudberryUtils.cmake:
   - Greenplum has GreenplumUtils.cmake with utility functions
   - May need similar utilities for Cloudberry-specific features

🔍 Current State

CMake configuration completed but:
- Cloudberry was NOT detected (the FindCloudberry code was never executed)
- Only PostgreSQL and Greenplum detection ran
- Build directory shows only postgres/ and greenplum/ subdirectories

However, there IS a Cloudberry installation:
- Location: /usr/local/cloudberry/
- Version: Based on PostgreSQL 14.4 with GP_VERSION_NUM 30000 (Cloudberry v3.0.0)
- This matches the src/ports/cloudberry/3/ directory structure

📊 Summary

The Cloudberry port is partially implemented. The detection logic and
version-specific configs exist, but they're not wired into the build
system.

To complete the implementation, you would need:

1. Create src/ports/cloudberry/CMakeLists.txt (modeled after Greenplum's)
2. Add add_subdirectory(cloudberry) to src/ports/CMakeLists.txt
3. Potentially create CloudberryUtils.cmake for Cloudberry-specific features
4. Test the full build process with Cloudberry detection

Have fixed all mentioned problems and license lose

edespino · 2025-10-22T05:43:53Z

PR Review: Cloudberry MADlib Build Issues

CMake Configuration Command

cmake \
    -DCLOUDBERRY_3_PG_CONFIG=/usr/local/cloudberry/bin/pg_config \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local/madlib \
    -DCLOUDBERRY_3_EXECUTABLE=/usr/local/cloudberry/bin/postgres \
    ..

CMake Configuration Error

Error:
CMake Error at src/CMakeLists.txt:202 (add_library):
  Cannot find source file:

    /home/cbadmin/bom-parts/madlib/src/ports/cloudberry/dbconnector/Compatibility.hpp

Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
.hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc

CMake Error at src/CMakeLists.txt:202 (add_library):
No SOURCES given to target: madlib_cloudberry_3

CMake Generate step failed. Build files cannot be regenerated correctly.

Location: Referenced in src/ports/cloudberry/CMakeLists.txt:61

Observation: The directory /home/cbadmin/bom-parts/madlib/src/ports/cloudberry/dbconnector/ does not exist, while the equivalent Greenplum
directory does exist at /home/cbadmin/bom-parts/madlib/src/ports/greenplum/dbconnector/ containing:

Compatibility.hpp
dbconnector.hpp

Additional Build Errors (After Manual Directory Creation)

After manually creating the missing directory and copying files from Greenplum, cmake succeeded but compilation fails with multiple errors in
Compatibility.hpp:

AggState API change: aggcontext member doesn't exist (suggests aggcontexts)
WindowState renamed: T_WindowState not declared (suggests T_WindowAggState)
Missing function: format_procedure not declared
Function conflict: Ambiguous AggCheckCallContext - both the compatibility shim and PostgreSQL's native version exist

These errors indicate API differences between Greenplum's PostgreSQL base and Cloudberry's PostgreSQL base.

edespino · 2025-10-22T05:45:32Z

@zhangwenchao-123 - Unless absolutely necessary, there is no need to force push additional PR commits. This will allow us to view the PR history easily.

Fix SEGFAULT memory bugs There're weird SEGFAULT bug due to custom allocation erroneously paired with std::free (should be custom free) and we're unable to solve them. This is a workaround.

zhangwenchao-123 · 2025-10-23T07:01:03Z

PR Review: Cloudberry MADlib Build Issues

CMake Configuration Command
cmake \
    -DCLOUDBERRY_3_PG_CONFIG=/usr/local/cloudberry/bin/pg_config \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local/madlib \
    -DCLOUDBERRY_3_EXECUTABLE=/usr/local/cloudberry/bin/postgres \
    ..
CMake Configuration Error
Error:
CMake Error at src/CMakeLists.txt:202 (add_library):
  Cannot find source file:

    /home/cbadmin/bom-parts/madlib/src/ports/cloudberry/dbconnector/Compatibility.hpp
Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc

CMake Error at src/CMakeLists.txt:202 (add_library): No SOURCES given to target: madlib_cloudberry_3

CMake Generate step failed. Build files cannot be regenerated correctly.

Location: Referenced in src/ports/cloudberry/CMakeLists.txt:61

Observation: The directory /home/cbadmin/bom-parts/madlib/src/ports/cloudberry/dbconnector/ does not exist, while the equivalent Greenplum directory does exist at /home/cbadmin/bom-parts/madlib/src/ports/greenplum/dbconnector/ containing:

Compatibility.hpp

dbconnector.hpp

Additional Build Errors (After Manual Directory Creation)

After manually creating the missing directory and copying files from Greenplum, cmake succeeded but compilation fails with multiple errors in Compatibility.hpp:

AggState API change: aggcontext member doesn't exist (suggests aggcontexts)

WindowState renamed: T_WindowState not declared (suggests T_WindowAggState)

Missing function: format_procedure not declared

Function conflict: Ambiguous AggCheckCallContext - both the compatibility shim and PostgreSQL's native version exist

These errors indicate API differences between Greenplum's PostgreSQL base and Cloudberry's PostgreSQL base.

Yeah, there are some other commits not picked, I will continue to complete this PR and test it.

edespino

I have a few changes to consider. I have more testing of this PR to perform.

Please do not force push changes to this PR. I want to be able to follow the history of this work. Force pushing is not helping.

edespino · 2025-10-23T07:10:42Z

src/config/Ports.yml

    name:   Greenplum DB
+
+cloudberry:
+    name:   Cloudberry DB


Cloudberry DB should be Apache Cloudberry

edespino · 2025-10-23T07:16:04Z

src/ports/cloudberry/cmake/FindCloudberry.cmake

+    )
+    set(${PKG_NAME}_ADDITIONAL_INCLUDE_DIRS
+        "${${PKG_NAME}_ADDITIONAL_INCLUDE_DIRS}/internal")
+    message("-- Detected Cloudberry")


message("-- Detected Cloudberry") should be message("-- Detected Apache Cloudberry")

edespino · 2025-10-23T07:18:59Z

src/madpack/madpack.py

                    # only need the first two digits for <= 4.3.4
                    dbver = '.'.join(map(str, dbver_split[:2]))
+            elif portid == 'cloudberry':
+                # Assume Cloudberry will stick to semantic versioning


Assume Cloudberry will stick to semantic versioning should be Assume Apache Cloudberry will stick to semantic versioning

edespino · 2025-10-23T08:04:36Z

src/ports/cloudberry/cmake/FindCloudberry_1.cmake

Is this symlink needed? I believe we should only be providing support for the Apache Cloudberry 2 & 3 (future) releases.

edespino · 2025-10-23T08:08:28Z

src/madpack/utilities.py

            # 4.3.5+ from versions < 4.3.5
            match = re.search("Greenplum[a-zA-Z\s]*(\d+\.\d+\.\d+)", versionStr)
+        elif portid == 'cloudberry':
+            match = re.search("Cloudberry[a-zA-Z\s]*(\d+\.\d+\.\d+)", versionStr)


"Cloudberry[a-zA-Z\s]*(\d+\.\d+\.\d+)" should be "Apache Cloudberry[a-zA-Z\s]*(\d+\.\d+\.\d+)" ?

I am not entirely sure about this.

Cloudberry is enough to achieve our goal, while Apache Cloudberry is more accurate that maybe is better.

edespino · 2025-10-23T08:12:24Z

requirements.txt

Why is this empty file needed?

I noticed this when I ran the Apache Release Audit tool (mvn apache-rat:check).

In apache cloudberry, it's not needed. I will remove it.

tuhaihe · 2025-10-23T08:38:38Z

src/ports/postgres/cmake/PostgreSQLUtils.cmake

                # implying we only need 1 folder for same major versions
                set(VERSION ${${PORT_UC}_VERSION_MAJOR})
+            elseif(${PORT_UC} STREQUAL "CLOUDBERRY")
+                # Assumes CBDB always follows semantic versioning


Suggested change

# Assumes CBDB always follows semantic versioning

# Assumes Apache Cloudberry always follows semantic versioning

edespino · 2025-10-23T08:52:56Z

src/madpack/madpack.py

                libdir = libdir.decode()

-            libdir = libdir.strip()+'/postgresql'
+            libdir = str(libdir.strip(), encoding='utf-8')+'/postgresql'


Testing Note: Encountering TypeError: decoding str is not supported when installing on PostgreSQL 14.19.

Root Cause:
For Postgres 13+ (line 1347), libdir is already decoded to a string via .decode(), but line 1349 attempts to decode it again with
str(libdir.strip(), encoding='utf-8'), which fails because you cannot decode a string that's already been decoded.

Recommended Solution:
Ensure libdir is always decoded to a string before line 1349, then simply strip and append the path:

libdir = subprocess.check_output(['pg_config','--libdir']) if ((portid == 'greenplum' and is_rev_gte(dbver_split, get_rev_num('7.0'))) or (portid == 'postgres' and is_rev_gte(dbver_split, get_rev_num('13.0')))): libdir = libdir.decode() else: libdir = libdir.decode('utf-8') libdir = libdir.strip() + '/postgresql' This ensures libdir is consistently a string for all code paths (older and newer versions), eliminating the type inconsistency that causes the error. Request for Review: Please validate this fix works correctly for both: - Older versions (Postgres <13, Greenplum <7) where subprocess.check_output() returns bytes - Newer versions (Postgres 13+, Greenplum 7+) where explicit decoding is needed

zhangwenchao-123 · 2025-10-24T07:22:39Z

All mentioned comments have been fixed and have tested it in cloudberry 3.0.

tuhaihe · 2025-10-29T09:37:47Z

Hi @zhangwenchao-123 could you rebase your commits on the latest madlib2-master? Let's see if the CI can pass successfully. Thanks!

zhangwenchao-123 · 2025-10-30T02:49:24Z

Hi @zhangwenchao-123 could you rebase your commits on the latest madlib2-master? Let's see if the CI can pass successfully. Thanks!

It's the NOTICE file check failed, I have fixed and test whether it can pass.

tuhaihe · 2025-10-31T03:33:17Z

src/ports/cloudberry/cmake/FindCloudberry_2.cmake

I noticed these two files, FindCloudberry_2.cmake & FindCloudberry_3.cmake, are all symbolic links to FindCloudberry.cmake. Should we create them like GP / PG as ASCII text files? FYI.

tuhaihe · 2025-10-31T04:27:01Z

Based on the new codebase, I can build and deploy the MADlib into the Cloudberry 2.0 + 3.0 (main) gpdemo database:

Build the Cloudberry gpdemo env following the docs
Build and deploy the MADlib

## Download this PR change
git clone https://github.com/apache/madlib.git
cd madlib
git fetch origin pull/627/head:zhangwenchao-123/support_cloudberry
git switch zhangwenchao-123/support_cloudberry


## Set Python env
sudo alternatives --install /usr/bin/python python /usr/bin/python3 1

## Install required depencies to the Cloudberry Dev container
sudo dnf install boost-devel -y
sudo dnf install -y graphviz # for docs
sudo dnf install --enablerepo=crb doxygen -y # for docs

## 
cd ~/madlib
mkdir build ; cd build

## for Cloudberry 3.0
cmake \
    -DCLOUDBERRY_3_PG_CONFIG=$GPHOME/bin/pg_config \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local/madlib \
    -DCLOUDBERRY_3_EXECUTABLE=$GPHOME/bin/postgres \
    ..

## for Cloudberry 2.0
cmake \
    -DCLOUDBERRY_2_PG_CONFIG=$GPHOME/bin/pg_config \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local/madlib \
    -DCLOUDBERRY_2_EXECUTABLE=$GPHOME/bin/postgres \
    ..

## Make, deploy, and run test
make -j$(nproc)
./src/bin/madpack -p cloudberry -c gpadmin@localhost:7000/postgres install
./src/bin/madpack -p cloudberry -c gpadmin@localhost:7000/postgres install-check

If something wrong, please help correct me. Thanks!

gfphoenix78 added 2 commits October 21, 2025 17:03

Add sized delete operator

76df2b9

tuhaihe reviewed Oct 21, 2025

View reviewed changes

edespino self-requested a review October 22, 2025 01:23

edespino requested changes Oct 22, 2025

View reviewed changes

zhangwenchao-123 force-pushed the support_cloudberry branch from 00d24da to e6d3c42 Compare October 22, 2025 02:55

zhangwenchao-123 force-pushed the support_cloudberry branch 2 times, most recently from 00de02c to 1aad3dd Compare October 22, 2025 06:57

Support madlib2 for lighting.

c725ce5

Fix SEGFAULT memory bugs There're weird SEGFAULT bug due to custom allocation erroneously paired with std::free (should be custom free) and we're unable to solve them. This is a workaround.

zhangwenchao-123 force-pushed the support_cloudberry branch from 6b73eaa to c725ce5 Compare October 23, 2025 06:12

edespino requested changes Oct 23, 2025

View reviewed changes

edespino reviewed Oct 23, 2025

View reviewed changes

tuhaihe reviewed Oct 23, 2025

View reviewed changes

edespino reviewed Oct 23, 2025

View reviewed changes

zhangwenchao-123 added 3 commits October 23, 2025 18:09

Always use python3 for madpack.

303c3fb

fix

783958e

fix

b5289b2

fix

8dac48a

fix NOTICE

4546cfb

tuhaihe reviewed Oct 31, 2025

View reviewed changes

	# Assumes CBDB always follows semantic versioning
	# Assumes Apache Cloudberry always follows semantic versioning

Support cloudberry #627

Are you sure you want to change the base?

Support cloudberry #627

Uh oh!

Conversation

zhangwenchao-123 commented Oct 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edespino left a comment

Choose a reason for hiding this comment

Uh oh!

edespino commented Oct 22, 2025

Uh oh!

edespino commented Oct 22, 2025

Uh oh!

zhangwenchao-123 commented Oct 22, 2025

Uh oh!

tuhaihe commented Oct 22, 2025

Uh oh!

zhangwenchao-123 commented Oct 22, 2025

Uh oh!

zhangwenchao-123 commented Oct 22, 2025

Uh oh!

edespino commented Oct 22, 2025

Uh oh!

edespino commented Oct 22, 2025

Uh oh!

zhangwenchao-123 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edespino left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangwenchao-123 commented Oct 23, 2025 •

edited

Loading