Upgrade from (Apache Cloudberry) 2.0.0 to (Apache Cloudberry) 2.1.0

[adnanhamdussalam]

Hi,
I have try to upgrade database from 2.0.0 to 2.1.0 but after changing the binary I facing the below error during startup.

2026-03-26 08:03:56.981525 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","starting PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit",,,,,,,0,,"postmaster.c",1380,
2026-03-26 08:03:56.981835 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","listening on IPv4 address ""0.0.0.0"", port 55432",,,,,,,0,,"pqcomm.c",623,
2026-03-26 08:03:56.981940 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","listening on IPv6 address ""::"", port 55432",,,,,,,0,,"pqcomm.c",623,
2026-03-26 08:03:56.983356 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","listening on Unix socket ""/tmp/.s.PGSQL.55432""",,,,,,,0,,"pqcomm.c",618,
2026-03-26 08:03:56.994658 EDT,,,p2473763,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted; last known up at 2026-03-25 13:50:32 EDT",,,,,,,0,,"xlog.c",6887,
2026-03-26 08:03:56.995061 EDT,,,p2473763,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","Synchronization of the wal directory starts.",,,,,,,0,,"fd.c",3453,
2026-03-26 08:03:56.995994 EDT,,,p2473763,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","synchronization of the wal directory finishes.",,,,,,,0,,"fd.c",3455,
2026-03-26 08:03:56.996553 EDT,,,p2473763,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","invalid primary checkpoint record at location 0/4995010",,,,,,,0,,"xlog.c",8981,
2026-03-26 08:03:57.032523 EDT,,,p2473763,th-1677551424,,,,0,,,seg-1,,,,,"PANIC","XX000","could not locate a valid checkpoint record",,,,,,,0,,"xlog.c",7201,"Stack trace:
1 0xd6a018 postgres errstart + 0x258
2 0x6f8a16 postgres + 0x6f8a16
3 0xb79a65 postgres StartupProcessMain + 0x105
4 0x8c6382 postgres AuxiliaryProcessMain + 0x5a2
5 0xb74835 postgres + 0xb74835
6 0xb79378 postgres PostmasterMain + 0xde8
7 0x7c45eb postgres main + 0x56b
8 0x7ffa9ba2a610 libc.so.6 + 0x9ba2a610
9 0x7ffa9ba2a6c0 libc.so.6 __libc_start_main + 0x80
10 0x7d5075 postgres _start + 0x25
"
2026-03-26 08:03:57.754716 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","startup process (PID 2473763) was terminated by signal 6: Aborted",,,,,,,0,,"postmaster.c",4285,
2026-03-26 08:03:57.754899 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","aborting startup due to startup process failure",,,,,,,0,,"postmaster.c",3483,
2026-03-26 08:03:57.786504 EDT,,,p2473761,th-1677551424,,,,0,,,seg-1,,,,,"LOG","00000","database system is shut down",,,,,,,0,,"miscinit.c",1030,

[my-ship-it]

Hi @adnanhamdussalam,

Thanks for reporting this issue and providing the detailed logs — they're really helpful for diagnosing the problem!

I took a look and compared the internal version identifiers (CATALOG_VERSION_NO, PG_CONTROL_VERSION, XLOG_PAGE_MAGIC, GP schema version) between 2.0.0 and 2.1.0. Good news: they are all identical, which means an in-place binary upgrade between these two versions should work just fine in general.

So the issue you're seeing is most likely not caused by version incompatibility. Looking at the logs more closely, this line stands out:

database system was interrupted; last known up at 2026-03-25 13:50:32 EDT

This tells us the old 2.0.0 cluster wasn't cleanly shut down before the binary swap. Because of that, the database needs to perform WAL replay (crash recovery) to get back to a consistent state — but the checkpoint record it's looking for appears to be incomplete or invalid.

Here's what I'd recommend to recover:

1. Restore the original 2.0.0 binaries.
2. Start the cluster with gpstart -a to let it perform crash recovery with the matching binaries.
3. Once it's up and running, do a clean shutdown: gpstop -a -M fast.
4. Now you can safely swap in the 2.1.0 binaries and start the cluster again.

If step 2 also fails (meaning even the old binaries can't recover), the WAL/checkpoint data may have been affected by the interrupted startup attempt with the new binary. In that case, pg_resetwal could be used as a last resort, or you may need to restore from a backup.

Feel free to follow up if you run into any other issues along the way — happy to help! 😊

[adnanhamdussalam]

Hi @my-ship-it,

Yes, your analysis is 100 % correct I have used pg_resetwal -f and then database started successfully.
Then I retested with database grace full shutdown the version upgraded from 2.0.0 to 2.1.0 so there was not version compatibility.

Thank you for your support guys @tuhaihe / @my-ship-it