[Bug] Tests uao_compaction fail intermittenly

[leborchuk]

Apache Cloudberry version

main branch

What happened

I see that tests uao_compaction fail from time to time

Example of failed output:

test uao_compaction/full_stats    ... ok          218 ms (diff   60 ms)
test uao_compaction/stats         ... FAILED      333 ms (diff  123 ms)
test uao_compaction/index_stats   ... FAILED      575 ms (diff  139 ms)
test uao_compaction/index         ... ok          161 ms (diff   59 ms)
test uao_compaction/drop_column   ... FAILED      165 ms (diff  122 ms)

regression.diff

diff -I HINT: -I CONTEXT: -I GP_IGNORE: -U3 /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/stats.out /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/stats.out
--- /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/stats.out       2024-12-01 01:55:53.193407004 -0800
+++ /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/stats.out        2024-12-01 01:55:53.193407004 -0800
@@ -38,7 +38,7 @@
 SELECT relname, reltuples FROM pg_class WHERE relname = 'uao_stats_index';
      relname     | reltuples
 -----------------+-----------
- uao_stats_index |        88
+ uao_stats_index |       113
 (1 row)

 -- re-setup for next case
@@ -61,6 +61,6 @@
 SELECT relname, reltuples FROM pg_class WHERE relname = 'uao_stats_index';
      relname     | reltuples
 -----------------+-----------
- uao_stats_index |        85
+ uao_stats_index |       185
 (1 row)

diff -I HINT: -I CONTEXT: -I GP_IGNORE: -U3 /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/index_stats.out /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/index_stats.out
--- /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/index_stats.out 2024-12-01 01:55:53.929414023 -0800
+++ /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/index_stats.out  2024-12-01 01:55:53.933414061 -0800
@@ -33,7 +33,7 @@
 SELECT relname, reltuples FROM pg_class WHERE relname = 'mytab_int_idx1';
     relname     | reltuples
 ----------------+-----------
- mytab_int_idx1 |         2
+ mytab_int_idx1 |         4
 (1 row)

 -- Test to ensure that reltuples is updated for an index after lazy vacuum.
diff -I HINT: -I CONTEXT: -I GP_IGNORE: -U3 /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/drop_column.out /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/drop_column.out
--- /__w/cloudberry/cloudberry/src/test/regress/expected/uao_compaction/drop_column.out 2024-12-01 01:55:54.493419402 -0800
+++ /__w/cloudberry/cloudberry/src/test/regress/results/uao_compaction/drop_column.out  2024-12-01 01:55:54.493419402 -0800
@@ -39,7 +39,7 @@
 SELECT relname, reltuples FROM pg_class WHERE relname = 'uao_drop_col_index';
       relname       | reltuples
 --------------------+-----------
- uao_drop_col_index |         7
+ uao_drop_col_index |        13
 (1 row)

Cannot reproduce it on my local environment, but see failed builds:

https://github.com/apache/cloudberry/actions/runs/12083671444/job/33747428603?pr=741
https://github.com/apache/cloudberry/actions/runs/12350255796/job/34465146518
https://github.com/apache/cloudberry/actions/runs/12297472495/job/34319275423

All these 3 examples contains the same failed uao_compaction/index_stats test which succeeds on restart. All PRs do not relate to UAO compaction functionality.

Flaky tests are annoying. It is better to fix them somehow.

What you think should happen instead

No response

How to reproduce

run ic-good-opt-off test workflow

Operating System

rocky linux 8/9

Anything else

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.

[github-actions]

Hi, @leborchuk welcome!🎊 Thanks for taking the time to point this out.🙌

[my-ship-it]

Hi @leborchuk, the diff is introduced by cherry pick from Greenplum.

[my-ship-it]

Hi @leborchuk, the diff is introduced by cherry pick from Greenplum.

The PR to fix this problem will be sent out later.

[gongxun0928]

after cherry-pick e9d862f from greenplum.
Index dead tuples will not always be cleaned up completely after VACUUM, resulting
index stats pg_class->reltuples will not always be accurate.

and we should revert the changes of the expected files in this commit fdfea20