Skip to content

[fix](regression) Wait for auto partition recycle #65375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shuke987 wants to merge 1 commit into
base: branch-4.1
Choose a base branch
from
Open

[fix](regression) Wait for auto partition recycle #65375

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 19 additions & 16 deletions regression-test/suites/partition_p0/auto_partition/test_auto_new_recycle.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
// specific language governing permissions and limitations
// under the License.

import java.util.concurrent.TimeUnit
import org.awaitility.Awaitility

suite("test_auto_new_recycle", "nonConcurrent") {
sql "drop table if exists auto_recycle"
test {
Expand Down Expand Up @@ -113,6 +116,16 @@ suite("test_auto_new_recycle", "nonConcurrent") {

waitUntilSafeExecutionTime("NOT_CROSS_DAY_BOUNDARY", 20)

def waitPartitionCount = { int expectedCount ->

@github-actions github-actions Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The longer poll makes this final now()-based case much easier to race with midnight. The only guard is still waitUntilSafeExecutionTime("NOT_CROSS_DAY_BOUNDARY", 20), but this helper can now wait up to 1200 seconds, and this final call replaces a fixed 8-second sleep after inserting date_add(now(), interval number-5 day). The recycle code decides whether a partition is historical using the scheduler's current time, so if the rows are inserted shortly before midnight and this await observes the recycle after midnight, the partition for the insert day becomes historical and retention can leave 5 partitions instead of the asserted 6. Please either make this case use a fixed base date/time or move/resize the day-boundary guard so it covers the maximum wait through this final assertion.

Awaitility.await().atMost(1200, TimeUnit.SECONDS).pollInterval(1, TimeUnit.SECONDS).until {
res = sql "show partitions from auto_recycle"
logger.info("partition count: ${res.size()}, expected: ${expectedCount}")
res.size() == expectedCount
}
res = sql "show partitions from auto_recycle"
assertEquals(expectedCount, res.size())
}

sql """
insert into auto_recycle select date_add('2020-01-01 00:00:00', interval number day) from numbers("number" = "100");
"""
Expand All @@ -124,9 +137,7 @@ suite("test_auto_new_recycle", "nonConcurrent") {
assertTrue(res[0][1].contains('"partition.retention_count" = "3"'))

sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '1') """
sleep(8000)
res = sql "show partitions from auto_recycle"
assertEquals(res.size(), 3)
waitPartitionCount.call(3)

// before everytime insert data, we should avoid recycle because if they conflict, insert will fail
sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '600') """
Expand All @@ -135,15 +146,11 @@ suite("test_auto_new_recycle", "nonConcurrent") {
insert into auto_recycle select date_add('2020-01-01 00:00:00', interval number day) from numbers("number" = "100");
"""
sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '1') """
sleep(8000)
res = sql "show partitions from auto_recycle"
assertEquals(res.size(), 3)
waitPartitionCount.call(3)
qt_sql0 "select * from auto_recycle order by k0"

sql "alter table auto_recycle set ('partition.retention_count' = '1')"
sleep(5000)
res = sql "show partitions from auto_recycle"
assertEquals(res.size(), 1)
waitPartitionCount.call(1)

sql "alter table auto_recycle set ('partition.retention_count' = '10')"
sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '600') """
Expand All @@ -152,9 +159,7 @@ suite("test_auto_new_recycle", "nonConcurrent") {
insert into auto_recycle select date_add('2022-01-01 00:00:00', interval number day) from numbers("number" = "100");
"""
sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '1') """
sleep(8000)
res = sql "show partitions from auto_recycle"
assertEquals(res.size(), 10)
waitPartitionCount.call(10)
qt_sql1 "select * from auto_recycle order by k0"

// verify not consider future partition when do recycle
Expand All @@ -175,11 +180,9 @@ suite("test_auto_new_recycle", "nonConcurrent") {
insert into auto_recycle select date_add(now(), interval number-5 day) from numbers("number" = "8");
"""
sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '1') """
sleep(8000)
res = sql "show partitions from auto_recycle"
assertEquals(res.size(), 6) // [-5, -1] -> [-3, -1], [0, 2] -> [0, 2]
waitPartitionCount.call(6) // [-5, -1] -> [-3, -1], [0, 2] -> [0, 2]
res = sql "select * from auto_recycle order by k0"
assertEquals(res.size(), 6)

sql """ admin set frontend config ('dynamic_partition_check_interval_seconds' = '600') """
}
}
Loading