Skip to content

2.31.0.0-b249

2.31.0.0-b249: [#32302] YSQL: Fix flaky PgIndexBackfillReadBeforeConcurrentUpdate test
@myang2021 myang2021 tagged this 10 Jul 20:27 
Summary:
The test `PgIndexBackfillReadBeforeConcurrentUpdate` (and others
inheriting from `PgIndexBackfillBlockDoBackfill`) sets
`--TEST_block_do_backfill=true` to pause the backfill process so it can
perform concurrent updates. The write thread waits for the index to
reach `indisready` (with a 30s timeout) and then waits for the backfill
job to acquire a safe time (with a 60s timeout).

However, the default `master_ysql_operation_lease_ttl_ms` is 30s, which
causes the master to reject `WaitForYsqlBackendsCatalogVersion` RPCs for
the first 30 seconds after becoming leader. Because
`wait_for_ysql_backends_catalog_version_client_master_rpc_timeout_ms`
was reverted to 20s in D54240, the PG backend retries the RPC until the
30s lease expires, delaying the `CREATE INDEX` operation by ~30 seconds
before it reaches `indisready`.

Meanwhile, the main thread unblocks the backfill immediately after
starting the write thread, and then waits for 60s for the threads to
finish. Since `CREATE INDEX` is delayed by 30s, the unblock happens too
early. When `CREATE INDEX` finally reaches `DoBackfill()`,
`TEST_block_do_backfill` is already false, so it doesn't block and
finishes the backfill immediately. The write thread, which was also
delayed by 30s waiting for `indisready`, then tries to wait for the
backfill job to have a safe time, but the job has already been removed,
causing it to time out after 60s.

This commit fixes the flakiness by restoring
`--master_ysql_operation_lease_ttl_ms=10000` to the base
`PgIndexBackfillTest` class (which was removed by D54240),
reducing the lease TTL so `CREATE INDEX` proceeds quickly
and reaches `DoBackfill()` while `TEST_block_do_backfill` is still true.

The regression commit D54240 (b65fbe2dda) was backported to 2026.1 and
2025.2, so this fix needs to be backported to those branches as well.

Test Plan: ./yb_build.sh release --cxx-test pgwrapper_pg_index_backfill-test --gtest_filter 'PgIndexBackfillReadBeforeConcurrentUpdate.PartialIndex/1' -n 10

Reviewers: jason

Reviewed By: jason

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D55466
Assets 2
Loading