Skip to content

2.25.0.0-b146

2.25.0.0-b146: [#8096] YSQL: Fix flaky test TestPgDdlConcurrency#testModifiedTableWrite
@myang2021 myang2021 tagged this 16 Oct 03:59 
Summary:
The test is runs a DDL thread concurrently with 50 DML threads. It expects to
see some race condition and the corresponding resulting error messages. In the
end of the test, it asserts there must be at least one expected error message
that indicates DDL and DML race condition has happened.

The two DDL statements of the DDL thread are:

```
              lstmt.execute("ALTER TABLE t DROP COLUMN IF EXISTS v");
              lstmt.execute("ALTER TABLE t ADD COLUMN v INT DEFAULT 100");
```

The DML statements of the DML threads are:
```
                lstmt.execute(String.format("INSERT INTO t(k) VALUES(%d), (%d)",
                                            idx * 10000000L + item_idx,
                                            idx * 10000000L + item_idx + 1));
```

I have found two reasons for this test to be flaky:

1. After the DDL atomicity project, a new expected error message is introduced:
```
ts1|pid64786|:31504 2024-10-11 20:21:17.524 UTC [65421] ERROR:  [Invalid argument (yb/docdb/pgsql_operation.cc:473): Column with id 4 marked for deletion in table 000034cb000030008000000000004000]
```

We need to add this new error message to the set of excepted exceptions.

2. A known issue is "Invalid column number" which can happen when PG table
metadata is out of sync with DocDB table metadata. After the DDL atomicity
project we should not see persistent out of sync between them but because they
are not read atomically from a PG backend they can still be out of sync. We can
potentially have an automatic retry to mitigate the problem. For now I treat it
as an expected error to allow the test to pass.

3. The test only ran a pair of DDL statements and only ran 10 seconds. If there
is no expected errors encountered the test will fail. To make the test more
reliable, I added 20 iterations of executing the DDL statements and also
increased the test duration to 100 seconds.
Jira: DB-4809

Test Plan:
./yb_build.sh release --java-test 'org.yb.pgsql.TestPgDdlConcurrency#testModifiedTableWrite' -n 20 --tp 1
./yb_build.sh debug --java-test 'org.yb.pgsql.TestPgDdlConcurrency#testModifiedTableWrite' -n 20 --tp 1

Reviewers: kfranz, mihnea

Reviewed By: kfranz

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D39036
Assets 2
Loading