changefeedccl: add randomized test#134160
changefeedccl: add randomized test#134160rharding6373 wants to merge 1 commit intocockroachdb:masterfrom
Conversation
This PR adds TestChangefeedRandomized, a randomized test that uses sqlsmith to exercise changefeeds on generated tables with generated queries. It performs the following: * Create a random table using sqlsmith. * Insert initial random data into the table. * Set up changefeed validators. * Create changefeed. * Execute 100 mutations using sqlsmith query generator. The test has the following limitations, which will be lifted in future PRs: * Does not support multiple tables. * Does not support geo types. * Does not support column families. * Does not support all validators. * Only exercises a couple changefeed options. * Only supports kafka. Epic: CRDB-42866 Informs: cockroachdb#124146 Release note: None
| log.Infof(ctx, "generated query %d in %s", i, time.Since(start)) | ||
| t.Logf("query %d: %s", i, query) | ||
| start = time.Now() | ||
| _, _ = s.DB.Exec(query) |
There was a problem hiding this comment.
I notice this as I am working with adding sql smith to TestChangefeedNemeses.
s.DB.exec actually returns an error on this line quite often. Is it expected that sql smith may return insert statements that may not be valid to be executed on?
randomized_test.go:121: error executing query 99: pq: duplicate key value violates unique constraint "rand_table_expr_key"
There was a problem hiding this comment.
I saw this error when changing _, _ = s.DB.Exec(query) to
_, err = s.DB.Exec(query)
if err != nil {
t.Errorf("error executing query %d: %s", i, err)
}
There was a problem hiding this comment.
Looks like this seems to be expected and we just skip them in TestChangefeedRandomExpressions
cockroach/pkg/ccl/changefeedccl/changefeed_test.go
Lines 1138 to 1142 in 928e835
sample output:
changefeed_test.go:1139: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE similar_escape(e'\U00002603':::STRING::STRING, ('1978-09-11':::DATE::DATE || NULL::STRING)::STRING::STRING)::STRING::STRING IN (SELECT (seed._string::STRING || parse_time(lpad(e'TP\x7f\x13 ':::STRING::STRING, (seed._int8::INT8 % seed._int4::INT8)::INT8::INT8, seed._string::STRING)::STRING::STRING, seed._string::STRING)::TIME::TIME)::STRING::STRING AS col_10 FROM d.public.seed AS tab_4) because error pq: parse_time(): unknown DateStyle parameter: 1
Update: Thinking more about this: I guess this makes sense since sql smith probably doesn't track the internal state of the table and knows what would be valid?
This PR adds TestChangefeedRandomized, a randomized test that uses sqlsmith to exercise changefeeds on generated tables with generated queries. It performs the following:
The test has the following limitations, which will be lifted in future PRs:
Epic: CRDB-42866
Informs: #124146
Release note: None