Skip to content

changefeedccl: add randomized test#134160

Draft
rharding6373 wants to merge 1 commit intocockroachdb:masterfrom
rharding6373:20240808_cdc_randomized_test_124146_2
Draft

changefeedccl: add randomized test#134160
rharding6373 wants to merge 1 commit intocockroachdb:masterfrom
rharding6373:20240808_cdc_randomized_test_124146_2

Conversation

@rharding6373
Copy link
Collaborator

This PR adds TestChangefeedRandomized, a randomized test that uses sqlsmith to exercise changefeeds on generated tables with generated queries. It performs the following:

  • Create a random table using sqlsmith.
  • Insert initial random data into the table.
  • Set up changefeed validators.
  • Create changefeed.
  • Execute 100 mutations using sqlsmith query generator.

The test has the following limitations, which will be lifted in future PRs:

  • Does not support multiple tables.
  • Does not support geo types.
  • Does not support column families.
  • Does not support all validators.
  • Only exercises a couple changefeed options.
  • Only supports kafka.

Epic: CRDB-42866
Informs: #124146

Release note: None

This PR adds TestChangefeedRandomized, a randomized test that uses
sqlsmith to exercise changefeeds on generated tables with generated
queries. It performs the following:
* Create a random table using sqlsmith.
* Insert initial random data into the table.
* Set up changefeed validators.
* Create changefeed.
* Execute 100 mutations using sqlsmith query generator.

The test has the following limitations, which will be lifted in future
PRs:
* Does not support multiple tables.
* Does not support geo types.
* Does not support column families.
* Does not support all validators.
* Only exercises a couple changefeed options.
* Only supports kafka.

Epic: CRDB-42866
Informs: cockroachdb#124146

Release note: None
@rharding6373 rharding6373 requested a review from a team as a code owner November 3, 2024 05:04
@cockroach-teamcity
Copy link
Member

This change is Reviewable

log.Infof(ctx, "generated query %d in %s", i, time.Since(start))
t.Logf("query %d: %s", i, query)
start = time.Now()
_, _ = s.DB.Exec(query)
Copy link
Contributor

@wenyihu6 wenyihu6 Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice this as I am working with adding sql smith to TestChangefeedNemeses.

s.DB.exec actually returns an error on this line quite often. Is it expected that sql smith may return insert statements that may not be valid to be executed on?

randomized_test.go:121: error executing query 99: pq: duplicate key value violates unique constraint "rand_table_expr_key"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw this error when changing _, _ = s.DB.Exec(query) to

_, err = s.DB.Exec(query)
if err != nil {
t.Errorf("error executing query %d: %s", i, err)
}

Copy link
Contributor

@wenyihu6 wenyihu6 Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this seems to be expected and we just skip them in TestChangefeedRandomExpressions

if err := rows.Scan(&expectedRowIDsStr); err != nil {
t.Logf("Skipping query %s because error %s", query, err)
cancel()
continue
}
?

sample output:

    changefeed_test.go:1139: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE similar_escape(e'\U00002603':::STRING::STRING, ('1978-09-11':::DATE::DATE || NULL::STRING)::STRING::STRING)::STRING::STRING IN (SELECT (seed._string::STRING || parse_time(lpad(e'TP\x7f\x13 ':::STRING::STRING, (seed._int8::INT8 % seed._int4::INT8)::INT8::INT8, seed._string::STRING)::STRING::STRING, seed._string::STRING)::TIME::TIME)::STRING::STRING AS col_10 FROM d.public.seed AS tab_4) because error pq: parse_time(): unknown DateStyle parameter: 1

Update: Thinking more about this: I guess this makes sense since sql smith probably doesn't track the internal state of the table and knows what would be valid?

@rharding6373 rharding6373 marked this pull request as draft December 9, 2024 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants