[SPARK-46683] Write a subquery generator that generates subqueries permutations to increase testing coverage

[andylam-db]

What changes were proposed in this pull request?

This PR creates a suite, GeneratedSubquerySuite, that generates SQL with variations of subqueries. These variations include:

1. The location of the subquery in the main query (SELECT, FROM, WHERE)
2. Whether the subquery is correlated, if it is in SELECT or WHERE.
3. The type of subquery predicate, if it is in WHERE.
4. Whether the subquery has a DISTINCT projection.
5. The operators in the subquery: currently there are JOINS, SET OPS, LIMIT and AGGREGATE (sum, count, groupby, no-groupby).

How this works is that this suite generates SQL queries, and are then run against Postgres using docker integration tests.

Why are the changes needed?

There are a lot of subquery correctness issues, ranging from very old bugs to new ones that are being introduced due to work being done on subquery correlation optimization. This is especially in the areas of COUNT bugs and null behaviors.

To increase test coverage and robustness in this area, we want to write a subquery generator that writes variations of subqueries, producing SQL tests.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

This PR adds test. NA.

Was this patch authored or co-authored using generative AI tooling?

No.

[andylam-db]

@cloud-fan @jchen5 Open for reviews! I listed some potential issues in the PR description:

1. There are a lot of queries. There are 14 files, each containing ~300 queries. Adding more variations (such as new operators like WINDOW, OFFSET) would only increase the number of queries. This might slow down tests significantly.
2. A small change in the SubquerySQLGeneratorSuite might introduce a lot of golden file changes. This makes this potentially very difficult to maintain.

[andylam-db]

@cloud-fan @jchen5 Changed it to not produce golden files instead, just a normal test suite.

[andylam-db]

@cloud-fan Can we merge? Thanks!

[cloud-fan]

thanks, merging to master!