[SPARK-46446][SQL] Disable subqueries with correlated OFFSET to fix correctness bug

[jchen5]

What changes were proposed in this pull request?

Subqueries with correlation under LIMIT with OFFSET have a correctness bug, introduced recently when support for correlation under OFFSET was enabled but were not handled correctly. (So we went from unsupported, query throws error -> wrong results.) This is in master branch, not yet released.

This PR first disables correlated OFFSET by adding a feature flag for it, which is disabled. Next PR will add support for it and re-enable it. This PR also adds a feature flag for the related LIMIT support (which is enabled).

It’s a bug in all types of correlated subqueries: scalar, lateral, IN, EXISTS

Example repro:

create table x(x1 int, x2 int);
insert into x values (1, 1), (2, 2);
create table y(y1 int, y2 int);
insert into y values (1, 1), (1, 2), (2, 4);

select * from x where exists (select * from y where x1 = y1 limit 1 offset 2)

Correct result: empty set
Spark result: Array([2,2])

Why are the changes needed?

Correctness bug

Does this PR introduce any user-facing change?

Disables correlated OFFSET query shape which was not handled correctly. (This was enabled on master branch but not yet released.)

How was this patch tested?

Add tests

Was this patch authored or co-authored using generative AI tooling?

No

[jchen5]

@agubichev @cloud-fan

[dongjoon-hyun]

Could you make the CI happy, @jchen5 ?

[cloud-fan]

The docker and pyspark test failures are unrelated, thanks, merging to master!

[jchen5]

I had some small test fixes that missed this PR, follow-up PR to fix the tests here: #44415