sql: fix decorrelation of correlated CTEs referenced from nested correlated scopes

[ggevay]

Stacked on top of #37505, the first commit should be reviewed there.

Motivation

Fixes SQL-349.

When a correlated CTE is referenced from a nested correlated scope, HIR-to-MIR lowering could decorrelate it with the wrong correlation, producing wrong results. Example (results verified against PostgreSQL):

SELECT *
FROM x,
     LATERAL(WITH a(m) AS (SELECT max(y.a) FROM y WHERE y.a < x.a)
             SELECT * FROM y INNER JOIN LATERAL(SELECT y.a FROM x WHERE (SELECT m FROM a) > 0) ON true);

Note

Stacked on #37505 and should be merged after it. That PR makes CTE LocalIds unique, which this PR's deeper CTE-discovery walk relies on (the Let id-uniqueness assertion in branch() would otherwise fire). Until #37505 merges, this PR's diff includes its commit (sql: allocate globally unique LocalIds for CTEs); review only the second commit here. I'll rebase onto main once #37505 lands.

Description

Two interacting causes in lowering's branch():

1. The walk that collects the outer columns of referenced CTEs used the deprecated visit, which does not descend into scalar subqueries. A CTE referenced only from inside a scalar subquery was therefore missed, so its outer columns were dropped from the branch key.
2. The branch key followed BTreeSet<ColumnRef> order (level ascending). The Get arm reconciles a CTE reference with the relation the CTE was applied to by equating the first cte_outer_arity columns of both sides, which assumes the CTE's outer columns sit at the key's prefix, in order. A level-1 column sorting before the CTE's outer column broke that assumption.

The fix addresses both: discover CTE references with visit_post (descends into subqueries at any depth) and partition the branch key so the CTE outer columns form its prefix, ordered by position. Referenced CTEs' outer relations are nested prefixes of one another (nested scopes only ever append columns), so a single prefix covers them all.

Both changes are gated behind the new enable_fixed_correlated_cte_lowering optimizer feature flag, default on. With the flag off, lowering is byte-identical to before, which gives a rollback lever for upgrade safety: the plan change only lands for objects that reference a correlated CTE from a correlated scope, and flipping the flag off and restarting re-optimizes back to the old plans.

Follow-up outside this PR: the enable_fixed_correlated_cte_lowering flag must be created in LaunchDarkly to be controllable in production. It is temporarily added to KNOWN_MISSING_FROM_LD so the nightly LD-consistency check does not fail meanwhile.

Verification

Regression tests added to test/sqllogictest/cte_lowering.slt, along with a flag-off EXPLAIN DECORRELATED PLAN WITH(enable fixed correlated cte lowering = false) that documents the old (broken) plan next to the fixed one. Expected query results verified against PostgreSQL 16. Ran the CTE, subquery, and decorrelated/optimized/raw plan-as-text sqllogictest files with no fallout beyond the intended cte_lowering.slt re-bless.

Nightly subset run: https://buildkite.com/materialize/nightly/builds/17069

[ggevay]

@mgree, this is ready for review.

[mgree]

Looks good, though it's hard for me to follow every detail of the column tracking. Not sure my suggestion is an improvement, so feel free to ignore it.

[mgree]

If we know the exact length, could we instead write into the vec by index rather than sorting after? Might be clearer. (I'm definitely having some trouble tracking all of the various column bits and bobs!)

[ggevay]

Thanks! I went back and forth here. Writing into a fixed-size vec by index does drop the sort, but it trades it for Option slots plus separate duplicate- and gap-checks, so I think it's roughly a wash on complexity rather than a clear win — and the genuinely hard-to-follow part is the level/position/ColumnRef juggling, which that rewrite doesn't really change. So I kept the sort but added comments around it: naming what position is, spelling out that prefix-first-then-sorted is what makes key[i] == i (the property the Get arm relies on), and clarifying the -1 level compensation. Hopefully that makes it easier to follow — happy to revisit if you'd still prefer the index version.