release-24.1: opt: correctly reconcile column definition list with RECORD-returning UDFs

[blathers-crl]

Backport 5/5 commits from #119616 on behalf of @DrewKimball.

/cc @cockroachdb/release

---

sql: remove usages of types.IsRecordType

Previously, the types.IsRecordType function was used in different
contexts (with vs without OUT-params, function params vs return type).
This made it difficult to determine whether a particular usage was
correct, and led to a few bugs in cases where additional checks
were necessary.

This commit replaces usages of types.IsRecordType with either:

1. typ.Identical(types.AnyTuple), or
2. typ.Oid() == oid.T_record

The former should be used for a RECORD-returning routine with no
OUT-parameters, as well as for a RECORD-typed variable. The latter
should be used to match either a RECORD-returning routine, or one
with multiple OUT-parameters.

Informs #114846

Release note: None

optbuilder: use actual arg types when building routine with wildcard types

Previously, we would always pass the static parameter types when
building the routine. However, in some cases the static type is
a wildcard, so we actually need to use the actual argument type. Note
that always using the actual argument type can be incorrect (e.g. we'd
lose the tuple labels present in the static type).

Release note: None

opt: refactor optbuild paths for routines and generator functions

This commit heavily refactors the type-handling logic for routines and
generator functions. The hope is to make the code more readable, and also
make the changes in the next commit easier.

Informs #114846

Release note: None

opt: add assignment casts for UDFs used as a data source

Previously, attempting to use a RECORD-returning UDF as a data source
(e.g. SELECT * FROM syntax) would result in an internal error if the
column definition list types didn't match the columns of the last
statement. This commit fixes that by adding validation that the types
are either identical or can be assignment-casted, and adding assignment
casts if necessary.

Fixes #114846
Fixes #113186

Release note (bug fix): Fixed a bug that could cause an internal error of
the form invalid datum type given: ..., expected ... when a RECORD-returning
UDF used as a data source was supplied a column definition list with
mismatched types. This bug has existed since v23.1.0.

opt/optbuilder: check for coercibility instead of tuple types

This commit changes the handling of tuple-returning routines to mirror
that of postgres. In particular, when the routine return type is a tuple,
postgres first attempts to coerce result columns to the return type of the
routine. Only if that attempt fails, postgres wraps the result column in
a tuple, and again attempts the coercion. This change affects the handling
of routines that return (for example) a single composite-typed column. For
example, the following two logic tests should produce the same result:

statement ok
CREATE TYPE two_typ AS (x INT, y INT);
CREATE FUNCTION f() RETURNS two_typ LANGUAGE SQL AS $$ SELECT 1, 2; $$;

query T
SELECT f();
----
(1,2)

vs

statement ok
CREATE TYPE two_typ AS (x INT, y INT);
CREATE FUNCTION f() RETURNS two_typ LANGUAGE SQL AS $$ SELECT ROW(1, 2); $$;

query T
SELECT f();
----
(1,2)

There is not release note, since this shouldn't affect versions prior to 24.1.

Fixes #120942

Release note: None

---

Release justification: bug fix

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[yuzefovich]

Reviewed 9 of 9 files at r1, 2 of 2 files at r2, 12 of 12 files at r3, 5 of 5 files at r4, 7 of 7 files at r5, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @rytaft)

[DrewKimball]

Closing due to failures for routines involving polymorphic types: #119616 (comment).

[yuzefovich]

@DrewKimball should we actually proceed and merge this? IIUC this PR improves the situation for more common scenarios while regressing in less common ones? It also fixes issues that we marked as GA-blockers, so they have been closed but the fix is not present on 24.1 branch which seems wrong.

[DrewKimball]

@DrewKimball should we actually proceed and merge this? IIUC this PR improves the situation for more common scenarios while regressing in less common ones? It also fixes issues that we marked as GA-blockers, so they have been closed but the fix is not present on 24.1 branch which seems wrong.

I think that makes sense for 24.1. I'll try and fix the polymorphic types problems this week.

[yuzefovich]

👍 This probably needs a rebase though.