release-23.1.0: opt: check privileges after data sources in memo staleness check

[DrewKimball]

Backport 1/1 commits from #102405.

/cc @cockroachdb/release

---

Previously, the opt.Metadata staleness check would check the access privileges for a data source immediately after checking that the data source still resolves as it originally did. This meant that the privilege check for one data source could occur before checking staleness of other data sources. Before #96045 this wasn't a problem, since these checks would happen in the order in which the data sources were resolved. But after #96045, the data sources were stored in a map instead of a slice, which made the order in which the checks are performed nondeterministic.

This change in behavior can cause queries to return insufficient privilege errors even when the current user has access privileges on all data sources referenced by the query. This is because the query may implicitly reference a table by ID if there is a foreign key relation from an explicitly referenced data source. If the database is changed, this ID reference will still resolve the same, but if the user is changed to one without privileges on that table, the staleness check will result in an error. This wasn't a problem before because the referencing table would always be checked first, causing the staleness check to finish without an error.

This patch fixes the bug by moving the privilege checks until after all data source resolution checks have succeeded.

Fixes #102375

Release note (bug fix): Fixed a bug introduced in versions 22.1.19, 22.2.8, and pre-release versions of 23.1 that could cause queries to return spurious insufficient privilege errors. For the bug to occur, two databases would need to have duplicate tables each with a foreign key reference to another table. The error would then occur if the same SQL string was executed against both databases concurrently by users that have privileges over only one of the tables.

Release justification: fix for bug introduced by previous backport

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is 

[rytaft]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DrewKimball)

[mgartner]

Nice find!

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained