sql: improve inverted index validation observability and runtime

[ajwerner]

Is your feature request related to a problem? Please describe.

Inverted index creation is reasonably efficient. It uses the usual bulk-io index addition goodness. At the end of creating the index we run a validation step. Unlike unique indexes, this step is purely a sanity check. Unfortunately this check is far less optimized than the index construction itself. The check works by running two parallel operations, counting the entries in the new index and reading the primary index to count the implied entries to confirm they match. This can make inverted index validation very slow. This is made dramatically worse by the general lack of observability.

Describe the solution you'd like

• Determine whether we need this validation at all.
  • Why are we even running these queries? What leads us to lack confidence in the adder?
• Provide visibility into the progress of validation (assuming we keep it around)
  • Very loosely related to ui: show estimated schema change progress #10265
  • We actually may have an answer here pending verification, seems like sql: No way to get insight into VALIDATE CONSTRAINT progress #26639 landed in 20.1 so we may be able to leverage it.
• Optimize the validation to use distribution and parallelism, reading the data should not take as long as writing it unless we're doing a bad job!

I think my preference would be to remove the validation from the creation path and move it into something akin to SCRUB (but better because that thing is not good). Then, wherever we move it, we should provide good observability and better performance.

Additional context

Related to #26639

Jira issue: CRDB-3756

[blathers-crl]

Hi @ajwerner, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}