coord: enable select sys table persistence behind a flag (default off)

[danhhz]

Rename --persistent_tables to --persistent_user_tables and add
--persistent_system_tables. The latter currently defaults to off and
when on, only opts in the system tables that have self-elected to be
persisted (mz_metrics and mz_metrics_histogram). Once we've stabilized
persistence, we'll default --persistent_system_tables to on, but keep
the flag as a CYA.

Along the way, fix a bug where turning on persisted system tables also
created an unnecessary persistent stream for all system views.

Also improve the persistent stream naming, which would have helped debug
what was going on with this view bug.

[danhhz]

cc @benesch in case he has any opinions on this

[benesch]

No opinions! Seems good!

[danhhz]

Hi! Friendly ping on this @ruchirK :)

[ruchirK]

LGTM! Just to double check -- if a user runs with --persistent-user-tables --experimental creates, a table and then inserts some rows, then restarts with just --experimental they will observe the table but not the previously persisted rows from the last run. Then if they insert some rows those will not be persisted. Then if they run mz again with --persistent-user-tables --experimental they will observe the table with only the rows inserted in the very first run.

If that understanding is correct then this is all great!

[danhhz]

TFTR!

Just ran through your script locally and verified that it does indeed work exactly like that. At some point, we'll want to figure out exactly how all this should work for existing tables, new tables (in existing vs new cluster), etc and write tests since it's easy to mess this stuff up. Feels a bit early for that still IMO, but it's coming up soon

Btw, it looks like SELECT is hanging on persisted user tables at HEAD. I suspect either #7686 or #7659 but we should get that fixed ASAP