feat: catalog support for Databricks native

[betodealmeida]

SUMMARY

Add support for catalogs in Databricks, and improve the migration that updates permissions and models with the catalog.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Before running the upgrade, add a Databricks database, create a dataset, make a chart, run and save a query. They all should have no catalog information:

databricks_catalogs=# SELECT catalog FROM saved_query UNION ALL SELECT catalog FROM query UNION ALL SELECT catalog FROM tab_state UNION ALL SELECT catalog FROM table_schema UNION ALL SELECT catalog FROM tables;
 catalog
---------





(5 rows)

databricks_catalogs=# SELECT schema_perm FROM slices UNION ALL SELECT schema_perm FROM tables;      schema_perm
------------------------
 [Databricks].[default]
 [Databricks].[default]
(2 rows)

databricks_catalogs=# SELECT * FROM ab_view_menu JOIN ab_permission_view ON ab_view_menu.id = view_menu_id JOIN ab_permission ON ab_permission.id = permission_id WHERE ab_permission.name IN ('schema_access', 'catalog_access');
 id  |             name              | id  | permission_id | view_menu_id | id |      name
-----+-------------------------------+-----+---------------+--------------+----+----------------
  76 | [Databricks].[default]        | 167 |            74 |           76 | 74 | schema_access
(1 row)

Run the upgrade, and check that the catalog is present:

databricks_catalogs=# SELECT catalog FROM saved_query UNION ALL SELECT catalog FROM query UNION ALL SELECT catalog FROM tab_state UNION ALL SELECT catalog FROM table_schema UNION ALL SELECT catalog FROM tables;
    catalog
----------------
 hive_metastore
 hive_metastore
 hive_metastore
 hive_metastore
 hive_metastore
(5 rows)

databricks_catalogs=# SELECT schema_perm FROM slices UNION ALL SELECT schema_perm FROM tables;
               schema_perm
-----------------------------------------
 [Databricks].[hive_metastore].[default]
 [Databricks].[hive_metastore].[default]
(2 rows)

databricks_catalogs=# SELECT * FROM ab_view_menu JOIN ab_permission_view ON ab_view_menu.id = view_menu_id JOIN ab_permission ON ab_permission.id = permission_id WHERE ab_permission.name IN ('schema_access', 'catalog_access');
 id  |                  name                   | id  | permission_id | view_menu_id | id |      name
-----+-----------------------------------------+-----+---------------+--------------+----+----------------
  76 | [Databricks].[hive_metastore].[default] | 167 |            74 |           76 | 74 | schema_access
 110 | [Databricks].[hive_metastore]           | 192 |            76 |          110 | 76 | catalog_access
(3 rows)

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[eschutho]

LGTM