Update table names for databricks_workspace_workspace and databricks_catalog_catalog

[karanpopat]

Example query results

Results

> select
  object_id,
  created_at,
  language,
  object_type,
  path,
  size,
  account_id
from
  databricks_workspace
+------------------+---------------------------+----------+-------------+----------------------------------------+--------+--------------------------------------+
| object_id        | created_at                | language | object_type | path                                   | size   | account_id                           |
+------------------+---------------------------+----------+-------------+----------------------------------------+--------+--------------------------------------+
| 1234497345048935 | <null>                    | <null>   | DIRECTORY   | /Users                                 | <null> | abcd11aa-1175-49c5-8bae-402800e232d5 |
| 1234497345048936 | <null>                    | <null>   | DIRECTORY   | /Shared                                | <null> | abcd11aa-1175-49c5-8bae-402800e232d5 |
+------------------+---------------------------+----------+-------------+----------------------------------------+--------+--------------------------------------+
> select
  name,
  catalog_type,
  comment,
  created_at,
  created_by,
  metastore_id,
  account_id
from
  databricks_catalog
+-----------------------+------------------+--------------------------------------------+---------------------------+-------------------+--------------------------------------+--------------------------------------+
| name                  | catalog_type     | comment                                    | created_at                | created_by        | metastore_id                         | account_id                           |
+-----------------------+------------------+--------------------------------------------+---------------------------+-------------------+--------------------------------------+--------------------------------------+
| test-catalog          | MANAGED_CATALOG  | Test                                       | 2023-07-26T14:08:17+05:30 | user@turbot.com   | abcdf9ea-586a-4d93-830e-84def43cfcf1 | abcd11aa-1175-49c5-8bae-402800e232d5 |
| main                  | MANAGED_CATALOG  | Main catalog (auto-created)                | 2023-07-21T19:16:43+05:30 | user@turbot.com   | abcdf9ea-586a-4d93-830e-84def43cfcf1 | abcd11aa-1175-49c5-8bae-402800e232d5 |
+-----------------------+------------------+--------------------------------------------+---------------------------+-------------------+--------------------------------------+--------------------------------------+

[cbruno10]

@karanpopat Please see comments, thanks!

[cbruno10]

Can this config arg just be called profile instead? Are there multiple types of profiles you can use with the Databricks CLI/SDK?

[cbruno10]

Reference points for naming:

For env vars:

• DATABRICKS_CONFIG_PROFILE
• DATABRICKS_CONFIG_FILE

From Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs#argument-reference):

• profile
• config_file

In their Config struct (https://github.com/databricks/databricks-sdk-go/blob/main/config/config.go#L54-L60):

• Profile
• ConfigFile

[cbruno10]

If we can add support to the plugin to pickup the DEFAULT profile by default if no other config args for auth are mentioned, we can re-add this line in with the appropriate info. I'm good with the change in this PR though as is.

[cbruno10]

Even though it's making an exception to the rule of following the SDK directory naming for services (the most consistent source we could find), we could treat job and pipeline as the service names instead of jobs and pipelines respectively. This would mean we'd have the table names:

• databricks_job
• databricks_job_run
• databricks_pipeline
• databricks_pipeline_event
• databricks_pipeline_update

These seem like better table names than what we originally have