Add new dsc command

[Gijsreyn]

Changes

Adds a new databricks dsc command providing Microsoft Desired State Configuration compatible resources for managing Databricks workspace resources.

The structure of the command operations is as follows:

databricks dsc <get|test|set|delete|export> --resource <type> --input '<json>'

To generate the schemas (and manifest), users can: databricks dsc manifest or databricks dsc schema --resource <type>.

Before implementing many more resources, this initial PR includes:

• Databricks.DSC/Secret - Manage secrets in a scope through DSC
• Databricks.DSC/SecretScope - Manage secret scopes through DSC
• Databricks.DSC/SecretAcl - Manage secret ACLs through DSC
• Databricks.DSC/User - Manage workspace users through DSC

It re-uses existing Databricks SDK types directly and uses the _exist property for declarative create/delete semantics if hooked into dsc.exe.

Why

I choose to integrate DSC resources directly into the executable rather than creating separate DSC resources (or exes) because it has significant advantages. Firstly, users can add databricks.exe to their PATH once, and all DSC resources are available immediately. Then, when users want to leverage it through dsc.exe, they can produce the resource manifest and hook it into the semantics of dsc.exe. Lastly, it simplifies the maintenance because it is added to one codebase and leverages existing SDK types + authentication.

I explored several designs:

• Initially considered separate commands per resource (databricks dsc secret get), but this doesn't align with how DSC manifests specify commands - DSC expects a single executable with arguments that include the resource type.
• The --resource flag allows the manifest to specify ["dsc", "get", "--resource", "Databricks.DSC/Secret", {"jsonInputArg": "--input"}] which DSC understands natively.
• The --input flag (with -i shorthand) accepts JSON directly or via stdin, matching DSC's jsonInputArg pattern.

This PR demonstrates how easy it is to add new DSC resources - each resource is a single file implementing the ResourceHandler interface with Get, Set, Delete, and Export methods. For more complex resources, it might be beneficial to include Test if the synthetic test that dsc.exe executed doesn't provide the proper details.

Tests

• Unit tests for helper functions (validation, JSON parsing, error formatting)
• Unit tests for schema generation (property descriptions, _exist property injection)
• Unit tests for command structure and manifest building

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 4349
• Commit SHA: 8f08fad8c63ee02b53be758bde74d13b40a9d791

Checks will be approved automatically on success.

[pietern]

We recommend using the Databricks Terraform Provider or Databricks Asset Bundles for declarative configuration.

DSC integration could merit its own dedicated CLI instead of being natively integrated.

If you're interested in making future contributions to this repository, please file an issue first so we can discuss if we're open to taking a contribution. We're not invested in DSC at the moment, so can't take this contribution.

[Gijsreyn]

@pietern - thank you for at least taking the time to review my PR and provide your feedback. I understand that Databricks is currently focused on Terraform and DABs for declarative configuration, and that DSC isn't part of the roadmap at this time.

My goal was mainly experimental and exploratory. I wanted to test the feasibility of integrating DSC natively into databricks.exe and see how it could modernize it in a command-based DSC resources with the existing tooling. It wasn't intended to replace any existing tools/workflows.

Regardless, I understand that this isn't a priority for the project right now and appreciate your guidance on filing an issue for future contributions. Thanks again, and I hope this experiment can at least serve as inspiration for others.

[pietern]

Understood, thanks for the reply!