Pipeline triggers don't work - Please reconsider filtering out "deprecated" fields from the experimental Python interface

[jdavidheiser]

Describe the issue

There are some configurations that have been marked as deprected in the API docs for a very long time, but which continue to be valuable. In https://github.com/databricks/cli/pull/2773/files, the asset bundle wrappers were updated to enforce that all deprecated fields are removed from the corresponding codegen for Dataclasses and TypedDicts.

Given that not all deprecations happen as planned, this is essentially removing useful functionality from DABs without providing an alternative.

Steps to reproduce the behavior

Try to use the triggers configuration option specified in the API at https://docs.databricks.com/api/workspace/pipelines/create

Expected Behavior

We should be able to continue using any API feature which currently works. Pending deprecation is not the same as actually all the way deprecated, and API teams sometimes mark things as deprecated even when they are still useful. Nobody on the Pipelines API team has provided any other interface for scheduling pipelines, despite that functionality still being readily available in the Databricks UI.

Actual Behavior

The Python DAB will reject the trigger field, since it's been filtered out of the Pipeline dataclass, despite still being a valid way to call the API.

This essentially makes it impossible to use DABs to create scheduled Delta Live Tables pipelines, which is important functionality.

OS and CLI version

Please include the version of the CLI (eg: v0.1.2) and the operating system (eg: windows). You can run databricks --version to get the version of your Databricks CLI

Is this a regression?

Possibly, I haven't tried this in DABs before, but it works in the Databricks SDK.

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks clusters list --log-level=debug. Redact if needed

[kanterov]

@jdavidheiser do you have an example of a configuration that successfully deploys when declared in YAML but not in Python? We have reviewed which fields are absent in dataclasses, and haven't found a use for them. We opted out cases when a useful field is marked as deprecated.

To my best knowledge, for the trigger field, Pipelines API returns an error when it's specified. So I'm wondering what is the use-case for it.

[kanterov]

@jdavidheiser I will close the issue for now. Please re-open if needed.