-
Notifications
You must be signed in to change notification settings - Fork 125
Add support for .dbalert.json files
#3602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
6c1c737
Add support for .dbalert.json files
shreyas-goenka 2a60238
Merge remote-tracking branch 'origin' into dbalert
shreyas-goenka 75034f1
merge
shreyas-goenka e187e23
update with latest changes
shreyas-goenka fdc778c
fixups
shreyas-goenka 3875002
only read the lines field
shreyas-goenka 8d7d38f
fix unit test
shreyas-goenka b74d8b1
fix
shreyas-goenka bec49a5
Merge remote-tracking branch 'origin' into dbalert
shreyas-goenka 0c9160c
address feedback
shreyas-goenka 0f4ffe1
gpoh
shreyas-goenka 02cba25
add export roundback
shreyas-goenka ee1ee53
update test
shreyas-goenka d1e03f8
disable test on local
shreyas-goenka 47395a3
Merge remote-tracking branch 'origin' into dbalert
shreyas-goenka 15faa0f
fix the with_file test
shreyas-goenka 0ef378d
Merge remote-tracking branch 'origin' into dbalert
shreyas-goenka 5a7a352
fix the alerts test
shreyas-goenka 9095a33
enable msys
shreyas-goenka 383c218
use the --strip-trailing-cr option
shreyas-goenka File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
acceptance/bundle/resources/alerts/with_file/alert.dbalert.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| { | ||
| "custom_summary": "My alert from file", | ||
| "evaluation": { | ||
| "source": { | ||
| "name": "1", | ||
| "display": "1", | ||
| "aggregation": "MAX" | ||
| }, | ||
| "comparison_operator": "EQUAL", | ||
| "threshold": { | ||
| "value": { | ||
| "double_value": 2.0 | ||
| } | ||
| }, | ||
| "notification": { | ||
| "retrigger_seconds": 1, | ||
| "notify_on_ok": false | ||
| } | ||
| }, | ||
| "schedule": { | ||
| "quartz_cron_schedule": "44 19 */1 * * ?", | ||
| "timezone_id": "Europe/Amsterdam" | ||
| }, | ||
| "query_lines": [ | ||
| "select 2" | ||
| ], | ||
| "custom_description_lines": [ | ||
| "My alert from file", | ||
| "with multiple lines", | ||
| "and a third line" | ||
| ] | ||
| } |
9 changes: 9 additions & 0 deletions
9
acceptance/bundle/resources/alerts/with_file/databricks.yml.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| bundle: | ||
| name: alerts-with-file-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| alerts: | ||
| myalert: | ||
| warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID | ||
| display_name: "myalert" | ||
| file_path: ./alert.dbalert.json |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
100 changes: 100 additions & 0 deletions
100
acceptance/bundle/resources/alerts/with_file/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
|
|
||
| >>> [CLI] bundle validate -o json | ||
| { | ||
| "custom_description": "My alert from file\nwith multiple lines\nand a third line\n", | ||
| "custom_summary": "My alert from file", | ||
| "display_name": "myalert", | ||
| "evaluation": { | ||
| "comparison_operator": "EQUAL", | ||
| "notification": { | ||
| "notify_on_ok": false, | ||
| "retrigger_seconds": 1 | ||
| }, | ||
| "source": { | ||
| "aggregation": "MAX", | ||
| "display": "1", | ||
| "name": "1" | ||
| }, | ||
| "threshold": { | ||
| "value": { | ||
| "double_value": 2 | ||
| } | ||
| } | ||
| }, | ||
| "file_path": "alert.dbalert.json", | ||
| "parent_path": "/Workspace/Users/[USERNAME]/.bundle/alerts-with-file-[UNIQUE_NAME]/default/resources", | ||
| "query_text": "select 2\n", | ||
| "schedule": { | ||
| "quartz_cron_schedule": "44 19 */1 * * ?", | ||
| "timezone_id": "Europe/Amsterdam" | ||
| }, | ||
| "warehouse_id": "[TEST_DEFAULT_WAREHOUSE_ID]" | ||
| } | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/alerts-with-file-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] alerts-v2 get-alert [ALERT_ID] | ||
| { | ||
| "display_name": "myalert", | ||
| "lifecycle_state": "ACTIVE", | ||
| "custom_summary": "My alert from file", | ||
| "evaluation": { | ||
| "comparison_operator": "EQUAL", | ||
| "notification": { | ||
| "notify_on_ok": false, | ||
| "retrigger_seconds": 1 | ||
| }, | ||
| "source": { | ||
| "aggregation": "MAX", | ||
| "display": "1", | ||
| "name": "1" | ||
| }, | ||
| "threshold": { | ||
| "value": { | ||
| "double_value": 2 | ||
| } | ||
| } | ||
| }, | ||
| "query_text": "select 2\n", | ||
| "schedule": { | ||
| "quartz_cron_schedule": "44 19 */1 * * ?", | ||
| "timezone_id": "Europe/Amsterdam" | ||
| }, | ||
| "warehouse_id": "[TEST_DEFAULT_WAREHOUSE_ID]" | ||
| } | ||
|
|
||
| === assert that the uploaded alert file is the same as the local one | ||
| >>> [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/alerts-with-file-[UNIQUE_NAME]/default/resources/myalert.dbalert.json | ||
|
|
||
| >>> diff exported_alert.dbalert.json alert.dbalert.json --strip-trailing-cr | ||
|
|
||
| >>> [CLI] alerts-v2 get-alert [ALERT_ID] | ||
| { | ||
| "display_name": "myalert", | ||
| "lifecycle_state": "ACTIVE" | ||
| } | ||
|
|
||
| >>> [CLI] bundle summary | ||
| Name: alerts-with-file-[UNIQUE_NAME] | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/alerts-with-file-[UNIQUE_NAME]/default | ||
| Resources: | ||
| Alerts: | ||
| myalert: | ||
| Name: myalert | ||
| URL: [DATABRICKS_URL]/sql/alerts-v2/[ALERT_ID] | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.alerts.myalert | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/alerts-with-file-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI bundle validate -o json | jq .resources.alerts.myalert | ||
|
|
||
| trace $CLI bundle deploy | ||
|
|
||
| alert_id=$($CLI bundle summary --output json | jq -r '.resources.alerts.myalert.id') | ||
|
|
||
| echo "$alert_id:ALERT_ID" >> ACC_REPLS | ||
|
|
||
| trace $CLI alerts-v2 get-alert $alert_id | jq '{display_name, lifecycle_state, custom_summary, evaluation, query_text, schedule, warehouse_id}' | ||
|
|
||
| alert_path=$($CLI bundle summary --output json | jq -r '.resources.alerts.myalert.parent_path')/myalert.dbalert.json | ||
| title "assert that the uploaded alert file is the same as the local one" | ||
| trace $CLI workspace export $alert_path > exported_alert.dbalert.json | ||
| trace diff exported_alert.dbalert.json alert.dbalert.json --strip-trailing-cr | ||
| rm exported_alert.dbalert.json | ||
|
|
||
| trace $CLI alerts-v2 get-alert $alert_id | jq '{display_name, lifecycle_state}' | ||
|
|
||
| trace $CLI bundle summary |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| Local = false | ||
| Cloud = true | ||
| RecordRequests = false | ||
| Ignore = [".databricks"] | ||
|
|
||
| # redact ?o=[NUMID]. Different clouds can have different URL serialization, like [DATABRICKS_URL]/sql/alerts-v2/[ALERT_ID] vs [DATABRICKS_URL]/sql/alerts-v2/[ALERT_ID]?o=[NUMID]. | ||
| [[Repls]] | ||
| Old = '\?o=\d+' | ||
| New = '' | ||
| Order = 0 | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_BUNDLE_ENGINE = ["terraform", "direct"] | ||
|
|
||
| [Env] | ||
| # MSYS2 automatically converts absolute paths like /Users/$username/$UNIQUE_NAME to | ||
| # C:/Program Files/Git/Users/$username/UNIQUE_NAME before passing it to the CLI | ||
| # Setting this environment variable prevents that conversion on windows. | ||
| MSYS_NO_PATHCONV = "1" |
33 changes: 33 additions & 0 deletions
33
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/alert.dbalert.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| { | ||
| "custom_summary": "My alert from file", | ||
| "evaluation": { | ||
| "comparison_operator": "EQUAL", | ||
| "notification": { | ||
| "notify_on_ok": false, | ||
| "retrigger_seconds": 1 | ||
| }, | ||
| "source": { | ||
| "aggregation": "MAX", | ||
| "display": "1", | ||
| "name": "1" | ||
| }, | ||
| "threshold": { | ||
| "value": { | ||
| "double_value": 2 | ||
| } | ||
| } | ||
| }, | ||
| "query_lines": [ | ||
| "select 2" | ||
| ], | ||
| "custom_description_lines": [ | ||
| "My alert from file", | ||
| "with multiple lines", | ||
| "and a third line" | ||
| ], | ||
| "schedule": { | ||
| "pause_status": "UNPAUSED", | ||
| "quartz_cron_schedule": "44 19 */1 * * ?", | ||
| "timezone_id": "Europe/Amsterdam" | ||
| } | ||
| } |
11 changes: 11 additions & 0 deletions
11
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/databricks.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| bundle: | ||
| name: alerts-not-allowed-field | ||
|
|
||
| resources: | ||
| alerts: | ||
| myalert: | ||
| warehouse_id: "0123456789012345" | ||
| display_name: "My alert" | ||
| file_path: ./alert.dbalert.json | ||
| # This field is not allowed when file_path is specified | ||
| query_text: "select 3" | ||
5 changes: 5 additions & 0 deletions
5
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/out.test.toml
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
19 changes: 19 additions & 0 deletions
19
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
|
|
||
| >>> trace [CLI] bundle validate | ||
|
|
||
| >>> [CLI] bundle validate | ||
| Error: field query_text is not allowed in the bundle configuration. | ||
| at resources.alerts.myalert.query_text | ||
| in databricks.yml:11:19 | ||
|
|
||
| When a .dbalert.json is specified, only the following fields are allowed in the bundle configuration: warehouse_id, display_name, file_path, permissions, lifecycle | ||
|
|
||
| Name: alerts-not-allowed-field | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/alerts-not-allowed-field/default | ||
|
|
||
| Found 1 error | ||
|
|
||
| Exit code: 1 |
1 change: 1 addition & 0 deletions
1
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| errcode trace trace $CLI bundle validate |
3 changes: 3 additions & 0 deletions
3
acceptance/bundle/resources/alerts/with_file_not_allowed_field_error/test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Local = true | ||
| RecordRequests = false | ||
| Ignore = [".databricks"] |
33 changes: 33 additions & 0 deletions
33
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/alert.dbalert.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| { | ||
| "custom_summary": "Alert for ${var.target}", | ||
| "evaluation": { | ||
| "comparison_operator": "EQUAL", | ||
| "notification": { | ||
| "notify_on_ok": false, | ||
| "retrigger_seconds": 1 | ||
| }, | ||
| "source": { | ||
| "aggregation": "MAX", | ||
| "display": "1", | ||
| "name": "1" | ||
| }, | ||
| "threshold": { | ||
| "value": { | ||
| "double_value": 2 | ||
| } | ||
| } | ||
| }, | ||
| "query_lines": [ | ||
| "select 2" | ||
| ], | ||
| "custom_description_lines": [ | ||
| "Alert for ${var.target}", | ||
| "with multiple lines", | ||
| "and a third line" | ||
| ], | ||
| "schedule": { | ||
| "pause_status": "UNPAUSED", | ||
| "quartz_cron_schedule": "44 19 */1 * * ?", | ||
| "timezone_id": "Europe/Amsterdam" | ||
| } | ||
| } |
9 changes: 9 additions & 0 deletions
9
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/databricks.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| bundle: | ||
| name: alerts-variable-error | ||
|
|
||
| resources: | ||
| alerts: | ||
| myalert: | ||
| warehouse_id: "0123456789012345" | ||
| display_name: "My alert" | ||
| file_path: ./alert.dbalert.json |
5 changes: 5 additions & 0 deletions
5
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/out.test.toml
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
17 changes: 17 additions & 0 deletions
17
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
|
|
||
| >>> [CLI] bundle validate | ||
| Error: .alert file alert.dbalert.json must not contain variable interpolations. | ||
| at resources.alerts.myalert.file_path | ||
| in databricks.yml:9:18 | ||
|
|
||
| Please inline the alert configuration in the bundle configuration to use variables | ||
|
|
||
| Name: alerts-variable-error | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/alerts-variable-error/default | ||
|
|
||
| Found 1 error | ||
|
|
||
| Exit code: 1 |
1 change: 1 addition & 0 deletions
1
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| errcode trace $CLI bundle validate |
3 changes: 3 additions & 0 deletions
3
acceptance/bundle/resources/alerts/with_file_variable_interpolation_error/test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Local = true | ||
| RecordRequests = false | ||
| Ignore = [".databricks"] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For future: We should allow override of the query text even when using a file_path, unless there will be another way to accept parameters like {catalog} and {schema} in the alert file.