Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions acceptance/acceptance_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -215,16 +215,6 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
t.Setenv("CLI", execPath)
repls.SetPath(execPath, "[CLI]")

pipelinesPath := filepath.Join(buildDir, "pipelines") + exeSuffix
if _, err := os.Stat(pipelinesPath); err == nil {
err := os.Remove(pipelinesPath)
require.NoError(t, err)
}
err = os.Symlink(execPath, pipelinesPath)
require.NoError(t, err)
t.Setenv("PIPELINES", pipelinesPath)
repls.SetPath(pipelinesPath, "[PIPELINES]")

paths := []string{
// Make helper scripts available
filepath.Join(cwd, "bin"),
Expand Down
1 change: 0 additions & 1 deletion acceptance/pipelines/deploy/auto-approve/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions acceptance/pipelines/deploy/auto-approve/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [ENTRYPOINT] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-auto-approve/default/files...
Deploying resources...
Updating deployment state...
Expand All @@ -10,7 +10,7 @@ View your pipeline foo here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]
>>> rm resources.yml

=== Try to redeploy without --auto-approve - should fail
>>> errcode [ENTRYPOINT] deploy
>>> errcode [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-auto-approve/default/files...

This action will result in the deletion or recreation of the following Lakeflow Spark Declarative Pipelines along with the
Expand All @@ -24,7 +24,7 @@ Error: the deployment requires destructive actions, but current console does not
Exit code: 1

=== Redeploy with --auto-approve - should succeed
>>> [ENTRYPOINT] deploy --auto-approve
>>> [CLI] pipelines deploy --auto-approve
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-auto-approve/default/files...

This action will result in the deletion or recreation of the following Lakeflow Spark Declarative Pipelines along with the
Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/deploy/auto-approve/script
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
trace $ENTRYPOINT deploy
trace $CLI pipelines deploy

title "Remove resources from configuration to test auto-approve"
trace rm resources.yml

title "Try to redeploy without --auto-approve - should fail"
trace errcode $ENTRYPOINT deploy
trace errcode $CLI pipelines deploy

title "Redeploy with --auto-approve - should succeed"
trace $ENTRYPOINT deploy --auto-approve
trace $CLI pipelines deploy --auto-approve
1 change: 0 additions & 1 deletion acceptance/pipelines/deploy/create-pipeline/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/create-pipeline/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [PIPELINES] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-create-pipeline/default/files...
Deploying resources...
Updating deployment state...
Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/create-pipeline/script
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
trace $ENTRYPOINT deploy
trace $CLI pipelines deploy
PIPELINE_ID=$($CLI bundle summary -o json | jq -r '.resources.pipelines.foo.id')
trace $CLI pipelines get "${PIPELINE_ID}" | jq "{spec}"
2 changes: 0 additions & 2 deletions acceptance/pipelines/deploy/create-pipeline/test.toml

This file was deleted.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions acceptance/pipelines/deploy/fail-on-active-runs/output.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@

>>> [ENTRYPOINT] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-fail-on-active-runs/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]

>>> [ENTRYPOINT] deploy --fail-on-active-runs
>>> [CLI] pipelines deploy --fail-on-active-runs
Error: pipeline [UUID] is running


Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/deploy/fail-on-active-runs/script
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
trace $ENTRYPOINT deploy
trace $CLI pipelines deploy

# We deploy the bundle again to check that the deploy is failing if the job is running
errcode trace $ENTRYPOINT deploy --fail-on-active-runs
errcode trace $CLI pipelines deploy --fail-on-active-runs
1 change: 0 additions & 1 deletion acceptance/pipelines/deploy/force-lock/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions acceptance/pipelines/deploy/force-lock/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
>>> [CLI] workspace import /Workspace/Users/[USERNAME]/.bundle/test-pipeline-force-lock/default/state/deploy.lock --format AUTO --file ./deploy.lock

=== test deployment without force-lock (should fail)
>>> errcode [ENTRYPOINT] deploy
>>> errcode [CLI] pipelines deploy
Error: Failed to acquire deployment lock: deploy lock acquired by user-with-lock@databricks.com at [TIMESTAMP] +0000 UTC. Use --force-lock to override
Error: deploy lock acquired by user-with-lock@databricks.com at [TIMESTAMP] +0000 UTC. Use --force-lock to override


Exit code: 1

=== test deployment with force-lock (should succeed)
>>> [ENTRYPOINT] deploy --force-lock
>>> [CLI] pipelines deploy --force-lock
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-force-lock/default/files...
Deploying resources...
Updating deployment state...
Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/deploy/force-lock/script
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title "upload lock file"
trace $CLI workspace import /Workspace/Users/$username/.bundle/test-pipeline-force-lock/default/state/deploy.lock --format AUTO --file ./deploy.lock

title "test deployment without force-lock (should fail)"
trace errcode $ENTRYPOINT deploy
trace errcode $CLI pipelines deploy

title "test deployment with force-lock (should succeed)"
trace $ENTRYPOINT deploy --force-lock
trace $CLI pipelines deploy --force-lock
1 change: 0 additions & 1 deletion acceptance/pipelines/deploy/oss-spark-error/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/oss-spark-error/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

=== Fail to deploy OSS Spark pipelines-formatted YAML file
>>> musterr [ENTRYPOINT] deploy
>>> musterr [CLI] pipelines deploy
Error: [TEST_TMP_DIR]/oss-pipeline.yml seems to be formatted for open-source Spark Declarative Pipelines.
Pipelines CLI currently only supports Lakeflow Spark Declarative Pipelines development.
To see an example of a supported pipelines template, create a new Pipelines CLI project with "pipelines init".
Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/oss-spark-error/script
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
title "Fail to deploy OSS Spark pipelines-formatted YAML file"
trace musterr $ENTRYPOINT deploy
trace musterr $CLI pipelines deploy

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [ENTRYPOINT] deploy
>>> [CLI] pipelines deploy
Warning: unknown field: unknown_property
at resources.pipelines.test-pipeline
in databricks.yml:8:7
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
trace $ENTRYPOINT deploy
trace $CLI pipelines deploy
# print newline to comply with whitespace linter
printf "<EOL>\n"
2 changes: 0 additions & 2 deletions acceptance/pipelines/deploy/test.toml

This file was deleted.

1 change: 0 additions & 1 deletion acceptance/pipelines/deploy/var-flag/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/var-flag/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

=== Test --var flag usage
>>> [PIPELINES] deploy --var=catalog=custom_catalog --var=schema=custom_schema --auto-approve
>>> [CLI] pipelines deploy --var=catalog=custom_catalog --var=schema=custom_schema --auto-approve
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-var-flag/default/files...
Deploying resources...
Updating deployment state...
Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/deploy/var-flag/script
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
title "Test --var flag usage"
trace $ENTRYPOINT deploy --var="catalog=custom_catalog" --var="schema=custom_schema" --auto-approve
trace $CLI pipelines deploy --var="catalog=custom_catalog" --var="schema=custom_schema" --auto-approve

title "Verify: Check that variables were substituted correctly"
PIPELINE_ID=$($CLI bundle summary -o json | jq -r '.resources.pipelines.foo.id')
Expand Down
2 changes: 0 additions & 2 deletions acceptance/pipelines/deploy/var-flag/test.toml

This file was deleted.

6 changes: 3 additions & 3 deletions acceptance/pipelines/destroy/auto-approve/output.txt
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@

=== Test auto-approve pipelines destroy
>>> [PIPELINES] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-destroy/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]

>>> errcode [PIPELINES] destroy
>>> errcode [CLI] pipelines destroy
Error: please specify --auto-approve since terminal does not support interactive prompts

Exit code: 1

>>> [PIPELINES] destroy --auto-approve
>>> [CLI] pipelines destroy --auto-approve
The following resources will be deleted:
delete resources.pipelines.my_pipeline

Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/destroy/auto-approve/script
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
title "Test auto-approve pipelines destroy"
trace $PIPELINES deploy
trace $CLI pipelines deploy
# Should fail without auto-approve
trace errcode $PIPELINES destroy
trace errcode $CLI pipelines destroy
# Should succeed with auto-approve
trace $PIPELINES destroy --auto-approve
trace $CLI pipelines destroy --auto-approve
4 changes: 2 additions & 2 deletions acceptance/pipelines/destroy/destroy-pipeline/output.txt
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@

=== Test basic pipelines destroy
>>> [PIPELINES] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-destroy/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]

>>> [PIPELINES] destroy --auto-approve
>>> [CLI] pipelines destroy --auto-approve
The following resources will be deleted:
delete resources.pipelines.my_pipeline

Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/destroy/destroy-pipeline/script
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
title "Test basic pipelines destroy"
trace $PIPELINES deploy
trace $CLI pipelines deploy
PIPELINE_ID=$($CLI bundle summary -o json | jq -r '.resources.pipelines.my_pipeline.id')
# --auto-approve is required because the test environment cannot provide interactive confirmation
trace $PIPELINES destroy --auto-approve
trace $CLI pipelines destroy --auto-approve

title "Assert pipeline is deleted"
# Should fail because the pipeline is deleted
Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/destroy/force-lock/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> errcode [PIPELINES] deploy
>>> errcode [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-force-lock/default/files...
Deploying resources...
Updating deployment state...
Expand All @@ -10,15 +10,15 @@ View your pipeline foo here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]
>>> [CLI] workspace import /Workspace/Users/[USERNAME]/.bundle/test-pipeline-force-lock/default/state/deploy.lock --format AUTO --file ./deploy.lock

=== test deployment without force-lock (should fail)
>>> errcode [PIPELINES] destroy --auto-approve
>>> errcode [CLI] pipelines destroy --auto-approve
Error: Failed to acquire deployment lock: deploy lock acquired by user-with-lock@databricks.com at [TIMESTAMP] +0000 UTC. Use --force-lock to override
Error: deploy lock acquired by user-with-lock@databricks.com at [TIMESTAMP] +0000 UTC. Use --force-lock to override


Exit code: 1

=== test deployment with force-lock (should succeed)
>>> [PIPELINES] destroy --force-lock --auto-approve
>>> [CLI] pipelines destroy --force-lock --auto-approve
The following resources will be deleted:
delete resources.pipelines.foo

Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/destroy/force-lock/script
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
export username=$($CLI current-user me | jq -r .userName)
export MSYS_NO_PATHCONV=1
trace errcode $PIPELINES deploy
trace errcode $CLI pipelines deploy

title "upload lock file"
trace $CLI workspace import /Workspace/Users/$username/.bundle/test-pipeline-force-lock/default/state/deploy.lock --format AUTO --file ./deploy.lock

title "test deployment without force-lock (should fail)"
trace errcode $PIPELINES destroy --auto-approve
trace errcode $CLI pipelines destroy --auto-approve

title "test deployment with force-lock (should succeed)"
trace $PIPELINES destroy --force-lock --auto-approve
trace $CLI pipelines destroy --force-lock --auto-approve

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions acceptance/pipelines/dry-run/dry-run-pipeline/output.txt
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@

>>> [ENTRYPOINT] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-run/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]

=== Dry running pipeline, should have validate_only set to true
>>> [ENTRYPOINT] dry-run
>>> [CLI] pipelines dry-run
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run '[DEPLOYCMD]' first to ensure they are included.
If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.

Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]

Expand All @@ -24,10 +24,10 @@ Update ID: [UUID]
}

=== Dry running pipeline with KEY, expect same output as without KEY
>>> [ENTRYPOINT] dry-run my_pipeline
>>> [CLI] pipelines dry-run my_pipeline
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run '[DEPLOYCMD]' first to ensure they are included.
If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.

Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]

Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/dry-run/dry-run-pipeline/script
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ print_requests() {
rm out.requests.txt
}

trace $ENTRYPOINT deploy
trace $CLI pipelines deploy
rm out.requests.txt
title "Dry running pipeline, should have validate_only set to true"
trace $ENTRYPOINT dry-run
trace $CLI pipelines dry-run
print_requests

title "Dry running pipeline with KEY, expect same output as without KEY"
trace $ENTRYPOINT dry-run my_pipeline
trace $CLI pipelines dry-run my_pipeline
print_requests
1 change: 0 additions & 1 deletion acceptance/pipelines/dry-run/no-wait/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions acceptance/pipelines/dry-run/no-wait/output.txt
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@

>>> [ENTRYPOINT] deploy
>>> [CLI] pipelines deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]

=== Dry running pipeline with --no-wait flag
>>> [ENTRYPOINT] dry-run --no-wait
>>> [CLI] pipelines dry-run --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run '[DEPLOYCMD]' first to ensure they are included.
If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.

Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]

Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/dry-run/no-wait/script
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
trace $ENTRYPOINT deploy
trace $CLI pipelines deploy
title "Dry running pipeline with --no-wait flag"
trace $ENTRYPOINT dry-run --no-wait
trace $CLI pipelines dry-run --no-wait
1 change: 0 additions & 1 deletion acceptance/pipelines/dry-run/restart/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading