You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
dbt task in asset bundle deployment, errors if artifacts included and git_source missing, inaccurate location if artifacts missing and git_source missing
#1246
Open
NodeJSmith opened this issue
Feb 29, 2024
· 0 comments
The more I attempt to troubleshoot this the less sure I am regarding what is a bug and what is by design but confusing.
I originally had an issue because I added a dbt task to my pipeline and forgot to add the git source for the dbt task.
When I attempted to deploy the updated asset I get the error message: build failed <package_name>: error chdir <bundle_path>: no such file or directory, output .
While troubleshooting this I found that if I remove the artifacts section from my asset bundle the deployment will succeed, but the dbt task assumes that the project directory is the asset bundle deployment location, e.g. /Shared/.bundle/dbx_data_quality/dev/files. I assume that this location being used as the project directory for the dbt task is the reason for the error and failed deployment, but this seems like a bug still because the path actually did it exist already.
Deploying this way results in a task that has these arguments:
I solved the issue by adding the git_source section to my job in the asset bundle, which keeps the project directory from being set at all on the dbt task.
Please provide a minimal reproducible configuration for the issue
Steps to reproduce the behavior
To reproduce the Error: build failed dbx_data_quality, ... error you need an asset bundle that contains a python task and a dbt task, with an artifacts section included in the yaml, using a relative path. The job cannot have a git_source section.
I'm not sure. The example of a dbt task in the docs shows a git_source section, so it seems that is the expected way of using a dbt task. I think that likely we would want to require a git_source section or ensure that if we do not have one and we have a relative path to the python wheel artifact that the dbt task does not cause a deployment failure.
Actual Behavior
With the artifacts section the deployment fails with a confusing error message. With the git_source section the deployment succeeds. Without either the artifacts or git_source section the deployment succeeds with the artifact directory as the project directory for the dbt task.
OS and CLI version
OS: Ubuntu 22.04 on WSL2 via Windows 11
CLI Version: Databricks CLI v0.214.1
Is this a regression?
I tried this in 0.213.0 and it did not work in that version either.
Describe the issue
The more I attempt to troubleshoot this the less sure I am regarding what is a bug and what is by design but confusing.
I originally had an issue because I added a dbt task to my pipeline and forgot to add the git source for the dbt task.
When I attempted to deploy the updated asset I get the error message:
![image](https://private-user-images.githubusercontent.com/8505845/309086040-4e73968e-dde5-4591-b8fa-79f0836dd414.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTg4OTMzMTEsIm5iZiI6MTcxODg5MzAxMSwicGF0aCI6Ii84NTA1ODQ1LzMwOTA4NjA0MC00ZTczOTY4ZS1kZGU1LTQ1OTEtYjhmYS03OWYwODM2ZGQ0MTQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjBUMTQxNjUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YWFmZDM0ZGVlOTM4ZWQ5Y2FmNGYzYjYzNDMyYzA4MDVhZDAzMDJkMzFkNzI4MzZkYzQyNzg2ZTg1ZTU4NGUxZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.qtyNZfW5AUWTksDKGoJhCIZ-DjRdCiIVeLdTwf39EKc)
build failed <package_name>: error chdir <bundle_path>: no such file or directory, output
.While troubleshooting this I found that if I remove the
![image](https://private-user-images.githubusercontent.com/8505845/309085423-82d110d8-aa7a-407c-87e5-c63f35fce45d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTg4OTMzMTEsIm5iZiI6MTcxODg5MzAxMSwicGF0aCI6Ii84NTA1ODQ1LzMwOTA4NTQyMy04MmQxMTBkOC1hYTdhLTQwN2MtODdlNS1jNjNmMzVmY2U0NWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjBUMTQxNjUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTk2MzliN2Y2ZjM0YzUwMzdjOWJmOWVjZDA4NWZkY2ZmOTI0ZmZkY2VmYzI1MjdhZjRiMjk0YjQyNDFhMjZjZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.9g62FdhUKcVVJxzxIJzHb_APNLbKJL6jnoa2Gi8P8PE)
artifacts
section from my asset bundle the deployment will succeed, but the dbt task assumes that the project directory is the asset bundle deployment location, e.g./Shared/.bundle/dbx_data_quality/dev/files
. I assume that this location being used as the project directory for the dbt task is the reason for the error and failed deployment, but this seems like a bug still because the path actually did it exist already.Deploying this way results in a task that has these arguments:
I solved the issue by adding the
git_source
section to my job in the asset bundle, which keeps the project directory from being set at all on the dbt task.Configuration
Please provide a minimal reproducible configuration for the issue
Steps to reproduce the behavior
To reproduce the
Error: build failed dbx_data_quality, ...
error you need an asset bundle that contains a python task and a dbt task, with anartifacts
section included in the yaml, using a relative path. The job cannot have agit_source
section.Expected Behavior
I'm not sure. The example of a dbt task in the docs shows a
git_source
section, so it seems that is the expected way of using a dbt task. I think that likely we would want to require agit_source
section or ensure that if we do not have one and we have a relative path to the python wheel artifact that the dbt task does not cause a deployment failure.Actual Behavior
With the
artifacts
section the deployment fails with a confusing error message. With thegit_source
section the deployment succeeds. Without either theartifacts
orgit_source
section the deployment succeeds with the artifact directory as the project directory for the dbt task.OS and CLI version
OS: Ubuntu 22.04 on WSL2 via Windows 11
CLI Version: Databricks CLI v0.214.1
Is this a regression?
I tried this in 0.213.0 and it did not work in that version either.
Debug Logs
Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed
with_artifacts_section_and_git_source_section.txt
no_artifacts_section_no_git_source_section.txt
with_artifacts_section_no_git_source_section.txt
The text was updated successfully, but these errors were encountered: