Refactor `dbt ls` to run from a temporary directory #414

tatiana · 2023-07-28T12:00:16Z

As of Cosmos 1.0.0, LoadMode.DBT_LS runs dbt ls from within the original dbt project directory.

The dbt ls outputs files to the directory it's running from unless the environment variables DBT_LOG_PATH and DBT_TARGET_PATH are specified (as of dbt 1.6).

Depending on the deployment, the Airflow worker does not have write permissions to the dbt project directory. This can lead to an error message similar to the following:

20:43:06  Encountered an error:
[Errno 30] Read-only file system: '/usr/local/airflow/dags/dbt_grindr/logs/dbt.log'

This PR changes the behavior of dbt ls to try to make the dbt ls artifacts (logs and target directory) not be written to the original project directory.

In addition to the introduced test, this change was validated using airflow 2.6 and dbt 1.6, by following these steps:
(1) Delete folders logs and target from astronomer-cosmos/dev/dags/dbt/jaffle_shop
(2) Add a breakpoint after stdout, stderr = process.communicate() in dbt/graph.py
(3) Run a DAG that uses astronomer-cosmos/dev/dags/dbt/jaffle_shop, e.g.:

airflow dags test basic_cosmos_dag `date -Iseconds`

(4) When the breakpoint happens, check that no target or logs folder was created after running dbt ls in astronomer-cosmos/dev/dags/dbt/jaffle_shop

A limitation with the current approach is that, although dbt ls is not creating these directories in the given circumstances, if the user is using the local executor or an earlier version of dbt, the files will still be written to the project directory.

Closes: #411

netlify · 2023-07-28T12:00:21Z

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	`7bd4e30`
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/64d6136543a5b3000778ba9e

codecov · 2023-07-28T12:07:43Z

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.07% 🎉

Comparison is base (1233f9c) 91.43% compared to head (7bd4e30) 91.51%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #414      +/-   ##
==========================================
+ Coverage   91.43%   91.51%   +0.07%     
==========================================
  Files          50       50              
  Lines        1752     1768      +16     
==========================================
+ Hits         1602     1618      +16     
  Misses        150      150

Files Changed	Coverage Δ
cosmos/constants.py	`100.00% <100.00%> (ø)`
cosmos/dbt/graph.py	`100.00% <100.00%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tests/dbt/test_graph.py

cosmos/dbt/graph.py

As of Cosmos 1.0.0, `LoadMode.DBT_LS` ran `dbt ls` from within the original dbt project directory. The `dbt ls` outputs files to the directory it's running from unless the environment variables `DBT_LOG_PATH` and `DBT_TARGET_PATH` are specified. Depending on the deployment, the Airflow worker does not have write permissions to the dbt project directory. This PR changes the behavior of `dbt ls` to make a copy of the original project directory into a temporary directory and run the command `dbt ls` from there. Closes: #411

…a separate dir Unfortunately this does not work in dbt 1.5 or previous versions

tatiana · 2023-08-10T15:21:13Z

Thanks, @jlaneve , I think I addressed all the feedback!

.pre-commit-config.yaml

tatiana · 2023-08-10T16:12:46Z

A different approach we could adopt to solve this issue is not to pass project-dir to dbt-ls and to create symbolic links for the original project-dir files to the temp-dir. This way, the target & logs will be written to the temp dir.

Still, this change will not solve the limitation that ATM local operators run things locally. @jlaneve @harels is the scope of this ticket only dbt ls or all local operator executions?

cosmos/dbt/graph.py

@tatiana

Feature (pending documentation!) * Support dbt global flags (via dbt_cmd_global_flags in `operator_args` by @tatiana in #469 Enhancements * Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471 Bug fixes * Fix bug on select node add exclude selector subset ids logic by @jensenity in #463 * Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414 Others * Docs: Fix RenderConfig load argument by @jbandoro in #466 * Enable CI integration tests from external forks by @tatiana in #458 * Improve CI tests runtime by @tatiana in #457 * Change CI to run coverage after tests pass by @tatiana in #461 * Fix forks code revision in code coverage by @tatiana in #472 * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467" i

Since Cosmos 1.0, `load_method.DBT_LS` is the default dbt project parsing method, unless the user gives a manifest. Using the original dbt project path has been a source of issues when that path is Read-Only. This issue was faced when running commands that generate `{project-dir}/target/` and `{project-dir}/logs/`, which was solved as part of #414. This issue is particularly problematic if we want to run `dbt deps` from the original project directory since dbt 1.6 saves adaptors to `{project_dir}/dbt_packages` unless specified in the user's `dbt_project.yml`. To our knowledge, dbt currently does not allow users to define this directory via flags or environment variables, as discussed in #481. This change aims to solve these issues, by creating a temporary directory and creating symbolic links to the original directory. Finally, during the development of this task, it was observed that when running dbt ls in a project with `packages.yml`, dbt raises a 'Compilation Error'. Since dbt may raise other errors in stdout, this PR captures "Errors" more generically - making it more evident potential issues to the end-users.

@tatiana

**Features** * Support dbt global flags (via dbt_cmd_global_flags in operator_args) by @tatiana in #469 * Support parsing DAGs when there are no connections by @jlaneve in #489 **Enhancements** * Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471 * Consistent Airflow Dataset URIs, inlets and outlets with `Openlineage package <https://pypi.org/project/openlineage-integration-common/>`_ by @tatiana in #485. `Read more <https://astronomer.github.io/astronomer-cosmos/configuration/lineage.html>`_. * Refactor ``LoadMethod.DBT_LS`` to run from a temporary directory with symbolic links by @tatiana in #488 * Run ``dbt deps`` when using ``LoadMethod.DBT_LS`` by @DanMawdsleyBA in #481 * Update Cosmos log color to purple by @harels in #494 * Change operators to log ``dbt`` commands output as opposed to recording to XCom by @tatiana in #513 **Bug fixes** * Fix bug on select node add exclude selector subset ids logic by @jensenity in #463 * Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414 * Fix profile_config arg in DbtKubernetesBaseOperator by @david-mag in #505 * Fix SnowflakePrivateKeyPemProfileMapping private_key reference by @nacpacheco in #501 * Fix incorrect temporary directory creation in VirtualenvOperator init by @tatiana in #500 * Fix log propagation issue by @tatiana in #498 * Fix PostgresUserPasswordProfileMapping to retrieve port from connection by @jlneve in #511 **Others** * Docs: Fix RenderConfig load argument by @jbandoro in #466 * Enable CI integration tests from external forks by @tatiana in #458 * Improve CI tests runtime by @tatiana in #457 * Change CI to run coverage after tests pass by @tatiana in #461 * Fix forks code revision in code coverage by @tatiana in #472 * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467 * Drop support to Python 3.7 in the CI test matrix by @harels in #490 * Add Airflow 2.7 to the CI test matrix by @tatiana in #487 * Add MyPy type checks to CI since we exceeded pre-commit disk quota usage by @tatiana in #510

tatiana force-pushed the issue-411-dbt-ls-tmp-dir branch from 2353167 to 5821a9c Compare July 28, 2023 12:00

tatiana added the team/astro-devex label Jul 28, 2023

tatiana marked this pull request as ready for review July 28, 2023 15:58

tatiana requested review from a team as code owners July 28, 2023 15:58

jlaneve reviewed Jul 28, 2023

View reviewed changes

tests/dbt/test_graph.py Outdated Show resolved Hide resolved

jlaneve reviewed Jul 28, 2023

View reviewed changes

cosmos/dbt/graph.py Outdated Show resolved Hide resolved

jlaneve reviewed Jul 28, 2023

View reviewed changes

cosmos/dbt/graph.py Outdated Show resolved Hide resolved

tatiana force-pushed the issue-411-dbt-ls-tmp-dir branch from 29171f9 to e2ecb39 Compare August 7, 2023 09:51

tatiana mentioned this pull request Aug 7, 2023

Write logs to a temp dir when using dbt_ls parsing mode #411

Closed

tatiana added 5 commits August 10, 2023 15:19

Fix for Python==3.7

cd5a07f

Set test as integration

21f9b9c

(unstable) dbt 1.6 solution for dbt ls outputting logs and target to …

6d52ba4

…a separate dir Unfortunately this does not work in dbt 1.5 or previous versions

Change appraoch to run from tmp dir without copying the project

4360fd1

tatiana force-pushed the issue-411-dbt-ls-tmp-dir branch from e2ecb39 to 4360fd1 Compare August 10, 2023 15:11

tatiana requested a review from jlaneve August 10, 2023 15:20

tatiana commented Aug 10, 2023

View reviewed changes

.pre-commit-config.yaml Show resolved Hide resolved

Enforce logs to be written to tempdir

cdb633e

tatiana added 2 commits August 11, 2023 11:41

Improve test

731cceb

Remove comment

7bd4e30

harels approved these changes Aug 11, 2023

View reviewed changes

cosmos/dbt/graph.py Show resolved Hide resolved

tatiana merged commit 41bf623 into main Aug 14, 2023
41 checks passed

tatiana deleted the issue-411-dbt-ls-tmp-dir branch August 14, 2023 09:09

tatiana mentioned this pull request Aug 16, 2023

Release 1.1.0a1 #475

Merged

tatiana mentioned this pull request Aug 23, 2023

Refactor dbt ls to run from tempdir with symbolic links #488

Merged

tatiana mentioned this pull request Sep 6, 2023

Release 1.1.0 #515

Merged

tatiana mentioned this pull request Oct 12, 2023

Improve error handling when using astro-cli and running dbt deps to render DAG #593

Closed

dwreeves mentioned this pull request Feb 12, 2024

Support partial parsing #800

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor `dbt ls` to run from a temporary directory #414

Refactor `dbt ls` to run from a temporary directory #414

tatiana commented Jul 28, 2023 •

edited

netlify bot commented Jul 28, 2023 •

edited

codecov bot commented Jul 28, 2023 •

edited

tatiana commented Aug 10, 2023

tatiana commented Aug 10, 2023

Refactor dbt ls to run from a temporary directory #414

Refactor dbt ls to run from a temporary directory #414

Conversation

tatiana commented Jul 28, 2023 • edited

netlify bot commented Jul 28, 2023 • edited

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

codecov bot commented Jul 28, 2023 • edited

Codecov Report

tatiana commented Aug 10, 2023

tatiana commented Aug 10, 2023

Refactor `dbt ls` to run from a temporary directory #414

Refactor `dbt ls` to run from a temporary directory #414

tatiana commented Jul 28, 2023 •

edited

netlify bot commented Jul 28, 2023 •

edited

codecov bot commented Jul 28, 2023 •

edited