Dbt project must be in the current working directory

[dmateusp]

Related #597

The current implementation of the dbt templater assumes that the project root is the current working directory. It would be good to have a config to support:

• Having the project in another directory
• Having multiple dbt projects

for example:

dbt_project_roots = ./project_team_1,./project_team_2

This is actually quite painful right now with pre-commit, I tried on our project at Earnest which is in a sub-directory and had to write the following:

          entry: |
            python -c '
            import os
            import sys
            import subprocess as sp

            os.chdir("earnest_shared")
            file_names = [
              f.lstrip("earnest_shared/")
              for f in sys.argv[1:]
            ]
            return_code = sp.run(["sqlfluff", "lint"] + file_names).returncode

            if return_code:
              raise Exception("Some linting issues were found")'

cc. @NiallRees

[cnolanminich]

I can't tell if this is related, or if it's even a sqlfluff bug, but I'll post it here in case others will experience something similar.

We are getting ready to use the dbt templater for our CI. Our CI runs on an on-prem Jenkins server (that is itself running as a Docker container). Our (non-dbt) templater has been working 💯 👌 for months now.

I upgraded the the Docker image we use to use the 0.4.1 version. When I spin the container up, clone my dbt project, and run sqlfluff lint etl/ (our directory with dbt models), everything runs correctly (I also exported any environment variables to try for an exact match).

However, when the Jenkins server does the same thing, I get this error message:

sqlfluff lint etl
The path(s) ('etl',) could not be accessed. Check it/they exist(s).

As a few fun tests, I ran this within the Jenkins survey:

python -c import os;print(os.path.exists('etl'))
True

python -c import os;print(os.getcwd())
/var/jenkins_home/jobs/continuous-integration/jobs/chop-data-blocks/branches/feature-use-.8onc47.-0-4-0-alpha/workspace

python -c import os; path_walk = os.walk('etl'); print([dirname for dirname in path_walk])
[('etl', ['ADT', 'Access', 'Capacity', 'Department', 'Diagnosis', 'Encounter', 'Finance', 'Flowsheet', 'GI', 'IPC', 'Medication', 'Neonatology', 'Neuroscience', 'Note', 'Ops',

In other words, python seems to find the files ok, the dbt templater works when not on Jenkins, and everything lints as long as I'm not using the dbt templater. The only thing I can think of is that the paths are based on a mounted volume, and somehow that might be related to how either sqlfluff or dbt-via-sqlfluff is reading these files.

Again, I'm not convinced this is a sqlfluff issue (so feel free to ignore), but posting it here in case it nudges any of ya'll in an "aha" kind of a way for some of these dbt templater filepath / project directory issues.

Summary Configuration files

```docker # our own custom quay deployment FROM /analytics/python:3.8

WORKDIR /opt

COPY Pipfile* /opt/

RUN mkdir /opt/.dbt
COPY include/profiles.yml /opt/.dbt

RUN pipenv install --system --deploy --dev

pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
sqlfluff = '*'

[packages]
dbt-netezza = {git = "https://github.com/chop-analytics/analytics/dbt-netezza"}
dbt-core = '==0.18.1'
pyodbc = '*'

[requires]
python_version = "3.8"

 .sqlfluff

[sqlfluff]
templater = dbt
exclude_rules = L003,L013,L022,L027,L009,L031

[sqlfluff:templater:jinja:context]
target = 'dev'

[sqlfluff:indentation]
indented_joins = True

[sqlfluff:rules]
max_line_length = 115

[sqlfluff:rules:L010]
capitalisation_policy = lower

</details>

[pvonglehn]

I've made a first attempt at resolving this issue (Having the project in another directory - not yet having multiple project directories)

Would be happy for any feedback.

#898

[andyenkeboll-iex]

Would also love to see config support for dbt's --profile-dir argument. This might be easily added to @pvonglehn's #898?

[pvonglehn]

Would also love to see config support for dbt's --profile-dir argument. This might be easily added to @pvonglehn's #898?

@andyenkeboll-iex, yes PR #898 will implement support for --profile-dir. Just resolving a few issues in the PR.

[alanmcruickshank]

I think this issue is resolved now that #898 is merged 👍

[aik-liva]

Hi I have a question related to this PR.

My repo structure is:

.
├── .circleci
    └── config.yml
├── dbt_project_1
│   ├── models
│   ├── .sqlfluff
│   ├── .sqlfluffignore

my .sqlfluff config is:

[sqlfluff]
templater = dbt
dialect = postgres
rules =  ...

I would like to run sqlfluff on dbt_project_1 in circleci based on the .sqlfluff in dbt_project_1. Is it possible to do so?

I tried putting the .sqlfluff in the .circleci folder and running the command sqlfluff lint dbt_project_1/models in circleci config but i get the error message No dialect was specified which tells me that the .sqlfluff is not being detected.

I also tried sqlfluff using CLI method by running the command sqlfluff lint dbt_project_1/models --dialect postgres --templater dbt in circleci config, without having .sqlfluff in the same folder, and i get the following error: dbt_profiles_dir: /home/circleci/.dbt could not be accessed. Check it exists.