Debug logs are default - cannot be turned off

[cvlendistry]

Running on MWAA 2.2.2
I upgraded to 0.15.2 and airflow-redshift==1.2.1. Prior to the upgrade, logs appeared like this:

[2022-12-01, 11:00:09 UTC] {{logging_mixin.py:109}} INFO - 11:00:09 | Concurrency: 4 threads (target='prod')
[2022-12-01, 11:00:09 UTC] {{logging_mixin.py:109}} INFO - 11:00:09 |
[2022-12-01, 11:00:09 UTC] {{logging_mixin.py:109}} INFO - 11:00:09 | 1 of 15 START table model ***.................... [RUN]
[2022-12-01, 11:00:09 UTC] {{logging_mixin.py:109}} INFO - 11:00:09 | 2 of 15 START table model ****.......................... [RUN]
[2022-12-01, 11:00:09 UTC] {{logging_mixin.py:109}} INFO - 11:00:09 | 3 of 15 START table model ***................. [RUN]

After the upgrade,

[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.678976 [debug] [MainThread]: Partial parsing not enabled
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.691536 [debug] [MainThread]: Parsing macros/generate_schema_name.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.692644 [debug] [MainThread]: Parsing macros/adapters.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.710620 [debug] [MainThread]: Parsing macros/catalog.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.718226 [debug] [MainThread]: Parsing macros/relations.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.718708 [debug] [MainThread]: Parsing macros/adapters/apply_grants.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.719046 [debug] [MainThread]: Parsing macros/utils/datediff.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.719490 [debug] [MainThread]: Parsing macros/utils/listagg.sql
[2022-12-02, 05:48:58 UTC] {{functions.py:230}} DEBUG - �[0m05:48:58.721693 [debug] [MainThread]: Parsing macros/utils/split_part.sql

Not sure if this is due to dbt or airflow-dbt-python
Since the filename is functions.py (previously logging_mixin.py), I assume that this is due to the dbt upgrade. But when I run dbt from the command line, I do not see these debug logs, so I wanted to check.

I've tried

config:
    debug: False

in profiles.yml but this doesn't seem to help

[cvlendistry]

with dbt-redshift 1.2.1 and airflow-dbt-python 0.14.5 I get somewhat of the old behavior except that there are duplicate logs (one with nanosecond resolution). Same as #60

[deepakbansal08]

Facing same issue with MWAA 2.4.3, airflow-dbt-python 0.15.3 and dbt-core 1.3.1. Any possible ways to turn this off till the issue is fixed? I tried passing 'debug=False' in the DbtRunOperator but the debug logs are still getting generated.

[tomasfarias]

This is likely a duplicate of #60 and likely to be fixed by #93.

The reason I'm using 'likely' so much is that dbt has substantially changed they way they manage logging with every minor release, so there isn't a stable API for me to work with. This means that with every dbt version release we need to learn how to deal with dbt logging again...

That being said, while testing out v1.0.0 (#93) with dbt-core 1.4 and 1.3 I did not see any more debug log line issues. You can expect a beta release of v1.0.0 soon and hopefully you'll help me test this one out before the full release. Thank you!

[tomasfarias]

Beta releases of airflow-dbt-python v1.0.0 have been out for a couple of weeks. On my local tests, the debug parameter is now controlling whether full logs are shown in Airflow or not. Your experience may vary with different dbt versions, but at least 1.4 appears to behave properly.

I'll be closing this issue for the time being, but feel free to open a new one if the problem persists.