Print warning if Variable.get () or any other DB method is executed during DAG parsing

[potiuk]

Description

It would be great to make a big, hairy warning when someone tries to use Variable.get() or Connection or any other method that hits Airflow's database during DAG parsing. This is a very bad practice, resulting in opening and closing DB connections and it is too easy to use and too difficult to spot that it is actually a problem.

Use case / motivation

Guide our users to write good DAGs.

[potiuk]

cc: @mik-laj @turbaszek @ashb @kaxil WDYT ?

[kaxil]

This will be a bit tricky now that Variable.get() can get variable from any Secret Backends including environment Variable too

[LucasMMota]

Maybe we could alert just in the DAG's UI after reading its definition code. This way we don't alert everyone that uses this method.

[potiuk]

This will be a bit tricky now that Variable.get() can get variable from any Secret Backends including environment Variable too

I think we can get it at the DB access level. Set an ENV variable when parsing and set some sqlalchemy hook to warn if ANY db call is made.

[kaxil]

We should also provide a way to disable the warning, since there might be users who know it but want to use it

[potiuk]

Yeah

[mik-laj]

I support this idea with all my heart. I will try to prepare the POC tomorrow to push the discussion further.

[kaxil]

Cool

[kaxil]

Please do the same for Connections

[ashb]

Please add a way to silence this warning too -- sometimes there are a few unavoidable cases where you still want to do this, and it would be good for people to say "I know what I'm doing, don't warn me every time"

(As Kaxil said. Read properly Ash.)

[mik-laj]

I created a PR: #11960

don't warn me every time

I added these warnings to a separate view because they are not critical to the proper work of the instance and at the same time there can be a lot of these errors. We don't have a one-to-one relationship here, but one file can have many problems.