Remove remaining deprecated classes and replace them with PEP562

[potiuk]

This is a follow-up after #26153 - removal of all remaining
deprecated classes and replace them with PEP-562 dynamic attribute
loading.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

This feels SO. MUCH. BETTER.

Compare:

with:

[potiuk]

I had to manually adjust the changes a bit - there was one case I have not automatically pick up (when class was extending from deprecated class) - I will take a close look at the contrib ones if there was no similar case.

[potiuk]

@eladkal @ashb @uranusjr @mik-laj -> there was a bt of trouble with the "DummyOperator" - the tests started to fail (and for a good reason). There was a custom logic for the inheritance check and "is_empty" but as I understand the reason behind the logic of the check, after this PEP check we do not need the custom logic any more.

The difference with PEP 562, we are not extending the EmptyOperator any more (to create a DummyOperator that can be imported via from airflow.operators.dummy import DummyOperator. With PEP 562, the operator imported via from airflow.operators.dummy import DummyOperator IS EmptyOperator, so the check for serialization and "is_empty" will just work. But I wold love if someone verifies my logic :).

[potiuk]

As a bonus point - the "dummy" import will raise DeprecationWarning immediately rather than when the DummyOperator is instantiated (so test_dummy.py can also be safely removed - it was testing if the warning was generated at instantiation time).

[potiuk]

I will also have to update the contrib with the case where we had "extending" classes. Stay tuned :)

[potiuk]

Ok. Fixed missing classes automatically.

[potiuk]

missing contrib classes added in #26179

[potiuk]

:D ?

[ashb]

I've done a quick-n-dirty checkof performance costs -- because importing airflow.hooks.subprocess (which isn't deprecated) and it seems to have no difference to performance outside of the variance (0.26-0.29s to import airflow.hooks.process in a new python process on my laptop.)

One downside (which I don't think has any real tangible cost) is that importing airflow.operators.bash now has 66 extra modules as a result of the code in airflow/operators/__init__.py. I think that's fine.

[ashb]

@eladkal @ashb @uranusjr @mik-laj -> there was a bt of trouble with the "DummyOperator" - the tests started to fail (and for a good reason). There was a custom logic for the inheritance check and "is_empty" but as I understand the reason behind the logic of the check, after this PEP check we do not need the custom logic any more.

The difference with PEP 562, we are not extending the EmptyOperator any more (to create a DummyOperator that can be imported via from airflow.operators.dummy import DummyOperator. With PEP 562, the operator imported via from airflow.operators.dummy import DummyOperator IS EmptyOperator, so the check for serialization and "is_empty" will just work. But I wold love if someone verifies my logic :).

Yes, your logic is sound

[ashb]

@potiuk Conflicting now -- can you resolve so we can merge this and get 2.4 beta out today?

[potiuk]

@potiuk Conflicting now -- can you resolve so we can merge this and get 2.4 beta out today?

rebased. Merging shortly.

[potiuk]

Merged preemptively - there is nothing there that would break after merge.

[potiuk]

One downside (which I don't think has any real tangible cost) is that importing airflow.operators.bash now has 66 extra modules as a result of the code in airflow/operators/__init__.py. I think that's fine.

Yeah. Those imports are really equivalent to parsing literally very few init,.py files, parsing the dict and creating 60 very simple objects in memory with one getattr function added to them. None of the actual referred classes are imported until they are used, so all the heavy-lifting is deferred. This is far less involved than many other "import" overhead, where importing one package might drag walking through a number of folders and files, and parsing multiple of them and instantiating multiple complex objects

It is literally as "lightweight" as it can be. I even thoguht about doing some kind of lazy-proxy style of module initialization, but I think those instantiated modules are even more light-weight than any lazy-proxy style objects - they are pretty much lazy-proxies themselves.

[potiuk]

I think what really helps here is that we have literally one __init__.py file to process per package and that everything (one dict) is inside of that single file - even the overhead on opening /closing the single file is negligible in this case.