Follow zero internal warning in unit tests

[Taragolis]

Body

This is follow up task after #38504, we have quite a few tests which raise one of the internal deprecation warning:

• airflow.exceptions.AirflowProviderDeprecationWarning
• airflow.exceptions.RemovedInAirflow3Warning
• airflow.utils.context.AirflowContextDeprecationWarning

All current exclusions which should be resolved stored into the tests/deprecations_ignore.yml in format test node id: {relative path to the test}::{ClassName}::{test_case} or {relative path to the test}::{test_case} without parametrised part

How to resolve specific test suite

Remove /commented out specific line in exclusion list a have a look on error by run removed node id,

root@e68ed83041ce:/opt/airflow# pytest tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3
======================================================================= test session starts ========================================================================
platform linux -- Python 3.8.19, pytest-7.4.4, pluggy-1.4.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: custom-exit-code-0.3.0, instafail-0.5.0, rerunfailures-14.0, asyncio-0.23.6, cov-5.0.0, xdist-3.5.0, timeouts-1.2.1, anyio-4.3.0, icdiff-0.9, mock-3.14.0, requests-mock-1.11.0, time-machine-2.14.1
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item                                                                                                                                                   

tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 FAILED                                                              [100%]

============================================================================= FAILURES =============================================================================
___________________________________________________________ TestPythonVirtualenvDecorator.test_python_3 ____________________________________________________________

self = <tests.decorators.test_python_virtualenv.TestPythonVirtualenvDecorator object at 0xffff855dd760>
dag_maker = <tests.conftest.dag_maker.<locals>.DagFactory object at 0xffff8593da30>

    def test_python_3(self, dag_maker):
        @task.virtualenv(python_version=3, use_dill=False, requirements=["dill"])
        def f():
            import sys
    
            print(sys.version)
            try:
                {}.iteritems()
            except AttributeError:
                return
            raise Exception
    
        with dag_maker():
>           ret = f()

tests/decorators/test_python_virtualenv.py:171: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
airflow/decorators/base.py:372: in __call__
    op = self.operator_class(
airflow/models/baseoperator.py:483: in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
airflow/models/baseoperator.py:483: in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
airflow/decorators/python.py:52: in __init__
    super().__init__(
airflow/models/baseoperator.py:483: in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
airflow/decorators/base.py:257: in __init__
    super().__init__(task_id=task_id, **kwargs_to_upstream, **kwargs)
airflow/models/baseoperator.py:483: in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
airflow/operators/python.py:620: in __init__
    warnings.warn(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <airflow.utils.decorators._autostacklevel_warn object at 0xffff85ebb820>
message = 'Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.'
category = <class 'airflow.exceptions.RemovedInAirflow3Warning'>, stacklevel = 2, source = None

    def warn(self, message, category=None, stacklevel=1, source=None):
>       self.warnings.warn(message, category, stacklevel + 2, source)
E       airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.

airflow/utils/decorators.py:108: RemovedInAirflow3Warning
---------------------------------------------------------------------- Captured stdout setup -----------------------------------------------------------------------
========================= AIRFLOW ==========================
Home of the user: /root
Airflow home /root/airflow
Skipping initializing of the DB as it was initialized already.
You can re-initialize the database by adding --with-db-init flag when running tests.
[2024-03-31T10:06:44.945+0000] {dagbag.py:540} INFO - Filling up the DagBag from /dev/null
------------------------------------------------------------------------ Captured log setup ------------------------------------------------------------------------
INFO     airflow.models.dagbag.DagBag:dagbag.py:540 Filling up the DagBag from /dev/null
===================================================================== short test summary info ======================================================================
FAILED tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 - airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.

After that need to determine which kind of a problem here

Option 1: It might be test deprecation functional without explicit usage of pytest.warn. This case is pretty simple and required to add pytest.warn context manager wit appropriate matcher.

Option 2: Use deprecated functional in tests, time moves on and not all test updated. In this case need to rewrite test to avoid use deprecated part

Option 3: Use deprecated functional in tests utilities and fixtures, the same as option 2, however it might spread across multiple tests suits, replace/add functional to use not deprecated functional it this utilities/fixtures. This might required to use full test needed label in pr, If you not a Commiter or PMC please ask to add this label.

Option 4: Incorrect depreciation in code base. This one could be cause if one part of functional deprecated in code base, but the other still use it. There is no a universal solution for this situation, it might required to avoid use deprecated functional in codebase or even undeprecate.

Other cases: It might be other cases, feel free to share you experience with non listed above in this issue or into the Slack

Tip

Feel free to ask a suggestion/helps in slack channels #contributors or #new-contributors

Note

There main target of this task it resolve deprecation warnings, but not hide all of them under pytest.warn, use this context manager, as well as warnings.catch_warnings context manager only for test suites around deprecated functional

Warning

Please do not mixin changes in code base and only tests into the same PR. Better to split it into the separate PRs.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.