Skip to content

Comments

Better cleanup LazyDictWithCache and ProviderManager#38730

Merged
potiuk merged 2 commits intoapache:mainfrom
Taragolis:providers-manager-better-cleanup
Apr 4, 2024
Merged

Better cleanup LazyDictWithCache and ProviderManager#38730
potiuk merged 2 commits intoapache:mainfrom
Taragolis:providers-manager-better-cleanup

Conversation

@Taragolis
Copy link
Contributor

This changes most attempt to fix cleanup ProviderManager, which I guess only use in tests.
Right now it might keep some values between the test as result it might not unwrap some values.

Sample for demonstrate it

class TestFoo:
    def test_http(self, cleanup_providers_manager):
        from airflow.providers_manager import ProvidersManager

        manager = ProvidersManager()
        hooks = manager.hooks
        info = hooks["http"]
        assert not callable(info)
        assert "http" in hooks._resolved

    def test_aws(self, cleanup_providers_manager):
        from airflow.providers_manager import ProvidersManager

        manager = ProvidersManager()
        hooks = manager.hooks
        info = hooks["aws"]
        assert not callable(info)
        assert "aws" in hooks._resolved
        assert "http" not in hooks._resolved

    def test_aws_again(self):
        from airflow.providers_manager import ProvidersManager

        manager = ProvidersManager()
        assert manager.hooks["aws"].package_name

and result

pytest tests/test_spam.py::TestFoo
================================================ test session starts ================================================
platform darwin -- Python 3.9.10, pytest-7.4.4, pluggy-1.4.0 -- /Users/taragolis/.pyenv/versions/3.9.10/envs/airflow-dev-env-39/bin/python
cachedir: .pytest_cache
rootdir: /Users/taragolis/Projects/common/airflow
configfile: pyproject.toml
plugins: cov-5.0.0, instafail-0.5.0, timeouts-1.2.1, anyio-4.3.0, custom-exit-code-0.3.0, rerunfailures-14.0, icdiff-0.9, xdist-3.5.0, time-machine-2.14.1, asyncio-0.23.6, mock-3.14.0, requests-mock-1.12.1
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 3 items                                                                                                   

tests/test_spam.py::TestFoo::test_http PASSED                                                                 [ 33%]
tests/test_spam.py::TestFoo::test_aws FAILED                                                                  [ 66%]
tests/test_spam.py::TestFoo::test_aws_again FAILED                                                            [100%]

===================================================== FAILURES ======================================================
_________________________________________________ TestFoo.test_aws __________________________________________________

self = <tests.test_spam.TestFoo object at 0x10519ef70>, cleanup_providers_manager = None

    def test_aws(self, cleanup_providers_manager):
        from airflow.providers_manager import ProvidersManager
    
        manager = ProvidersManager()
        hooks = manager.hooks
        info = hooks["aws"]
        assert not callable(info)
        assert "aws" in hooks._resolved
>       assert "http" not in hooks._resolved
E       AssertionError: assert 'http' not in {'aws', 'http'}
E        +  where {'aws', 'http'} = <airflow.providers_manager.LazyDictWithCache object at 0x10994dfd0>._resolved

tests/test_spam.py:19: AssertionError
______________________________________________ TestFoo.test_aws_again _______________________________________________

self = <tests.test_spam.TestFoo object at 0x1051bd310>

    def test_aws_again(self):
        from airflow.providers_manager import ProvidersManager
    
        manager = ProvidersManager()
>       assert manager.hooks["aws"].package_name
E       AttributeError: 'functools.partial' object has no attribute 'package_name'

tests/test_spam.py:25: AttributeError
============================================== short test summary info ==============================================
FAILED tests/test_spam.py::TestFoo::test_aws - AssertionError: assert 'http' not in {'aws', 'http'}
FAILED tests/test_spam.py::TestFoo::test_aws_again - AttributeError: 'functools.partial' object has no attribute 'package_name'
============================================ 2 failed, 1 passed in 1.79s ============================================

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis requested a review from potiuk April 3, 2024 23:11
@Taragolis Taragolis force-pushed the providers-manager-better-cleanup branch from 9029531 to a29fd57 Compare April 3, 2024 23:11
@potiuk potiuk merged commit 586a67d into apache:main Apr 4, 2024
@potiuk
Copy link
Member

potiuk commented Apr 4, 2024

Nice :)

@Taragolis Taragolis deleted the providers-manager-better-cleanup branch April 4, 2024 11:04
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants