Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test becomes flaky by upgrading pytest from 7.4.4 to 8.1.0 -> fails normally but is successfull when run in verbose more #12071

Closed
4 tasks done
mkashwin opened this issue Mar 4, 2024 · 7 comments

Comments

@mkashwin
Copy link

mkashwin commented Mar 4, 2024

  • a detailed description of the bug or problem you are having
    In my github actions, I am seeing a strange behaviour of one of my unit tests failing when executed normally but when executed in debug mode or with -v ( verbose mode) the test passes. In the normal mode, it appears that my application cache / a object level dict attribute is being cleared, causing Key error later on. Details of the test case and the logs from the test runs on github runner are provided below. This was not the behaviour with pytest 7.4.4

  • output of pip list from the virtual environment you are using
    The poetry lock file with all dependencies is here

  • pytest and operating system versions
    Ubuntu 22.04.4 LTS ( as provided in the github runner)
    pytest version 8.1.0

  • minimal example if possible
    The link to the unit test case which is failing is
    test_spb2unspublisher

    Run1 : done without debug mode and fails

    • The error here KeyError: 'group1_/edge_node_1/MyDevice' makes no sense as the key was explicitly set in the actual code code
      Application Code with the logic to set the cache key

    Run3 : Done with debug mode and no further changes and passes

@The-Compiler
Copy link
Member

Can you try downgrading to 7.4.4 again and double-check it indeed passes there? This looks entirely unrelated to pytest...

@The-Compiler The-Compiler added the status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity label Mar 4, 2024
@mkashwin
Copy link
Author

mkashwin commented Mar 4, 2024

This is working fine on 7.4.4.

Link for where the test worked

https://github.com/mkashwin/unifiednamespace/actions/runs/8137480142/job/22235919114#step:4:1

@RonnyPfannschmidt
Copy link
Member

at first glance testpaths doesnt match the ini options from pyproject.toml - so something else is wrong as well

@RonnyPfannschmidt
Copy link
Member

nm, i missunderstood the nesting setup

@The-Compiler The-Compiler removed the status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity label Mar 4, 2024
@mkashwin
Copy link
Author

Issue also present in pytest 8.1.1 which was released last night

@Zac-HD
Copy link
Member

Zac-HD commented Mar 12, 2024

Can you provide a https://stackoverflow.com/help/minimal-reproducible-example to help us debug this? Ideally a single file which reproduces the issue.

@mkashwin
Copy link
Author

@Zac-HD Apologies for not giving that earlier, I though giving the link to the github project with direct reference to test case and code being tested would have made life easier.

Looks like 8.1.0 got yanked and now I am unable to reproduce this with 8.1.1.
I tried to create a minimal reproducible example but am unable to reproduce this with 8.1.1
Even when I try downgrading to 8.1.0 it is not reproducing.
Closing the issue for now, will reopen if this surfaces again along with the minimal reproducible example

Thanks for your efforts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants