Pytest crash with exit code 6

[jefer94]

With this script I see the exit code is 6

    exit_code = os.system(f'pytest {args} --disable-pytest-warnings')
    print(exit_code) # 6

This script prepare the environment and was used to run the tests pipenv run test

#!/bin/env python

from __future__ import absolute_import
import os
import sys
from .utils.environment import test_environment, reset_environment

if __name__ == '__main__':
    args = ''

    if len(sys.argv) > 1:
        sys.argv.pop(0)
        args = ' '.join(sys.argv)

    reset_environment()
    test_environment()
    exit_code = os.system(f'pytest {args} --disable-pytest-warnings')

    # python don't return 256
    if exit_code:
        sys.exit(1)

Output (i'm using Pytest 6.2.5 and Windows 11 with WSL 2 running Archlinux)

((apiv2) ) root@LACEY-MOSLEY ~/d/w/apiv2 (feature/fix-pdf-upload)# pipenv run test
Loading .env environment variables...
============================================= test session starts ==============================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
django: settings: breathecode.test_settings (from ini)
rootdir: /root/development/work/apiv2, configfile: pytest.ini
plugins: Faker-9.8.1, xdist-2.4.0, forked-1.3.0, cov-3.0.0, django-4.4.0
collected 1182 items                                                                                           

breathecode/activity/tests/urls/tests_academy_cohort_id.py ............                                  [  1%]
...
breathecode/certificate/tests/actions/tests_certificate_screenshot.py .                                  [ 54%]
breathecode/certificate/tests/actions/tests_generate_certificate.py ⏎
((apiv2) ) root@LACEY-MOSLEY ~/d/w/apiv2 (feature/fix-pdf-upload) [1]# 

I don't understand why the test runner scrash without show a error, or why the exit code is 6

pip list

((apiv2) ) root@LACEY-MOSLEY ~/d/w/apiv2 (feature/fix-pdf-upload) [1]# pip list
Package                           Version
--------------------------------- ---------------
activecampaign-python             1.0.3
amqp                              5.0.6
asgiref                           3.4.1
atomicwrites                      1.4.0
attrs                             21.2.0
backports.entry-points-selectable 1.1.0
beautifulsoup4                    4.10.0
billiard                          3.6.4.0
cachetools                        4.2.4
celery                            5.2.0
certifi                           2021.10.8
cffi                              1.15.0
cfgv                              3.3.1
charset-normalizer                2.0.7
click                             8.0.3
click-didyoumean                  0.3.0
click-plugins                     1.1.1
click-repl                        0.2.0
contextlib2                       21.6.0
coverage                          6.1.1
coveralls                         3.3.0
Deprecated                        1.2.13
distlib                           0.3.3
dj-database-url                   0.5.0
Django                            3.2.9
django-cors-headers               3.10.0
django-heroku                     0.0.0
django-phonenumber-field          6.0.0
django-redis                      5.0.0
django-sql-explorer               2.3
djangorestframework               3.11.2
djangorestframework-csv           2.1.1
docopt                            0.6.2
eventbrite                        3.3.5
execnet                           1.9.0
Faker                             9.8.1
filelock                          3.3.2
ghp-import                        2.0.2
google-api-core                   1.31.4
google-auth                       1.35.0
google-cloud-core                 1.7.2
google-cloud-datastore            1.15.3
google-cloud-ndb                  1.11.1
google-cloud-storage              1.42.3
google-crc32c                     1.3.0
google-resumable-media            2.1.0
googleapis-common-protos          1.53.0
grpcio                            1.41.1
gunicorn                          20.1.0
hiredis                           2.0.0
icalendar                         4.0.9
identify                          2.3.4
idna                              3.3
importlib-metadata                4.8.1
iniconfig                         1.1.1
Jinja2                            3.0.2
kombu                             5.2.1
lxml                              4.6.4
Markdown                          3.3.4
MarkupSafe                        2.0.1
mergedeep                         1.3.4
mixer                             7.2.0
mkdocs                            1.2.3
mkdocs-autorefs                   0.3.0
mkdocs-material                   7.3.6
mkdocs-material-extensions        1.0.3
mkdocstrings                      0.16.2
nodeenv                           1.6.0
packaging                         21.2
phonenumberslite                  8.12.37
Pillow                            8.4.0
pip                               21.3.1
pipenv                            2021.11.5.post0
platformdirs                      2.4.0
pluggy                            1.0.0
pre-commit                        2.15.0
prompt-toolkit                    3.0.22
protobuf                          3.19.1
psycopg2-binary                   2.9.2
py                                1.11.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycparser                         2.21
pyfcm                             1.5.4
PyGithub                          1.55
Pygments                          2.10.0
PyJWT                             2.3.0
pymdown-extensions                9.0
pymemcache                        3.5.0
PyNaCl                            1.4.0
pyparsing                         2.4.7
pytest                            6.2.5
pytest-cov                        3.0.0
pytest-django                     4.4.0
pytest-forked                     1.3.0
pytest-xdist                      2.4.0
python-dateutil                   2.8.2
python-slugify                    5.0.2
pytkdocs                          0.12.0
pytz                              2021.3
PyYAML                            6.0
pyyaml_env_tag                    0.1
redis                             3.5.3
requests                          2.26.0
rollbar                           0.16.2
rsa                               4.7.2
schema                            0.7.4
serpy                             0.3.1
setuptools                        58.5.3
six                               1.16.0
soupsieve                         2.3
sqlparse                          0.4.2
text-unidecode                    1.3
toml                              0.10.2
tomli                             1.2.2
twilio                            7.3.0
unicodecsv                        0.14.1
uritemplate                       4.1.1
urllib3                           1.26.7
vine                              5.0.0
virtualenv                        20.10.0
virtualenv-clone                  0.5.7
watchdog                          2.1.6
wcwidth                           0.2.5
wheel                             0.37.0
whitenoise                        5.3.0
wrapt                             1.13.3
XlsxWriter                        3.0.2
yapf                              0.31.0
zipp                              3.6.0

[bluetech]

I don't think pytest returns exit code 6 (pytest.ExitCode ends at 5), so it likely comes from someplace else.

[jefer94]

That's not true with this repo ( https://github.com/jefer94/apiv2/tree/feature/fix-pdf-upload )

./scripts/test.py

#!/bin/env python

from __future__ import absolute_import
import os
import sys
from .utils.environment import test_environment, reset_environment

if __name__ == '__main__':
    args = ''

    if len(sys.argv) > 1:
        sys.argv.pop(0)
        args = ' '.join(sys.argv)

    reset_environment()
    test_environment()
    exit_code = os.system(f'pytest {args} --disable-pytest-warnings')
+   print('')
+   print(exit_code)

    # python don't return 256
    if exit_code:
        sys.exit(1)

With Gitpod ( https://gitpod.io/#https://github.com/jefer94/apiv2/tree/feature/fix-pdf-upload ) returns 34304, is a weird value, ans with WSL 2 in Archlinux ( https://github.com/yuk7/ArchWSL ) it return 6

But if the error was cause by my code, Pytest should report the error, but this closes abruptly, in this case Pytest have a bad behavior yet

Windows 11 with WSL 2 Archlinux

Gitpod

The docs says ( https://docs.python.org/3/library/os.html#os.system )

On Unix, the return value is the exit status of the process encoded in the format specified for wait().

On Windows, the return value is that returned by the system shell after running command. The shell is given by the Windows environment variable COMSPEC: it is usually cmd.exe, which returns the exit status of the command run; on systems using a non-native shell, consult your shell documentation.

[jefer94]

Gitpod

WSL 2 With Archlinux

[RonnyPfannschmidt]

This looks like one of the tests triggers a hard os._exit, in which case pytest can't do anything

Run with -v to find the exact test,
Then Figure if just that test will break things already

[jefer94]

Pytest should mock os.exit() before load the tests files 🤔

[RonnyPfannschmidt]

Nope, code that triggers such conditions should do those very invasive changes exactly where needed

[jefer94]

Why? this just need trigger an exception, it is a behavior better that not report the result of tests and crash, also this message of error should improve the productivity of testers because is better work over a stacktrace that over a crash, also says mock the os.exit() is "invasive" is very much subjetive, is a bad criterial considering the behavior still breaking and the -v option just show which test is failing

[bluetech]

You probably mean sys.exit(). sys.exit() just raises SystemExit which is a BaseException. pytest already catches exceptions and displays them as test failures.

[RonnyPfannschmidt]

@jefer94 the _exit series of calls pretty much annihilate the current process outside of the control of python

It's not something a Norma test runner is expected to handle

With any normal exit exception you get a traceback

What we see from your output is pretty much that something that's not pytest is completely anhillating the pytest process exiting with a exit code that doesn't belong to pytest

[jefer94]

I'm inspecting the code but fragment of code that break Pytest is inside Django, how could I test this crash?, the --pdb don't work

[jefer94]

I found --boxed option allow see this stacktrace

[jefer94]

A exception was raise in Django Signal https://github.com/jefer94/apiv2/blob/906f0e101c2513df19de5e46e685b7759c36bedc/breathecode/certificate/views.py#L52

Because this funtion can throw a exception
https://github.com/jefer94/apiv2/blob/906f0e101c2513df19de5e46e685b7759c36bedc/breathecode/certificate/tasks.py#L24

And this was running probably in other thread and because this flow is not the same the view of Django, this exceptions ends the execution of tests

Pytest should have a section of docs for troubleshooting, also you should works with the python team to mock sys.exit() or improve the mocking system, is common I have to move the import was mocked with unittest.MagicMock inside the function or class to get the mocked version, instead of just mock it and use this mock without any change in the code like Jest, if the unittest lib worked fine, you just have to add 10 lines replacing the sys.exit() but with the current bad behavior of unittest.MagicMock this is harder

[Zac-HD]

This out of scope for pytest to fix - it's not us crashing; it's that we haven't (and won't - it's futile with e.g. kill -9) install sufficiently invasive hooks to keep running when someone else tries to tear down the process.