Pytest failing with Travis Cl python 3 environments

[abhiTronix]

Hello everyone, while performing Travis Cl jobs on my vidgear python library I discovered this weird behavior with the latest pytest (5.0.1).

Bug Description:

In this latest pytest, weirdly the Python 2.7 (https://travis-ci.com/abhiTronix/vidgear/jobs/213695559) environment is passing tests successfully whereas all the python3 environments(i.e Python-3.5, Python-3.6 and Python-3.7) failing with the error The command "pytest -s -vvv" exited with 1. even if all test passed without any whatsoever significant error or warning .

Travis Cl System Details

Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
pytest-5.0.1, py-1.7.0, pluggy-0.12.0

Here is my Travis.yml configuration file: https://github.com/abhiTronix/vidgear/blob/c86385748c216ecd5f056fbeff1c3afc330a64ec/.travis.yml

Temporary Workaround:

Downgrading to pytest-4.6.4 seems to works and all test are passing on both python 2 and 3 environments.

[abhiTronix]

Update:

Even with Appveyor Cl, pytest-5.0.1 is failing on python 3 environments: https://ci.appveyor.com/project/abhiTronix/vidgear/builds/25789872

[asottile]

My guess is that some destructor is exit(1)ing or segfaulting -- though I'm just guessing -- have you tried debugging it locally?

locally for me I get 24 failures so it's a bit difficult to pinpoint (when I exclude the failing tests it exits zero however)

[abhiTronix]

@asottile

My guess is that some destructor is exit(1)ing or segfaulting -- though I'm just guessing

But why we can see PASSED and XPASS at the end of each test with pytest-5.0.1? If it is segfaulting that it must not continue the test further or throw same error somehow with pytest-4.6.4 too. Isn't it?

have you tried debugging it locally?

Not yet but doing it right now.

locally for me I get 24 failures so it's a bit difficult to pinpoint (when I exclude the failing tests it exits zero however)

Did you perform tests with my vidgear library? If yes, then can please share the log of tests that were failing so that I can correct them and try again. Btw you need to follow these setups to be able to successfuly pass the test with my vidgear library.

[asottile]

My guess is that some destructor is exit(1)ing or segfaulting -- though I'm just guessing

But why we can see PASSED and XPASS at the end of each test with pytest-5.0.1? If it is segfaulting that it must not continue the test further or throw same error somehow with pytest-4.6.4 too. Isn't it?

again it was just a guess -- but if it's right the process teardown (and global destructors) happens after pytest reports status so it would show it happening there

locally for me I get 24 failures so it's a bit difficult to pinpoint (when I exclude the failing tests it exits zero however)

Did you perform tests with my vidgear library? If yes, then can please share the log of tests that were failing so that I can correct them and try again. Btw you need to follow these setups to be able to successfuly pass the test with my vidgear library.

that's probably why the tests are failing for me -- didn't really want to run a script which spewed files into my homedir :S

[abhiTronix]

again it was just a guess -- but if it's right the process teardown (and global destructors) happens after pytest reports status so it would show it happening there

can you please tell what were the breaking changes between pytest 4.6.4 and 5.0.1? I might be using the old format for my test.

that's probably why the tests are failing for me -- didn't really want to run a script which spewed files into my homedir :S

No In my bash script, those files are few just opensource video files(6 probably) and an FFmpeg static binary required for encoding/decoding video files. Nothing suspicious or harmful but surely large files(all data is ~1Gb), all data will be stored in $HOME/Downloads directory only.

[asottile]

here's the changelog

it's very surprising to have a script write to ~/Downloads -- especially for test fixture data (I'd be much more comfortable if it wrote to either . or /tmp where I can more easily see and control it)

[abhiTronix]

here's the changelog

Thanks. Since pytest is not throwing any error/warning related to test, the problem is pretty hard to debug.

it's very surprising to have a script write to ~/Downloads -- especially for test fixture data (I'd be much more comfortable if it wrote to either . or /tmp where I can more easily see and control it)

I thought it may provide a common directory for both Windows and Linux platform. But I'm looking forward to changing it to /tmp in the next update. Thanks for suggesting.

[abhiTronix]

@asottile
Hi, I've successfully found the buggy test_IO.py file and the pytest are failing because of these two tests:

from vidgear.gears import WriteGear
import sys
import numpy as np
import os
from numpy.testing import assert_equal
import pytest

def test_assertfailedwrite():
	"""
	IO Test - made to fail with Wrong Output file path
	"""
	np.random.seed(0)
	# generate random data for 10 frames
	random_data = np.random.random(size=(10, 1080, 1920, 3)) * 255
	input_data = random_data.astype(np.uint8)

	with pytest.raises(AssertionError):
		# wrong folder path does not exist
		writer = WriteGear("wrong_path/output.mp4")
		writer.write(input_data)
		writer.close()



def test_failedextension():
	"""
	IO Test - made to fail with filename with wrong extention
	"""
	np.random.seed(0)
	# generate random data for 10 frames
	random_data = np.random.random(size=(10, 1080, 1920, 3)) * 255
	input_data = random_data.astype(np.uint8)
	
	# 'garbage' extension does not exist
	with pytest.raises(ValueError):
		writer = WriteGear("garbage.garbage")
		writer.write(input_data)
writer.close()

Can you identify what's wrong with them?

Update: I tested these tests locally but none of them produced the error. But Travis Cli environment is throwing the same error with this file only: https://travis-ci.org/abhiTronix/vidgear/builds/555202535

[asottile]

I don't think that file is causing the error in CI -- that's just the last file to run.

You might be able to isolate the problem by running pytest specific/filename.py individually?

[abhiTronix]

You might be able to isolate the problem by running pytest specific/filename.py individually?

@asottile Done. Only this file is buggy, I separately ran the pytest on this test_IO.py file only but throwing the same error on Travis Cl and Appveyor. Both failing on python3 environments only.

Here's the local(ran on my ubuntu bionic machine) pytest output, both test are passing on python 3.6.8:

[asottile]

this appears to be related to faulthandler

Here's the traceback I can extract by putting in some code:

    finally:
        einfo = sys.exc_info()
        import traceback; print(''.join(traceback.format_exception(*einfo)))
        print('finally?')

$ PATH=$PWD/ffmpeg:$PATH pytest -svv; echo $?
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /tmp/vidgear/venv/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/vidgear
collected 2 items                                                              

vidgear/tests/test_IO.py::test_assertfailedwrite PASSED
vidgear/tests/test_IO.py::test_failedextension BrokenPipeError caught: Wrong Values passed to FFmpeg Pipe, Kindly Refer Docs!
PASSED

=========================== 2 passed in 4.32 seconds ===========================
Traceback (most recent call last):
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 75, in main
    ret = config.hook.pytest_cmdline_main(config=config)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/main.py", line 250, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/main.py", line 245, in wrap_session
    config._ensure_unconfigure()
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 657, in _ensure_unconfigure
    self.hook.pytest_unconfigure(config=self)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/faulthandler.py", line 51, in pytest_unconfigure
    faulthandler.enable(file=_get_stderr_fileno())
  File "/tmp/vidgear/venv/lib/python3.6/site-packages/_pytest/faulthandler.py", line 31, in _get_stderr_fileno
    return sys.stderr.fileno()
ValueError: I/O operation on closed file

finally?
1

disabling faulthandler makes it go away:

$ PATH=$PWD/ffmpeg:$PATH pytest -svv -p no:faulthandler; echo $?
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /tmp/vidgear/venv/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/vidgear
collected 2 items                                                              

vidgear/tests/test_IO.py::test_assertfailedwrite PASSED
vidgear/tests/test_IO.py::test_failedextension BrokenPipeError caught: Wrong Values passed to FFmpeg Pipe, Kindly Refer Docs!
PASSED

=========================== 2 passed in 4.30 seconds ===========================
returning 0
NoneType: None

finally?
0

This is all caused by this line: https://github.com/abhiTronix/vidgear/blob/8d1fd2afec1e42b5583c37d411a29a9f25e0e2fb/vidgear/gears/writegear.py#L230

this breaks stderr for the rest of the process (and hides the traceback, since tracebacks are written to stderr)

[abhiTronix]

@asottile Thanks a lot for helping out.

Indeed ValueError: I/O operation on closed file was the real error that was causing this issue.

This is all caused by this line: https://github.com/abhiTronix/vidgear/blob/8d1fd2afec1e42b5583c37d411a29a9f25e0e2fb/vidgear/gears/writegear.py#L230

this breaks stderr for the rest of the process (and hides the traceback, since tracebacks are written to stderr)

Exactly this line was added in hurry to suppress numerous errors caused by FFmpeg pipeline. Thanks for pinpointing this and I've replaced this line with the correct one and this bug is now gone.

All test are now passing successfully on Travis Cli:: https://travis-ci.org/abhiTronix/vidgear and Appveyor: https://ci.appveyor.com/project/abhiTronix/vidgear/builds/25804074. You saved my day! :)

[asottile]

no problem!