Flaky test: test_pytest_exit_returncode fails due to ResourceWarning

[blueyed]

_________________________ test_pytest_exit_returncode __________________________
[gw0] linux -- Python 3.5.3 /home/travis/build/pytest-dev/pytest/.tox/pypy3-xdist/bin/python
testdir = <Testdir local('/tmp/pytest-of-travis/pytest-0/popen-gw0/test_pytest_exit_returncode0')>
    def test_pytest_exit_returncode(testdir):
        testdir.makepyfile(
            """
            import pytest
            def test_foo():
                pytest.exit("some exit msg", 99)
        """
        )
        result = testdir.runpytest()
        result.stdout.fnmatch_lines(["*! *Exit: some exit msg !*"])
>       assert result.stderr.lines == [""]
E       assert ['Exception i...name=16>', ''] == ['']
E         At index 0 diff: "Exception ignored in: <_io.FileIO name=16 mode='wb' closefd=True>" != ''
E         Left contains 2 more items, first extra item: 'ResourceWarning: unclosed file <_io.BufferedWriter name=16>'
E         Use -v to get the full diff
/home/travis/build/pytest-dev/pytest/testing/test_runner.py:584: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.5.3[pypy-6.0.0-final], pytest-4.4.1.dev66+g482a0ba, py-1.8.0, pluggy-0.9.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/travis/build/pytest-dev/pytest/.hypothesis/examples')
rootdir: /tmp/pytest-of-travis/pytest-0/popen-gw0/test_pytest_exit_returncode0
plugins: xdist-1.28.0, forked-1.0.2, hypothesis-4.15.0
collected 1 item
test_pytest_exit_returncode.py 
========================= no tests ran in 0.61 seconds =========================
!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: some exit msg !!!!!!!!!!!!!!!!!!!!!
----------------------------- Captured stderr call -----------------------------
Exception ignored in: <_io.FileIO name=16 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=16>
=============================== warnings summary ===============================
testing/test_warnings.py:701
  /home/travis/build/pytest-dev/pytest/testing/test_warnings.py:701: PytestExperimentalApiWarning: testdir.copy_example is an experimental api that may change over time
    testdir.copy_example("warnings/test_group_warnings_by_message.py")
-- Docs: https://docs.pytest.org/en/latest/warnings.html

https://travis-ci.org/pytest-dev/pytest/jobs/518502601#L5025
Python: pypy3.5-6.0
TOXENV=pypy3-xdist

[blueyed]

Again, same env: https://travis-ci.org/pytest-dev/pytest/jobs/519747824#L5032

[asottile]

looks like this has regressed

[nicoddemus]

We have implemented a workaround in #7405 and decided to leave this closed for now, we can reopen this if someone wants to spend more time working on this.