Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IOError: [Errno 0] Error raised when executing tox with --result-json option on Windows. #727

Closed
r2dan opened this issue Jan 5, 2018 · 6 comments

Comments

@r2dan
Copy link
Contributor

@r2dan r2dan commented Jan 5, 2018

Tox crashes on Windows when invoked with --result-json option. To reproduce the issue tests need to output a lot of text to stdout. Exception is raised only on Windows platform. The attached package reproduces the error on my machine (~30% reproduction rate).
Minimal package to reproduce the issue: tox_issue_result_log.tar.gz
Stack traceback: tox_result.log

The most convenient way to reproduce the error is to run tox in a loop:
for /l %x in (1, 1, 100) do tox --result-json result.json

Please note that pytest is invoked with --verbose flag to increase the amount of text written to stdout.

Platform: Windows 10/Windows 7 (reproduced on 3 machines)

C:\tox_issue_resultlog>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
apipkg (1.4)
attrs (17.4.0)
backports.shutil-get-terminal-size (1.0.0)
colorama (0.3.9)
decorator (4.0.11)
enum34 (1.1.6)
execnet (1.5.0)
fancycompleter (0.6.3)
funcsigs (1.0.2)
ipython (5.3.0)
ipython-genutils (0.2.0)
ordereddict (1.1)
pathlib2 (2.2.1)
pdbpp (0.8.3)
pickleshare (0.7.4)
pip (9.0.1)
pluggy (0.6.0)
prompt-toolkit (1.0.14)
py (1.5.2)
pycrypto (2.6.1)
Pygments (2.1.3)
pyreadline (2.1)
pytest (3.3.2)
pytest-forked (0.2)
pytest-xdist (1.21.0)
scandir (1.5)
setuptools (38.3.0)
simplegeneric (0.8.1)
six (1.11.0)
tox (2.9.1)
traitlets (4.3.2)
virtualenv (15.1.0)
wcwidth (0.1.7)
wheel (0.29.0)
win-unicode-console (0.5)
wmctrl (0.3)
@asottile
Copy link
Member

@asottile asottile commented Jan 8, 2018

Related, this function is broken pretty hard in python3:

$ tox --result-json out.json -vvvv -e py35
  removing /home/asottile/workspace/pyupgrade/.tox/log
using tox.ini: /home/asottile/workspace/pyupgrade/tox.ini
using tox-2.9.1 from /home/asottile/opt/venv/lib/python3.6/site-packages/tox/__init__.py
GLOB start: packaging 
GLOB sdist-make: /home/asottile/workspace/pyupgrade/setup.py
  removing /home/asottile/workspace/pyupgrade/.tox/dist
  /home/asottile/workspace/pyupgrade$ /home/asottile/opt/venv/bin/python3.6 /home/asottile/workspace/pyupgrade/setup.py sdist --formats=zip --dist-dir /home/asottile/workspace/pyupgrade/.tox/dist >/home/asottile/workspace/pyupgrade/.tox/log/tox-0.log
GLOB finish: packaging after 0.33 seconds
copying new sdistfile to '/home/asottile/.tox/distshare/pyupgrade-1.2.0.zip'
py35 start: getenv /home/asottile/workspace/pyupgrade/.tox/py35
py35 reusing: /home/asottile/workspace/pyupgrade/.tox/py35
py35 finish: getenv after 0.08 seconds
py35 start: installpkg /home/asottile/workspace/pyupgrade/.tox/dist/pyupgrade-1.2.0.zip
py35 inst-nodeps: /home/asottile/workspace/pyupgrade/.tox/dist/pyupgrade-1.2.0.zip
setting PATH=/home/asottile/workspace/pyupgrade/.tox/py35/bin:/home/asottile/workspace/watch-plz/venv/bin:/home/asottile/.gvm/bin:/home/asottile/bin:/home/asottile/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
  /home/asottile/workspace/pyupgrade$ /home/asottile/workspace/pyupgrade/.tox/py35/bin/pip install -U --no-deps /home/asottile/workspace/pyupgrade/.tox/dist/pyupgrade-1.2.0.zip >/home/asottile/workspace/pyupgrade/.tox/py35/log/py35-9.log
py35 finish: installpkg after 0.41 seconds
Traceback (most recent call last):
  File "/home/asottile/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/session.py", line 40, in main
    retcode = Session(config).runcommand()
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/session.py", line 392, in runcommand
    return self.subcommand_test()
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/session.py", line 567, in subcommand_test
    self.installpkg(venv, path)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/session.py", line 507, in installpkg
    venv.installpkg(path, action)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/venv.py", line 273, in installpkg
    self._install([sdistpath], extraopts=extraopts, action=action)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/venv.py", line 335, in _install
    action=action)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/venv.py", line 308, in run_install_command
    redirect=self.session.report.verbosity < 2)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/venv.py", line 414, in _pcall
    redirect=redirect, ignore_ret=ignore_ret)
  File "/home/asottile/opt/venv/lib/python3.6/site-packages/tox/session.py", line 163, in popen
    sys.stdout.write(data)
  File "/home/asottile/opt/venv/lib/python3.6/codecs.py", line 377, in write
    self.stream.write(data)
TypeError: write() argument must be str, not bytes

@fschulze
Copy link
Contributor

@fschulze fschulze commented Jan 8, 2018

@asottile that's tracked in #426. If you have a way to reproduce that, please report it there.

@fschulze
Copy link
Contributor

@fschulze fschulze commented Jan 8, 2018

@r2dan I wasn't able to reproduce this yet, but I have a suspicion what could cause this.

Could you give some more details?

  • From the for loop I assume you use cmd.exe not PowerShell.
  • Which Python version do you use?
  • Can you run chcp and report which code page is active (for me it's the default 850).

@fschulze
Copy link
Contributor

@fschulze fschulze commented Jan 8, 2018

@r2dan can you try with this patch?

diff --git a/tox/session.py b/tox/session.py
index 35cb4dc..0c8482d 100644
--- a/tox/session.py
+++ b/tox/session.py
@@ -127,7 +127,7 @@ class Action(object):
             fout.write("actionid: %s\nmsg: %s\ncmdargs: %r\n\n" % (self.id, self.msg, args))
             fout.flush()
             self.popen_outpath = outpath = py.path.local(fout.name)
-            fin = outpath.open()
+            fin = outpath.open('rb')
             fin.read()  # read the header, so it won't be written to stdout
             stdout = fout
         elif returnout:
@@ -155,7 +155,6 @@ class Action(object):
                     out = None
                     last_time = time.time()
                     while 1:
-                        fin_pos = fin.tell()
                         # we have to read one byte at a time, otherwise there
                         # might be no output for a long time with slow tests
                         data = fin.read(1)
@@ -173,7 +172,8 @@ class Action(object):
                             break
                         else:
                             time.sleep(0.1)
-                            fin.seek(fin_pos)
+                            # the seek updates internal read buffers
+                            fin.seek(0, 1)
                     fin.close()
                 else:
                     out, err = popen.communicate()

It's against the tox 2.9.1 tag, so it should apply to the release you use.

@r2dan
Copy link
Contributor Author

@r2dan r2dan commented Jan 8, 2018

@fschulze
I was using cmd.exe, Python 2.7.12.

C:\tox_issue_resultlog>chcp
Active code page: 437

I've tested the patch and it seems to fix the issue. Thanks!

fschulze added a commit to fschulze/tox that referenced this issue Jan 8, 2018
… with ``IOError: [Errno 0] Error`` on Windows, this was fixed by using a simpler method to update the read buffers.
fschulze added a commit to fschulze/tox that referenced this issue Jan 8, 2018
… with ``IOError: [Errno 0] Error`` on Windows, this was fixed by using a simpler method to update the read buffers.
fschulze added a commit to fschulze/tox that referenced this issue Jan 8, 2018
… with ``IOError: [Errno 0] Error`` on Windows, this was fixed by using a simpler method to update the read buffers.
fschulze added a commit to fschulze/tox that referenced this issue Jan 8, 2018
… with ``IOError: [Errno 0] Error`` on Windows, this was fixed by using a simpler method to update the read buffers.
gaborbernat added a commit that referenced this issue Jan 11, 2018
…`IOError: [Errno 0] Error`` on Windows, this was fixed by using a simpler method to update the read buffers. (#734)
@asottile
Copy link
Member

@asottile asottile commented Jul 1, 2018

via #734

@asottile asottile closed this Jul 1, 2018
@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants