New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError in test collection when sys.stdin isn't an iterator #3314

Closed
jaraco opened this Issue Mar 16, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@jaraco
Contributor

jaraco commented Mar 16, 2018

In this build, I found that because of the construction of a csv.DictReader(sys.stdin) during import, the file can't be collected.

Looking a --pdb trace on it, I see:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
jaraco/financial/lifo.py:82: in <module>
    input: compose(csv.DictReader, open)=csv.DictReader(sys.stdin),
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py:87: in __init__
    self.reader = reader(f, dialect, *args, **kwds)
E   TypeError: argument 1 must be an iterator
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py(87)__init__()
-> self.reader = reader(f, dialect, *args, **kwds)
(Pdb) f
<_pytest.capture.DontReadFromInput object at 0x1035952e8>

Obviously, I could refactor the code to bypass the error, but since the code runs, I feel it should be supported for pytest collection.

Perhaps DontReadFromInput could be adapted to be an iterator to satisfy csv.DictReader.

The issue can be minimally replicated with

$ cat > test.py
import sys
import csv
csv.DictReader(sys.stdin)
$ rwt -q pytest -- -m pytest test.py
======================================================== test session starts =========================================================
platform darwin -- Python 3.7.0b2, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /Users/jaraco/clean, inifile:
collected 0 items / 1 errors

=============================================================== ERRORS ===============================================================
______________________________________________________ ERROR collecting test.py ______________________________________________________
test.py:3: in <module>
    csv.DictReader(sys.stdin)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py:87: in __init__
    self.reader = reader(f, dialect, *args, **kwds)
E   TypeError: argument 1 must be an iterator
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================== 1 error in 0.09 seconds =======================================================
@pytestbot

This comment has been minimized.

pytestbot commented Mar 16, 2018

GitMate.io thinks possibly related issues are #3251 (Warnings not captured during test collection), #2532 (TypeError on conftest.py during collection), #33 (TypeError when module under test or function under test is named 'test'), #1976 (please don't suppress tracebacks on ImportError during test collection), and #149 (Distributed testing silently ignores collection errors).

@pytestbot pytestbot added the type: bug label Mar 16, 2018

@jaraco

This comment has been minimized.

Contributor

jaraco commented Mar 16, 2018

I thought maybe adding __next__ to DontReadFromInput would make it an iterator:

$ git diff
diff --git a/_pytest/capture.py b/_pytest/capture.py
index 36658acc..8d7ba6e7 100644
--- a/_pytest/capture.py
+++ b/_pytest/capture.py
@@ -566,6 +566,7 @@ class DontReadFromInput(object):
     readline = read
     readlines = read
     __iter__ = read
+    __next__ = read

     def fileno(self):
         raise UnsupportedOperation("redirected stdin is pseudofile, "

but it didn't:

jaraco/financial/lifo.py:82: in <module>
    input: compose(csv.DictReader, open)=csv.DictReader(sys.stdin),
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py:87: in __init__
    self.reader = reader(f, dialect, *args, **kwds)
E   TypeError: argument 1 must be an iterator
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/csv.py(87)__init__()
-> self.reader = reader(f, dialect, *args, **kwds)
(Pdb) f
<_pytest.capture.DontReadFromInput object at 0x11013e390>
(Pdb) f.__next__
<bound method DontReadFromInput.read of <_pytest.capture.DontReadFromInput object at 0x11013e390>>
@jaraco

This comment has been minimized.

Contributor

jaraco commented Mar 16, 2018

Here's the code where csv.reader detects an iterator. It uses PyObject_GetIter, which according to the docs is equivalent to __iter__.

Aha! and f.__iter__() raises OSError. I think I know the fix.

@jaraco

This comment has been minimized.

Contributor

jaraco commented Mar 16, 2018

This patch fixes the failure:

$ git diff
diff --git a/_pytest/capture.py b/_pytest/capture.py
index 6e213944..6545e6fa 100644
--- a/_pytest/capture.py
+++ b/_pytest/capture.py
@@ -574,7 +574,10 @@ class DontReadFromInput(object):
         raise IOError("reading from stdin while output is captured")
     readline = read
     readlines = read
-    __iter__ = read
+    __next__ = read
+
+    def __iter__(self):
+        return self

     def fileno(self):
         raise UnsupportedOperation("redirected stdin is pseudofile, "

It does it by making DontReadFromInput an iterable and only raising the OSError when something attempts to consume the iterator.

jaraco added a commit that referenced this issue Mar 16, 2018

jaraco added a commit that referenced this issue Mar 16, 2018

jaraco added a commit that referenced this issue Mar 16, 2018

jaraco added a commit that referenced this issue Mar 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment