-
-
Notifications
You must be signed in to change notification settings - Fork 31.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
csv.writer.writerows masks exceptions from __iter__ #70595
Comments
When passing a class implementing the dunder __iter__ that raises to csv.writer.writerows it hides the original exception and raises a TypeError instead. In the raised TypeError the __context__ and __cause__ both are None. For example: >>> class X:
... def __iter__(self):
... raise RuntimeError("I'm hidden")
...
>>> x = X()
>>> list(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __iter__
RuntimeError: I'm hidden
>>> import csv
>>> csv.writer(open('foo.csv', 'w', newline='')).writerows(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: writerows() argument must be iterable
>>> try:
... csv.writer(open('foo.csv', 'w', newline='')).writerows(x)
... except TypeError as e:
... e_ = e
>>> e_.__context__ is None
True
>>> e_.__cause__ is None
True It would seem that for reasons unknown to me either PyObject_GetIter loses the exception context or the call to PyErr_SetString in csv_writerows hides it. |
After doing some reading on https://docs.python.org/dev/c-api/exceptions.html it seems that this is possibly "as designed" or such, since csv_writerows explicitly calls PyErr_SetString on receiving NULL from PyObject_GetIter. Still, it feels like this could either let the original exception fall through (since it has nothing in the way of handling it) or form the chain in PY3 for easier debugging of the real cause. To give this thing some context we ran in to this while passing SQLAlchemy Query objects to csv_writerows. The Query object is compiled during call to __iter__ and the current behaviour masks possible SQL errors etc. |
The TypeError is correct. You passed a non-iterable. I agree that adding information, *if possible*, when 'non-iterable' is determined by an exception being raised, would be nice. I don't know how easy this would be. |
I have 0 beef with it being the TypeError as long as the exception chain is kept intact, especially since PY3 makes it possible. With current behaviour heisenbugs would produce some serious hair pulling, until one figures out that it is actually the __iter__ raising an exception. With that in mind, take an object implementing the __iter__ dunder correctly 999,999 times out of a million. Is it not iterable? Unfortunately I lack the experience with CPython C API to do something about this. Tests on the other hand I suppose I could manage, if a consensus on the behaviour can be reached. |
As a side note PyObject_GetIter actually raises a TypeError already for non-iterables (classes without __iter__ etc.), so csv_writerows duplicates the effort at the moment. |
I'm starting to think my initial example code was too simplified and misled from the issue at hand. It very explicitly showed what happens when some class with __iter__ raises and is passed to csv_writerows. Even then: >>> from collections.abc import Iterable
>>> class X:
... def __iter__(self):
... raise RuntimeError
...
>>> x = X()
>>> issubclass(X, Iterable)
True
>>> isinstance(x, Iterable)
True The glossary entry for iterable has nothing to say about exceptions. It only requires that they are able to return their members "one at a time". In a moderately complicated class this might be true most of the time, but that's not interesting; that 1 time when it can't is. Now imagine you have a class with __iter__ using 1..N foreign libraries that all may blow up at the wrong phase of the moon (humor me). With the current implementation you get "TypeError: writerows() argument must be iterable", which goes out of its way to actually mislead you. Just letting the original exception through would be the obviously helpful thing to do. Which is what PyObject_GetIter does, before csv_writerows overwrites it. |
It is definitely a bug. It can mask exceptions out of the control of the programmer like MemoryError and KeyboardInterrupt. |
This bug occurred not only in writerows(), but also in writerow() and csv.reader(). See also more general bpo-40824. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: