Misleading "not iterable" Error Message when generator return a "simple" type, and a tuple is expected #76440
Comments
|
I'm new with Python and I've been blocked for day on a "TypeError: 'Fraction' object is not iterable" error message, while the problem turned out to be completely different. I don't even know to what programming case this message would have been the right interpretation, but in this situation, it was completely wrong and I believe it can cause much loss of time for inexperienced python programmer (at least, I believe it should document alternative causes) Here is a sample code to show how misleading it can be (It's a program to compute pi with fractions) from fractions import Fraction
def order(x):
r, old_r, n, old_n = 2, 1, 1, 0
while (x>=r):
r, old_r, n, old_n = r*r, r, 2*n, n
return order(x >> old_n) + old_n if old_n > 0 else 0
def terms(m, n, i):
return Fraction(4 * m, n**(2*i+1) * (2*i+1))
def terms_generator(exp_prec):
ws = [ [terms(parm[1], parm[2], 0), 0] + list(parm)
for parm in ((1, 44, 57),
(1, 7, 239),
(-1, 12, 682),
(1, 24, 12943))]
digits = 0
while digits<exp_prec:
curws = max(ws, key=lambda col: col[0])
digits = int(0.30103 *
(order(curws[0].denominator))
- order(curws[0].numerator))
yield curws[2] * curws[0] #, digits
curws[2] = -curws[2]
curws[1] += 1
curws[0] = terms(curws[3], curws[4], curws[1])
expected_precision = 100
pi = 0
for term, dgts in terms_generator(expected_precision):
pi += term
print("{} digits".format(dgts))
print("pi = 3.{}".format(int((pi-3)*10**expected_precision)))Obviously, the problem came from having forgotten one argument in the "yield" line, which has nothing to do with a problem of iterability of my type. |
Camion
mannequin
added
type-bug
|
The error message is correct, but I'm sorry it's confusing. Here's an equivalent error: a, b = 3 You'll get the error "TypeError: 'int' object is not iterable". That's because Python sees 2 items to the left of the assignment, so it needs to extract 2 items from the right side. To get 2 values from the right side, it iterates over the right side. In this case, 3 cannot be iterated over, thus the error message. Now consider: a, b = 3, 4 Again, to get 2 items from the right side, Python iterates over it. In this case the thing on the right side is a 2 element tuple. It's a little confusing that it's a tuple because it doesn't have parentheses, but the comma between 3 and 4 makes it a tuple. In this case, Python can iterate over the tuple, and the tuple has exactly 2 elements, so the assignment to a and b succeeds with a = 3 and b = 4. I hope that clears it up. |
|
Ok, but the other explanation is valid as well. That's why I suggest to modify the error message like this : TypeError: '[TYPE]' object is not iterable
|
|
I concur with Eric. The current exception is correct and allows to identify your mistake. Changing the type of the exception will break an existing code, and the message proposed by you is misleading. |
|
I'm not talking about changing the type of the exception, Serhiy. but only to make the text message more explicit by adding a lead to a secondary possible cause. I do not understand how this addition would be misleading - more than the case I presented ? (Did you check my example ?) I agree that this message might possibly make my mistake obvious to an senior _python_ programmer, but doesn't the constraint of being _experienced in python_, not go in contradiction with the mantra "Explicit is better than implicit" ? |
|
I agree with Camion that the error message is misleading, and not just for beginners. It threw me for a loop too, when I first read it. Serhiy is right, the exception type cannot and should not be changed, but we can change the error message. I'm re-opening the ticket as an enhancement to improve the message. Here's my suggestion: TypeError: cannot unpack object ('Fraction' is not iterable) There is no stability guarantees on error messages, so we can include it in 3.6 (and possibly older if desired). By the way Camion, as interesting as your program to calculate the digits of pi is, in general it is better to cut the code down to the simplest version that demonstrates the problem. Something like this would demonstrate it equally as well: def gen():
yield 1
for x, y in gen():
pass |
stevendaprano
added
type-feature
|
I wrote it like that on purpose, Steven : My goal was not to show the message itself which (I believe) was sufficiently described in the explanation, but to show how hard it might be to understand the mistake in regard with the error message, even here, with a real life but not outrageously complicated code. :-) |
|
By the way, I guess if the problem arises like that, it's because it might be hard to distinguish both situations at the interpreter level, but if it was possible, it would be the best solution to have a different error message (even if I understand we should be very careful with the idea of changing the exception type, even in this case). I mean : Eric shown an equivalent error, but both are only equivalent at the present discrimination level of the interpreter & language grammar. I understand that distinguishing both those semantically different situation might be opening the Pandora box and it might be a bad idea... or not. The point is that, if ever both those situations were in some way distinguishable, then... well... we certainly have no idea of how much code confronted with this specific situation AND treated it with exception handling, BUT, we might make the assumption that a frequent confrontation with this case would then already have been raised before, and consequently, it might be relevant, at least to keep it in mind for a future evolution (4.0) of the language. |
|
My point is that the current error message is correct and is not misleading. And this is a standard error message raised in many other cases. There is no a bug. *Maybe* it can be improved in some specific cases. Do you want to provide a patch Steven? Without concrete patch the following discussion doesn't make sense. We even don't know if such patch is possible. |
|
Serhiy, I think I got a better understanding of what is happening. It is well described by the following example :
Traceback (most recent call last):
File "<pyshell#40>", line 1, in <module>
a, b = 1,
ValueError: need more than 1 value to unpack
>>> a, b = 1
Traceback (most recent call last):
File "<pyshell#41>", line 1, in <module>
a, b = 1
TypeError: 'int' object is not iterableAgain, the message might be correct, but in this case, the combination between the message and the ambiguous syntax, makes it lack from explicitness. Understanding that, I suggest to simply add "(expected 'tuple')" at the end of the message. |
|
On Sun, Dec 10, 2017 at 09:15:16AM +0000, Serhiy Storchaka wrote:
With respect Serhiy, in this bug report you have TWO PEOPLE who have I was especially confused because I was expecting a ValueError too Of course, with a bit more thought I realised that the error isn't If you prefer "lacking sufficient information" over misleading, I won't
I agreed that it isn't a bug, and changed this to an enhancement. Better
Someone who knows C and the interpreter internals better than me will |
|
On Sun, Dec 10, 2017 at 10:00:27AM +0000, Camion wrote:
That is incorrect: a tuple is not expected. Any iterable (sequence, We could say TypeError: 'int' object is not iterable (expected iterable) but that's redundants, since the comment in the parentheses is implied I think the right thing to do here (if possible!) is to distinguish for x in 20: ...
=> TypeError: 'int' object is not iterablea, b = 20 or similar. I'm less concerned about the exact wording than the fact |
|
I think the current error message should stay as is. The interpreter is in no position to guess what kind of iterable may have been intended. |
|
My objections were about the original report. The title of this issue still implies there is a bug that is not true. You proposition Steve may make sense, but we need a concrete code for discussing the advantages and the drawbacks of this change. |
|
@serhiy : I asked you to explain by what mean you supported the affirmation that our feeling (that the current message can be misleading in the specific situation) is wrong, but you didn't give us any element to understand your position. Steven and I did it: I gave you a sample code showing how and why a situation could be misinterpreted in regard of the current message, and Steven explained that as an experience python programmer, he experienced the same misunderstanding (I'm also an experienced programmer: started in the eighties, I'm new in Python only). May be, if you gave us some sound argument element, we could also better understand the reason why you affirm that our feeling is wrong. Because in the current situation, you only declared your own feeling, but didn't give it any support, and all we can conclude is that you are a very experienced python programmer who can understand what was beyond the text of the message, but it seems in contradiction with the principle of making things clear. -------------------- @Raymond : I do not understand how you make the link between both your affirmations. The initial problem is related with the fact that in some situation, the error message lead to think that there is a problem with the loop/generator/iterator, where the problem is in reality with a simple type which cannot be unpacked or iterated. Let's assume that the interpreter could never make the difference. In what way, does it support the fact that the message shouldn't in any way, hint the programmer on the fact that the problem is not with the generator/iterator/loop, but with the returned type ? -------------------- @steven (& everybody), -------------------- About patching the sources in c, the message "object is not iterable" appears in four places in the version 3.6.3 : in ./Objects/object.c:1023, twice in ./Objects/typeobject.c (lines 6329 & 6344), and in ./Objects/abstract.c:3135. I have patched them to make them recognizable from one another, and it appear that all observed messages have been generated from abstract.c:3135 (in 3.6.3) and not from the other lines. This poses problem, because not even the regression test show a case where those other 3 parts of the code are used, and I haven't figured out (yet?) what condition makes them occur. |
I just grepped the docs, and the term 'unpackable' does not appear anywhere in them. I don't think this would be an improvement. As for the earlier suggestion of adding "expected tuple" does not work, because it is not a tuple being expected, it is an iterable, and saying "expected iterable" would be redundant with saying "is not iterable". What about "TypeError: expected iterable but found <type>"? That's parallel to some other type error messages we already generate. Would that be any clearer? (I'm not sure it would be, it says the same thing.) |
|
I shouldn't have searched the docs for 'unpackable', I should have searched for 'unpack'. Doing that reveals that the term 'unpack' is used for other concepts, whereas the term 'iterable' is precise. So I think it would unacceptably lower the precision of the error message to add that word. |
|
@camion, please adopt a less insistent tone. We've devoting time and thought to your concerns but are under no obligation to make a change just because you are demanding it. Part of learning any programming language is learning to reason from error message to root cause. Good error messages help, but they don't eliminate the underlying not always comfortable path of learning the world-view of the language and its mechanisms. That is partly why the devguide includes this passage, "Be careful accepting requests for documentation changes from the rare but vocal category of reader who is looking for vindication for one of their programming errors". A instructor for several thousand engineers, I've not encountered a single case where the current message has been problematic, nor have we had tracker issue arise over Python's 27 year history. And so far all the proposed wording changes make the message less accurate in other contexts where the message might arise. Based on that and on the comments by the other respondents, it would reasonable to close this issue. |
|
Camion, the problem is not in the error message. The problem is in the complex expression that produces an error. If you have a complex expression you always have a change to misidentify the source of error. In your case the expression contains two implicit iter() invocations. I have looked at the code. It is possible to make the error message in this case more specific. There are precedences of this when pass wrong type as var-positional or var-keyword arguments: >>> print(*1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: print() argument after * must be an iterable, not int
>>> print(**1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: print() argument after ** must be a mapping, not intIn the following cases the error message is general. It is possible to make it more specific. >>> a, b = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> [*1]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> {*1}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> {**1}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not a mappingBut PyObject_GetIter() is called in 88 places in CPython core and extensions. In all these cases we can make error messages more specific. But I have large doubts that it is worth to do. And I'm not sure even about the above examples. If somebody provide a patch we could discuss it (I'm not sure it will be accepted at end). |
|
@Raymond: I know that you have no obligation to make changes just because I'm demanding them and that's why I'm just trying to convince people of the validity of my observations, here. I agree with the fact that part of learning a language is learning to reason from error message to root cause but I cannot ignore that Python's Zen asks for explicitness, and Steven agreed with that this situation was tricky to interpret. It would be sly to suggest I'm be looking for vindication for one of my own programming errors. I'm experienced enough in trouble-shooting to have solved it by myself. It's just that I do not believe normal that this situation took me so much time and energy to solve from the error message **in regard with other similar situations**. -------------------- @david : What other contexts are you talking about ? Instead of looking in the documentation which also talks about e.g.:unpacking archive files, You should probably grep the source code for strings containing the word unpack and it will show you that the word unpack is used in contexts fully related with the error in my example. -------------------- I don't know about what would be the right error message, but since I experimented on in and Serhiy asked for it, I made a sample patch on version 3.6.3 which contains all the change which I believe, would be required. |
|
Well I submitted a patch, but I don't see it anywhere on this page.. Is this normal ? |
|
Woops, my mistake... I fumbled this one : the file was empty |
|
I think generically changing 'iterable' to 'iterable/unpackable' is wrong and would engender more confusion than at present. Most uses of iteration have nothing to do with multiple assignment target unpacking. Some minimal examples resulting in "TypeError: 'int' object is not iterable", where this change would be wrong, include iter(1)
sorted(1)
for _ in 1: passWhat might be reasonable is to wrap iter(source) in multiple assignment target code with (the C equivalent, if it exists, of) try-except and augment the message before re-raising. The addition could be something explicit like "and cannot be the source for multiple target assignment". --- for term, dgts in terms_generator(expected_precision):into two simpler lines that each do less, for value in terms_generator(expected_precision):
term, dgts = valueyou would have likely have spent much less time looking in the wrong place. Isolating possible exception sources is a very useful debugging tool. |
|
Jerry, I've been troubleshooting thing for 30 years and I'm quite experienced at it, and in the end I was perfectly able to manage this problem and solve this problem by myself. My point is not about my own difficulty to solve this problem, but about the fact that an information was lacking clarity in a language which claims that things must be told. You mention the fact that Steven suggested playing playing around with simplified code, but it's not what Steven suggested. Steven suggested to make the example simpler, and I replied him that this example was not chosen to describe the programming situation but illustrate the troubleshooting situation and provide a good way to show how a real situation might be problematic to analyse with this message - and Steven confirmed that he was also put out on a wrong track by this message. For sure, when you are an experienced programmer, you will develop a different way to understand this. My point is that it seems to go in contradiction with "Python's Zen". You suggest that I should have split the packing and the unpacking on two separate lines. Ok, but this is an easy suggestion to make _when you already know the cause of the problem_. Each language has it's programming culture and; Ok, I'm new here but I haven't seen any recommendation going in that direction. On the contrary all the sample code and course I have seen on that topic encourage young programmer not to use this practice - and I believe it would be simpler to find a way to adapt the error message to make things clear that such an error can also arise for unpacking reasons (whatever the way you word it), than to change the programming/training culture around python. I have been made multiple propositions to re-word this message, and Steven also made some. My last proposition was motivated by the idea of keeping the message short. Some other were longer. One might also write : "not iterable (this problem may sometimes arise from unpacking)" or something else. The point is that in many situation, the problem comes from a mismatching between the number of values packed and the number of destination positions for unpacking; and in this case, you have a ValueError message stating clearly what happens. Only in this case, when the fact that you had only one value and didn't see it, makes that you no longer have an iterable type - do you get a TypeError. And Ok, I agree with Serhiy that the type of exception is correct, but I believe that the whole situation is misleading and should be corrected. (I apologize for my long sentences : My mother tongue is french and it's not always easy for me to to make them short like you do in English XD ) |
|
PR 4903 makes the error message more specific for this case. >>> a, b = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot unpack int objectI don't know whether it is worth to do this change. |
|
The PR looks okay to me. I'm also not sure it's worth the change, though. |
|
Thanks, Serhiy, for the C implementation, and Eric for reviewing it. I would add 'non-iterable', to get "cannot unpack non-iterable int object", to tell people what is needed instead. I do think this worthwhile. |
Doesn't this imply that there are iterable int objects which can be unpacked? |
|
In English, 'adjective noun' does not necessarily imply the existence of 'not-adjective' nouns, and the adjective may serve as a reminder or reason. For instance, "This job is too dangerous for mortal humans!" In the current context, failure of 'iter(ob)', by itself, only tells us that the particular object ob is not iterable. "'X' object is not iterable", rather that "'X' objects are not iterable", is correct. (Ditto for other messages.) That said, I think "cannot unpack int object, because not iterable" is better. |
|
Is it worth to emit more specific (but possible uniform) error messages in other unpacking cases (see msg307999)? FYI if a class implements __iter__ which returns non-iterable, the following error is raised: >>> class C:
... def __iter__(self):
... return 1
...
>>> a, b = C()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: iter() returned non-iterator of type 'int' |
|
I retested and iter(ob) (1) apparently raises 'not iterable' only when it can find neither __iter__ nor __getitem__. It (2) raises 'non-iterator', as above, when it finds __iter__, calls its, and get a non-iterator. (3) Exceptions in __iter__ are passed through. (If it finds and wraps __getitem__, errors only appear during iteration.) The current patch for assignment unpacking catches case (1) with "v->ob_type->tp_iter == NULL && !PySequence_Check(v))". Removing this would catch case (2) and incidentally case (3) TypeErrors. Not catching user TypeErrors would require looking at the message. I intentionally am not proposing to remove the TypeError check. Presuming the existence of a PyErr_' function to get exception messages, we could simply prefix the existing message with a short 'Cannot unpack. ' The same could be done for the other cases in msg307999, but I don't know if they are subject to the same need. |
|
thank You Serhiy for your C implementation. About this question of the dilemma around the meaning of "non iterable int" why not simply put the "non iterable" between parenthesis to avoid making it too verbose ? "cannot unpack (non-iterable) int object" However, I wonder why nearly all the changes I've seen, seem to be about int ? This situation might arise with any non-iterable type (I got it first with Fraction). |
|
'int' is a standin for non-iterable and easy to type. |
|
Even in the code !?? |
|
Yes, in the code for the the unpack message test, the specific non-iterable is not relevant. These are not tests of the iter builtin. |
|
Ok then. Thank you :-) It also seemed strange that there were so many messages about int ;-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: