Better explain "TypeError: 'tuple' object is not callable"

[o11c]

BPO	15248
Nosy	@gvanrossum, @rhettinger, @terryjreedy, @ezio-melotti, @merwok, @stevendaprano, @serhiy-storchaka, @o11c, @storymode7
PRs	bpo-15248: Emit a compiler warning when missed a comma before tuple or list. #11757 bpo-15248: Emit a compiler warning when missed a comma before tuple or list. #11757 Fix syntax warnings in tests introduced in bpo-15248. #11932 [3.7] Fix syntax warnings in tests introduced in bpo-15248. (GH-11932) #11935 Fix syntax warnings in tests introduced in bpo-35942. #11934
Files	compile-warnings.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-11.16:26:06.728>
created_at = <Date 2012-07-04.05:28:42.223>
labels = ['interpreter-core', 'easy', 'type-feature', '3.8']
title = 'Better explain "TypeError: \'tuple\' object is not callable"'
updated_at = <Date 2019-08-12.03:38:14.664>
user = 'https://github.com/o11c'

bugs.python.org fields:

activity = <Date 2019-08-12.03:38:14.664>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-11.16:26:06.728>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2012-07-04.05:28:42.223>
creator = 'o11c'
dependencies = []
files = ['47496']
hgrepos = []
issue_num = 15248
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['164637', '164711', '164716', '164741', '164774', '314197', '314218', '314220', '334306', '334324', '335668', '335934', '349375', '349380', '349395', '349432']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'rhettinger', 'terry.reedy', 'ezio.melotti', 'eric.araujo', 'steven.daprano', 'docs@python', 'serhiy.storchaka', 'o11c', 'storymode7']
pr_nums = ['11757', '11757', '11932', '11935', '11934']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue15248'
versions = ['Python 3.8']

[o11c]

I frequently construct lists of tuples, such as:
[
(1, 2, 3) # oops, missing comma!
(4, 5, 6)
]

It would be nice if the error message gave a hint on what was *actually* wrong.

Although I always use homogeneous containers, the type that's not callable could be something other than 'tuple'.

You could possibly cut down on false positives (at the risk of false negatives) by checking that the not-callable object is newly constructed.

A better way to cut down on false positives would be to check that a list, tuple, or set is being constructed from a literal, but this might be more complex.

[stevendaprano]

I think this suggested enhancement is unreasonable and of limited usefulness, and even if it were useful, too specialised to bother with.

The obvious message is badly misleading. When I read this:

TypeError: 'tuple' object is not callable, a comma may be missing

I look inside the tuple for a missing comma. But that's not the problem, and the error message sends me on a wild goose chase wondering how on earth a missing comma causes Python to try calling my tuple. I've been programming in Python for 15+ years and it mislead me -- what do you think it will do to beginners?

The problem is that the tuple is inside a list, and the LIST is missing a comma.

What about this example?

result = function(
    "the error has nothing to do with tuples"  # oops missed a comma
    (2, 3, 4),
    None
    )

Should the error message say something like:

TypeError: 'str' object is not callable, perhaps it is embedded in a list, tuple,
function call or some other place you need a comma, and you forgot one
or more commas?

I don't think so. Besides, most of the time when you get this TypeError, it will be because you genuinely tried to call what you thought was a function but wasn't, and the hint is pointless.

You tried to call a tuple as if it were a function. The reason for that is that you left out a comma in a list, but there are many other reasons that could happen, and "I frequently forget to add commas to lists" is a far too-specialised failure mode to single it out in the error message.

It is unreasonable to expect Python to debug your code for you. It tells you what you did wrong -- you called a tuple as a function -- and it is up to you to determine why.

[o11c]

This kind of "debug your code" is the kind of thing I've gotten used to from the Clang C/C++ compiler. Granted, compiled languages have an advantage here, but enough residual information remains for the interpreter at runtime.

And I am in no way suggesting that *every* attempt to call a non-function have the extra information.

For the cases where the error message is given, something like:
TypeError: 'tuple' object is not callable (missing preceding comma?)

The case of a homogenous container is the most important case.

I've offered two different ways to figure out whether it's a typo or an attempt to call an object that you honestly think is callable:

1. Is the called object a newly-constructed (refcount=1) tuple literal? (Also works for list, set, and dictionary literals; probably wouldn't work for string literals due to interning)
2. Does the false call occur within a container literal or function call?

I'm not intimately familiar with python bytecode or interpreter, but I'm sure anyone who is could extract this information.

[ezio-melotti]

A FAQ entry could be added to explain this error, like we already do for the UnboundLocalError: http://docs.python.org/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value

[terryjreedy]

Stephen is right; this sort of guess-the-bug heuristic is out of scope for the CPython interpreter. I believe you grossly under estimate the difficulty of such a this. Consider that idea rejected.

What *is* disconcerting is the exact form of the error message for this particular code pattern:

Traceback (most recent call last):
  File "<pyshell#0>", line 3, in <module>
    (4, 5, 6)
TypeError: 'tuple' object is not callable

Uh... There was no attempt to call the tuple (4,5,6), which was misinterpreted as a () call operator with 3 args. What might be a reasonable request is to print a size-limited representation of the object that is not callable but was attempted to be called. Truncated representations would be generally useful for traceback messages. I believe we already have them for unittest error messages.

A useful document for someone (or some people) to write would be 'How to interpret exception messages'. That could start with a catalog of messages and possible causes. 'type' would be used to stand for any specific type.

The section on TypeErrors would have an heading
'type' object is not callable
followed by an explanation
There are two general reasons for this message. The first is that you intentionally write "expression(args)" with the intent that expression evaluate to a callable, but it does not. The second is that you accidentally omit a comma in a sequence and write "expression (tuple_members)". Note that the error is triggered by the "(args)" call operator and its line is the one printed. So if "expression" is on the preceeding line, it will not appear in the traceback. This can happen with either type of error.

In the meanwhile, the above is a start for a faq entry.

[serhiy-storchaka]

It is possible to make the compiler producing warnings for such kind of errors. The following sample patch does this. I'm not sure it is worth to do.

[terryjreedy]

Reviewing Serhiy's patch, I changed my mind from 6 years ago (msg164774). I may have over-estimated the difficulty; in any case the code patch is written. (Tests are still needed.) And the code is rather straightforward and readable.

Beyond that, I see the patch as detecting a SyntaxError during compilation, as explained below, with the hint being frosting on top of that. (It is not an 'Error' as the grammar is now, but it is an error as we might like it to be, hence Serhiy used SyntaxWarning instead.)

In addition, we now have the 'print' precedent of 'print'.
  >>> print 1
  SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1)?
I don't remember when this was added, but I don't see any mention above.

And it seems from other discussions, like the recent python-ideas thread about possibly deprecating 'string' 'joining', that several people have much more of a problem with missing commas than some of us do.

If it were possible, we would like the 'call' production to be
call ::= callable "(" [argument_list [","] | comprehension] ")"
but 'callable' is not syntactically definable. What is syntactically definable and what Serhiy's patch implements is
possible_callable ::= primary - <obvious exclusions>
call ::= possible_callable ...

As it happens, our grammar metasyntax does not (I believe) include set difference, and I suspect that implementing 'possible_callable' by set addition would result in something that is not LL(1) and possibly ambiguous. Even if 'possible_callable' could be defined within the LL(1) constraint, I suspect it would be messy, and special-casing would still be needed for the special-case message.

Ditto for 'possibly_subscriptable'.

I have 3 questions:
1. Why 'chech' instead of 'check'?
2. Will chech_index catch "[1,2] [3,4]"? (I am guessing that is the intent.)
3. Does Syntax Warning stop compilation, or at least execution, as at a >>> prompt?

Aside from that, I would be inclined to mark this for 'Interpreter Core' and seriously consider it.

[serhiy-storchaka]

Actually the precedence was a warning for an assert like:

    assert(x % 2 == 0, "x is odd")

Currently it is the only syntax warning produced by the compiler.

1. Why 'chech' instead of 'check'?

Just a typo replicated with a copy-paste.

2. Will chech_index catch "[1,2] [3,4]"? (I am guessing that is the intent.)

Yes, it is. It could be extended to catch also "['foo','bar'] ['baz']".

3. Does Syntax Warning stop compilation, or at least execution, as at a >>> prompt?

No, it is just a warning unless you run Python with -Werror.

This patch was inspired by usability improvements in GCC 8 [1]. I haven't created a pull request because I have doubts about that this should be in the compiler rather of a third-party linter. But if several other core developers will support this idea I'll continue working in this direction.

[1] https://developers.redhat.com/blog/2018/03/15/gcc-8-usability-improvements/

[storymode7]

Hey, if nobody is working on this can I go for it?

[terryjreedy]

I am not sure what you could do at the moment. We have not yet settled whether this is a documentation or interpreter core issue.

I posted "Add more SyntaxWarnings?" to pydev to get more comments on Serhiy's patch, especially for other core developers. Depending on what people think, that *might* result in someone converting the diff (with the typo corrected) into a PR.

I already posted here a prototype doc entry for TypeError messages. I decided that I should take time out from IDLE to post my idea for a separate error message doc. I will include a fleshed-out TypeError entry (with code example) as an example entry.

[serhiy-storchaka]

New changeset 62e4481 by Serhiy Storchaka in branch 'master':
bpo-15248: Emit a compiler warning when missed a comma before tuple or list. (GH-11757)
62e4481

[serhiy-storchaka]

New changeset 8e79e6e by Serhiy Storchaka in branch 'master':
Fix syntax warnings in tests introduced in bpo-15248. (GH-11932)
8e79e6e

[rhettinger]

This isn't interacting well with IDLE.

====================================================
--- Works fine at the regular interactive prompt ---

$ python3.8
Python 3.8.0b3 (v3.8.0b3:4336222407, Jul 29 2019, 09:46:03)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> data = [
...          (1, 2, 3) # oops, missing comma!
...          (4, 5, 6)
...      ]
<stdin>:2: SyntaxWarning: 'tuple' object is not callable; perhaps you missed a comma?
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>

====================================================
--- Behaves differently in the IDLE Shell ---

Python 3.8.0b3 (v3.8.0b3:4336222407, Jul 29 2019, 09:46:03) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license()" for more information.
>>> data = [
         (1, 2, 3) # oops, missing comma!
         (4, 5, 6)
     ]
SyntaxError: 'tuple' object is not callable; perhaps you missed a comma?

[serhiy-storchaka]

I think it would be better to open a separate issue for IDLE.

[serhiy-storchaka]

This is not related to this particular warning, but to how IDLE handles SyntaxWarning. See bpo-34857.

[terryjreedy]

IDLE's Shell has treated SyntaxWarnings as errors since 2001. I don't know if any other IDE's do the same. Raymond's example did not concern me, as I think not calling known non-callables is preferable.

But not executing literal-with-is code, such as "a = 1; a is 1" and "0 is ''" is wrong. See msg349431 of bpo-34857 for more discussion of these cases.