Incorrect evaluation order of function arguments with *args

[JoshuaLandau]

BPO	23316
Nosy	@gvanrossum, @terryjreedy, @bitdancer, @serhiy-storchaka, @NeilGirdhar, @gvanrossum, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-01-25.15:46:56.223>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'Incorrect evaluation order of function arguments with *args'
updated_at = <Date 2021-06-22.22:13:28.002>
user = 'https://bugs.python.org/JoshuaLandau'

bugs.python.org fields:

activity = <Date 2021-06-22.22:13:28.002>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2015-01-25.15:46:56.223>
creator = 'Joshua.Landau'
dependencies = []
files = []
hgrepos = []
issue_num = 23316
keywords = []
message_count = 25.0
messages = ['234672', '234673', '234675', '234697', '234698', '234702', '234703', '234704', '234756', '234773', '235083', '395929', '395951', '395953', '395962', '395965', '395968', '395970', '395972', '395974', '395975', '395978', '396006', '396009', '396369']
nosy_count = 9.0
nosy_names = ['gvanrossum', 'terry.reedy', 'r.david.murray', 'SilentGhost', 'Joshua.Landau', 'serhiy.storchaka', 'NeilGirdhar', 'Guido.van.Rossum', 'iritkatriel']
pr_nums = []
priority = 'low'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23316'
versions = ['Python 3.11']

[JoshuaLandau]

It is claimed that all expressions are evaluated left-to-right, including in functions¹. However,

f(*a(), b=b())

will evaluate b() before a().

¹ https://docs.python.org/3/reference/expressions.html#evaluation-order

[SilentGhost]

It seems, if I read https://docs.python.org/3/reference/expressions.html#calls correctly that the evaluation order of the function arguments is not defined in general, as it depends on your use of keyword argument and exact function signature. Naturally, f(a(), b()) would be evaluated in the order you're expecting.

[bitdancer]

The resolution of bpo-16967 argues that this should probably be considered a bug. It certainly goes against normal Python expectations. I think it should also be considered to be of low priority, though.

[NeilGirdhar]

I assume this is the problem:

>>> dis.dis('f(*a(), b=b())')
  1           0 LOAD_NAME                0 (f)
              3 LOAD_NAME                1 (a)
              6 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              9 LOAD_CONST               0 ('b')
             12 LOAD_NAME                2 (b)
             15 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             18 CALL_FUNCTION_VAR      256 (0 positional, 1 keyword pair)
             21 RETURN_VALUE

— looks fine.

>>> dis.dis('f(b=b(), *a())')
  1           0 LOAD_NAME                0 (f)
              3 LOAD_NAME                1 (a)
              6 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              9 LOAD_CONST               0 ('b')
             12 LOAD_NAME                2 (b)
             15 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             18 CALL_FUNCTION_VAR      256 (0 positional, 1 keyword pair)
             21 RETURN_VALUE

Joshua, we could make function calls take:

x lists
y dictionaries
one optional list
z dictionaries

but we as well do all the merging in advance:

one optional list
one optional dictionary
one optional list
one optional dictionary

which is representable in three bits, but four is easier to decode I think.

[NeilGirdhar]

actually, we accept alternation, so maybe a bit to say whether we start with a list or a dict followed by a length of the alternating sequence?

[bitdancer]

Neil: I presume you are speaking of your in-progress PEP patch, and not the current python code? If so, please keep the discussion of handling this in the context of the PEP to the PEP issue. This issue should be for resolving the bug in the current code (if we choose to do so...if the PEP gets accepted for 3.5 this issue may become irrelevant, as I'm not sure we'd want to fix it in 3.4 for backward compatibility reasons).

[NeilGirdhar]

Yes, sorry David. I got linked here from the other issue. In any case, in the current code, the longest alternating sequence possible is 4. So one way to solve this is to change the CALL_FUNCTION parameters to be lists and dicts only and then process the final merging in CALL_FUNCTION.

[NeilGirdhar]

another option is to add a LIST_EXTEND(stack_difference) opcode that would allow us to take the late iterable and extend a list at some arbitrary stack position. I had to add something like that for dicts for the other issue, so it would follow that pattern.

[NeilGirdhar]

After thinking about this a bit more, my suggestion is not to fix it. Instead, I suggest that PEP-8 be modified to suggest that all positional arguments and iterable argument unpackings precede keyword arguments and keyword argument unpackings. Then, a tool like autopep8 is free to reorganize argument lists. Such reorganization will not be possible if f(*a(), b=b()) is different than f(b=b(), *a()).

[NeilGirdhar]

(I also suggest that the evaluation order within a function argument list to be defined to be positional and iterable before keyword, otherwise left-to-right — rather than strictly left-to-right).

[terryjreedy]

I expect left to right as documented (and designed by Guido). His OK or clarification would be to intentionally do differently.

[iritkatriel]

I don't think this is true anymore:

>>> def a(): 
...   print('a')
...   return (1,2)
... 
>>> def b():
...   print('b')
...   return (3,4)
... 
>>> def f(*args, b=None):
...   print('f')
... 
>>> f(*a(), b=b())
a
b
f
>>>

[terryjreedy]

The particular example of left-to-right function evaluation in
https://docs.python.org/3/reference/expressions.html#evaluation-order
is "expr1(expr2, expr3, *expr4, **expr5)".

Joshua's claim, without evidence, that "'f(*a(), b=b())' will evaluate b() before a()" was false, as shown by Neil with dis and is now, as shown by Irit with a code example.

But Neil also showed, again with dis, that a revised discrepancy claim, ""'f(b=b(), *a())' will evaluate a() before b()", is true, because reversing the order of these particular arguments does not reverse the evaluation order. The dis today is equivalent to the one 7 years ago, with CALL_FUNCTION_VAR expanded to two operations.

dis.dis('f(b=b(), *a())')
1 0 LOAD_NAME 0 (f)
2 LOAD_NAME 1 (a)
4 CALL_FUNCTION 0
6 LOAD_CONST 0 ('b')
8 LOAD_NAME 2 (b)
10 CALL_FUNCTION 0
12 BUILD_MAP 1
14 CALL_FUNCTION_EX 1
16 RETURN_VALUE

Irit's example, carried one step further confirms this.

>>> f(b=b(), *a())
a
b
f

Although I considered SilentGhost's reading of
https://docs.python.org/3/reference/expressions.html#calls
as too 'loose' (evaluation order of function argument *is* defined in general), that section address this exact case.

"A consequence of this is that although the *expression syntax may appear after explicit keyword arguments, it is processed before the keyword arguments ..."

I read this as an acknowledgement that this order violates the general rule.

I do wonder whether exception is specific to using a stack machine and implementation convenience or whether *all* implementations must follow this order. If C-Python specific, it should be labelled as such.

The doc continues with an example call, for a different 'f', where, unlike Irit's 'f', the result is "TypeError: f() got multiple values for keyword argument 'a'". It concludes "It is unusual for both keyword arguments and the *expression syntax to be used in the same call, so in practice this confusion does not arise."

The problem with mixing unpacking and named values might be even clearer if "f(b=1, *(2,1))" were added to the example box. Perhaps *exp after x=exp should be prohibited.

If the exception is not removed from the implementation, then perhaps it should be added to the evaluation order section. Something like

"The one exception is in function calls with *expression after a keyword argument: f(x=expr2, *expr1)."