Evaluation order of dictionary display is different from reference manual.

[takayuki]

BPO	11205
Nosy	@gvanrossum, @birkenfeld, @rhettinger, @terryjreedy, @ncoghlan, @nedbat, @benjaminp, @ezio-melotti, @merwok, @bitdancer, @sandrotosi, @freakboy3742, @ericsnowcurrently
Files	compile.diff issue11205.patch issue11205-v2.patch issue11205-v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-06.02:29:41.330>
created_at = <Date 2011-02-13.09:39:12.878>
labels = ['type-bug', 'docs']
title = 'Evaluation order of dictionary display is different from reference manual.'
updated_at = <Date 2017-10-06.02:29:41.295>
user = 'https://bugs.python.org/takayuki'

bugs.python.org fields:

activity = <Date 2017-10-06.02:29:41.295>
actor = 'ncoghlan'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-10-06.02:29:41.330>
closer = 'ncoghlan'
components = ['Documentation']
creation = <Date 2011-02-13.09:39:12.878>
creator = 'takayuki'
dependencies = []
files = ['20754', '39218', '39225', '39228']
hgrepos = []
issue_num = 11205
keywords = ['patch']
message_count = 33.0
messages = ['128482', '128487', '128489', '128490', '128492', '128493', '128494', '128498', '128500', '128502', '128507', '128520', '128521', '128522', '128530', '138223', '175120', '178133', '179585', '185438', '185446', '240222', '240225', '242157', '242161', '242183', '242200', '242204', '244336', '244339', '244340', '263219', '303791']
nosy_count = 19.0
nosy_names = ['gvanrossum', 'georg.brandl', 'rhettinger', 'terry.reedy', 'ncoghlan', 'freakboy', 'nedbat', 'benjamin.peterson', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'sandro.tosi', 'freakboy3742', 'docs@python', 'swamiyeswanth', 'takayuki', 'python-dev', 'eric.snow', 'sdougherty']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11205'
versions = ['Python 3.5']

[takayuki]

Running the following code shows "2 1 4 3", but in reference manual
http://docs.python.org/reference/expressions.html#expression-lists
the evaluation order described as
{expr1: expr2, expr3: expr4}

def f(i):
    print i
    return i

{f(1):f(2), f(3):f(4)}

I found some of past discussions about this problem, for example,
http://mail.python.org/pipermail/python-dev/2002-November/030461.html
http://mail.python.org/pipermail/python-dev/2002-November/030458.html
http://bugs.python.org/issue448679
http://svn.python.org/view?view=rev&revision=30148
, but current implementation seems not to reflect these accomplishment.

In present, which behaviour is legal?

[smontanaro]

Here's a patch.

[smontanaro]

Working on a test case too.

[smontanaro]

It's not so easy as first appeared.

[smontanaro]

Okay, this looks better. I was confusing myself with leftover .pyc
files I think. Test included.

[smontanaro]

Georg, I think this might need to sneak into 3.2.

[birkenfeld]

I don't think so -- it's a very minor deviation from the spec and not a critical bug.

[smontanaro]

3.2 and earlier versions are all frozen, but for 3.3 it might make sense to bump the version number of the bytecode and change STORE_MAP to take the key and value in the opposite order, thus allowing to remove the ROT_TWO I had to add to make this work.

[bitdancer]

Interesting. I would actually have expected the observed behavior. I think of the : in a dictionary literal as an assignment.

[birkenfeld]

BTW, it would be nice to know if this behavior was consistent with the docs at any time (the merge of the AST branch in 2.5 might be an obvious candidate where it was broken).

Also interesting would be what other implementations of Python do.

[smontanaro]

As Georg suggested, it is correct in 2.4, wrong in 2.5.

[rhettinger]

<sigh> If only a test had been checked-in eight years ago ...

It looks like a number of things have to be changed in order to bring behavior back to what the docs promise. Dict literals and dict comprehensions need to be fixed in both the compile.py and compile.c. To avoid introducing a ROT_TWO, the store STORE_MAP and MAP_ADD opcodes need minor modifications (just switch the u and w variable assignments).

[rhettinger]

takayuki, a special thanks to you for submitting such a well-researched bug report :-)

[smontanaro]

To avoid introducing a ROT_TWO, the store STORE_MAP and MAP_ADD
opcodes need minor modifications (just switch the u and w variable
assignments).

Either of which would not be possible in anything other that 3.3 or
later, right?

[rhettinger]

How much to change and how far to backport may make for a good python-dev discussion.

ISTM that changing generated code goes hand-in-hand with changing opcode semantics (as long as the magic number gets bumped). OTOH, none of this may be worth backporting at all since no one seems to have noticed or cared for eight years.

I don't have any feelings about it one way or the other, but it would great to see Py3.2.1 get fixed properly.

[terryjreedy]

If I understand, since the present patch uses present opcode semantics, adding a rotate, it could go into 2.7 and 3.2. Correct?

3.3 could get an improved patch that instead changed opcodes to expect key and value in the other order.

[terryjreedy]

Today in pydev thread "chained assignment weirdity" Guido said
http://permalink.gmane.org/gmane.comp.python.devel/135746
"I agree that we should be *very* conservative in changing the
meaning of existing opcodes (adding new one is a different story)."
...
"Hm. I really don't think that is a good development for Python to
compromise in the area of expression evaluation order where side
effects are involved."
...

"I haven't looked at the proposed fixes, but I think correctness is more important than saving an extra bytecode (OTOH keeping the set of opcodes the same trumps both). I can't imagine that this extra opcode will be significant in many cases."

To which Nick C. replied
"Since you've indicated the implementation is in the wrong here and you
also want to preserve opcode semantics, I think Skip's patch is
correct, but also needs to be applied to dict comprehensions (now we
have them). The extra bytecode is only ROT_TWO, which is one of the
cheapest we have kicking around."

To which Guido said "Ok, somebody go for it! (Also please refer to my pronouncement in the bug"

[ezio-melotti]

I came across the same problem in bpo-16777.

IMHO the current behavior is better, and the documentation should be fixed instead, for the following reasons:

1. it's consistent with assignments, where the RHS is evaluated before the LHS (see also msg128500). This should also be the behavior of the dict(k=v) syntax, and what I personally expect;
2. changing it back is not backward-compatible with any code written during the last 10 years;
3. keeping the current behavior and fixing the docs is simpler than fixing the code to match the docs;

In addition, I would avoid writing code with side-effects in a dict literal, even if the order was documented and guaranteed. The fact that we don't see many reports about this seems to indicate that people don't write such code, or if they do they rely on the current order.

[gvanrossum]

I am sticking with my opinion from before: the code should be fixed. It doesn't look like assignment to me.

I am fine with making this a "feature" only fixed in 3.4. (You can even fix the docs in 3.3 as long as you fix them back for 3.4.)

Minor note for Ezio: dict(k=v, ...) doesn't have this problem because k is just a keyword, not an expression.

[nedbat]

Since this is documented in the Python Language Reference, it doesn't make much sense to have it describe one way for 3.3 and another for 3.4, does it? By definition, doesn't that make this an implementation dependency? We should update the docs to say that pairs in dicts will be evaluated left-to-right, but that the order of key and value is implementation dependent, since it actually is.

[terryjreedy]

I strongly agree with Guido that we should fix the code. To me, defined left-to-right evaluation of sub-expressions within an expression is a feature. C's 'implementation defined' is a nuisance.

I do not think we should temporarily change the doc. In other cases where we have only applied a bugfix to a future version, we have not temporarily doc'ed the bug as 'correct'. The change *should* be in
What's New.

[sdougherty]

Any word on either changing the documentation to match the behaviour or fixing this as a bug?

[gvanrossum]

This needs a code patch. But it can only be fixed for 3.5.
On Apr 7, 2015 11:21 AM, "Steve Dougherty" <report@bugs.python.org> wrote:

Steve Dougherty added the comment:

Any word on either changing the documentation to match the behaviour or
fixing this as a bug?

----------
nosy: +sdougherty

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11205\>

---

[sdougherty]

I've added a patch to change the order of evaluation and of STORE_MAP's arguments. It includes a test incorporating the review on the previous version.

I noticed I had to remove existing .pyc files to avoid TypeErrors about values being unhashable. I take it the version numbers in the filenames are sufficient to prevent that problem in a release?

[ncoghlan]

The pyc issue suggests the magic number embedded in pyc files to indicate
bytecode compatibility needs to be incremented. If I recall correctly, that
lives in Lib/importlib/_bootstrap.py these days.

[sdougherty]

I've added a change to the bytecode magic number as well. I don't see a magic tag anymore, either in the code or for recent versions.

There are autogenerated changes to Python/importlibs.h. Is my understanding correct that these are not to be committed?

[ericsnowcurrently]

Changes to importlib.h should always be committed. It is the frozen importlib._bootstrap module, which is the implementation of the import system used by the interpreter. So failure to commit changes to importlib.h means your changes to importlib._bootstrap will not have any effect.

[sdougherty]

I've added the importlib.h changes and changed the name of the test to be more descriptive.

[sdougherty]

Anyone care to review bpo-11205-v3.patch ?

[terryjreedy]

I downloaded and tried to apply to 3.5 (then default) as it was last Saturday, before the .b1 cutoff. Only ceval.py and test_compile.py patches worked. Everything else failed. You need to update your repository (3.5 is now a branch and default is 3.6) and then the patch.

[python-dev]

New changeset 6f05f83c7010 by Benjamin Peterson in branch '3.5':
in dict displays, evaluate the key before the value (closes bpo-11205)
https://hg.python.org/cpython/rev/6f05f83c7010

New changeset ba9e4df5368c by Benjamin Peterson in branch 'default':
merge 3.5 (bpo-11205)
https://hg.python.org/cpython/rev/ba9e4df5368c

[ncoghlan]

Temporarily reopening this as a docs bug - I think it's worth mentioning in the "Porting to Python 3.5" section of the What's New docs and as a "version changed" note in the dis module docs, as even though it's obscure, anyone that was inadvertently relying on the prior deviation from the spec is going to be confused by the behavioural change in 3.5.

(The specific case where this came up was Russell Keith-Magee encountering the semantic change in BUILD_MAP's expectations for argument order on the stack for his VOD bytecode transpiler)

[ncoghlan]

I'm unlikely to ever get to this (especially as 3.5 is now in security-fix only mode), so closing it again.