SyntaxError when free variable name is also an exception target

[amauryfa]

BPO	4617
Nosy	@gvanrossum, @warsaw, @rhettinger, @terryjreedy, @pjeby, @amauryfa, @benjaminp, @ezio-melotti, @florentx
Files	delete_deref.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/amauryfa'
closed_at = <Date 2010-09-10.22:02:17.749>
created_at = <Date 2008-12-10.12:18:42.495>
labels = ['interpreter-core', 'type-feature']
title = 'SyntaxError when free variable name is also an exception target'
updated_at = <Date 2013-04-19.00:49:14.320>
user = 'https://github.com/amauryfa'

bugs.python.org fields:

activity = <Date 2013-04-19.00:49:14.320>
actor = 'barry'
assignee = 'amaury.forgeotdarc'
closed = True
closed_date = <Date 2010-09-10.22:02:17.749>
closer = 'amaury.forgeotdarc'
components = ['Interpreter Core']
creation = <Date 2008-12-10.12:18:42.495>
creator = 'amaury.forgeotdarc'
dependencies = []
files = ['12318']
hgrepos = []
issue_num = 4617
keywords = ['patch']
message_count = 20.0
messages = ['77536', '77691', '77693', '77696', '77703', '77704', '78434', '79228', '99247', '99855', '99866', '99880', '99911', '99915', '99918', '99950', '113312', '113335', '113340', '116048']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'jhylton', 'barry', 'rhettinger', 'terry.reedy', 'pje', 'amaury.forgeotdarc', 'benjamin.peterson', 'ezio.melotti', 'cmcqueen1975', 'flox']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue4617'
versions = ['Python 3.2']

[amauryfa]

This issue comes from bpo-4613. The following code raises a
SyntaxError("can not delete variable 'e' referenced in nested scope"):

def f():
    e = None
    def g():
        e

try:
    pass
except Exception as e:
    pass # SyntaxError here???

The reason is because of
http://www.python.org/dev/peps/pep-3110/#semantic-changes, a "del e"
statement is inserted.

The above code is correct, and should work.
I suggest that the limitation: "can not delete variable referenced in
nested scope" could be removed.

After all, the "variable referenced" has no value before it is set,
accessing it raises either NameError("free variable referenced before
assignment in enclosing scope") or UnboundLocalError("local variable
referenced before assignment")

The Attached patch adds a DELETE_DEREF opcode, that removes the value of
a cell variable, and put it in a "before assignment" state.

Some compiler experts should review it. Few regressions are possible,
since the new opcode is emitted where a SyntaxError was previously
raised. The patch could also be applied to 2.7, even if it is less
critical there.

Tests are to come, but I'd like other's suggestions.

[terryjreedy]

-1 as I understand the proposal. Your code is bugged and should fail as
soon as possible.

If I understand correctly, you agree that the SyntaxError is correct as
the language is currently defined, but you want the definition changed.
It is not clear if you only want implicit deletes at the end of except
clauses to work or if you only want explicit deletes to work.

If the latter, you want

def f():
  e = 1
  del e
  def g(): print(e)
  return g

to compile. I would not. Your reason "After all, the "variable
referenced" has no value before it is set," (duh, right) makes no sense
to me in this context. g must have a valid value of e to run. So you
seem to be suggesting that detection of buggy code should be delayed.

[rhettinger]

Not sure the "del e" idea was a good solution to the garbage collection
problem. Amaury's code looks correct to me. Maybe the existing e
variable should be overwritten and the left intact (as it used to be) or
perhaps it should be made both temporary and invisible like the
induction variable in a list comprehension.

Phillip, any thoughts?

[amauryfa]

Terry, my motivation is that the sample code above runs correctly with
python 2.6, but python 3.0 cannot even compile it. The sample looks valid
python code, and should run.
Yes, the same 'e' is used both as a nested variable and as an exception
target, but this should not matter with our dynamic language.

First I thought to turn the implicit "del e" into something else (and change
PEP-3110),
but then I saw that the error "can not delete variable referenced in nested
scope" is actually a limitation of the interpreter that is easy to remove.

[pjeby]

I could argue either way on this one; it's true that deleting a
nested-scope variable is sometimes desirable, but it also seems to me
like closing over an except: variable is a Potentially Bad Idea.

In neither case, however, do I think it's appropriate to drop the
temporary nature of the variable. I could perhaps get behind resetting
the variable to None instead of deleting it, but of course the PEP would
need changing. There's also a question of whether we should do the same
thing with "with ... as" variables.

(Btw, I'm not sure why this one's assigned to me; ISTM I might have
proposed the current except/as GC semantics, but I'm not familiar with
the actual implementation in 2.6 or 3.0)

[rhettinger]

Guido, any thoughts?

[benjaminp]

I think being able to delete free variables is reasonable and brings
more consistency as well as solving corner cases like this.

[gvanrossum]

I don't think this has much to do with try/except. That it works in 2.6
but not in 3.0 isn't a big deal; the semantics of variables used in
except clauses has changed dramatically.

It has to do with deletion of a variable that's held in a cell for
reference by an inner function, like this:

def outer():
  x = 0
  def inner(): return x
  del x  # SyntaxError

I suspect (but do not know for sure) that the reason this is considered
a SyntaxError is that the implementer of cells punted on the 'del'
implementation and inserted a SyntaxError instead. (You can tell it's a
pass-two SyntaxError because it doesn't mention a line number.)

I think it's fine to fix this in 2.7 and 3.1, but I don't see it as a
priority given that this has always been this way (and despite that it
now affects try/except). It will probably require a new opcode.

I don't see a reason to declare this a release blocker just because the
try/except code is affected, and I don't think try/except needs to be
changed to avoid this.

[cmcqueen1975]

There's also this one which caught me out:

def outer():
  x = 0
  y = (x for i in range(10))
  del x  # SyntaxError

[jhylton]

It's an interesting bug. Maybe the compiler shouldn't allow you to
use such a variable as a free variable in a nested function?

On Thu, Feb 11, 2010 at 9:09 PM, Craig McQueen <report@bugs.python.org> wrote:

Craig McQueen <python@craig.mcqueen.id.au> added the comment:

There's also this one which caught me out:

def outer():
 x = 0
 y = (x for i in range(10))
 del x  # SyntaxError

----------
nosy: +cmcqueen1975

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4617\>

---

---

Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu

[gvanrossum]

All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

---

(*) However there's also this example:

>>> def f():
...  try: 1/0
...  except Exception as a:
...   def g(): return a
...   return g
... 
SyntaxError: can not delete variable 'a' referenced in nested scope
>>>

[jhylton]

On Mon, Feb 22, 2010 at 6:10 PM, Guido van Rossum
<report@bugs.python.org> wrote:

Guido van Rossum <guido@python.org> added the comment:

All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

There's no reason we couldn't revise the language spec to explain that
except clauses and comprehensions are block statements, i.e.
statements that introduce a new block. For the except case, there
would be some weird effects.

y = 10
try:
  ...
except SomeError as err:
  y = 12
print y  # prints 10

In the example above, y would be a local variable in the scope of the
except handler that shadows the local variable in the block that
contains the try/except. It might be confusing that you couldn't
assign to a local variable in the except handler without using a
nonlocal statement.

As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

The general request here is to remove all the SyntaxErrors about
deleting cell variables, right? Instead, you'd get a NameError at
runtime saying that the variable is currently undefined. You'd want
that change regardless of whether we change the language as described
above.

hoping-for-some-bdfl-pronouncements-ly y'rs,
Jeremy

---

(*) However there's also this example:

>>> def f():
...  try: 1/0
...  except Exception as a:
...   def g(): return a
...   return g
...
SyntaxError: can not delete variable 'a' referenced in nested scope
>>>

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4617\>

---

[gvanrossum]

On Mon, Feb 22, 2010 at 6:51 PM, Jeremy Hylton <report@bugs.python.org> wrote:

There's no reason we couldn't revise the language spec to explain that
except clauses and comprehensions are block statements, i.e.
statements that introduce a new block.

However (even apart from the below example) it would be tough to
implement cleanly in CPython.

For the except case, there would be some weird effects.

y = 10
try:
 ...
except SomeError as err:
 y = 12
print y  # prints 10

In the example above, y would be a local variable in the scope of the
except handler that shadows the local variable in the block that
contains the try/except.  It might be confusing that you couldn't
assign to a local variable in the except handler without using a
nonlocal statement.

Yeah, there are all sorts of problems with less-conspicuous nested
scopes like this, for a language that defaults to local assignment
like Python. Hence the horrible hacks.

> As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

The general request here is to remove all the SyntaxErrors about
deleting cell variables, right?  Instead, you'd get a NameError at
runtime saying that the variable is currently undefined.  You'd want
that change regardless of whether we change the language as described
above.

Yeah, if we could kill those SyntaxErrors we can leave the rest as is.

[amauryfa]

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply here?

[gvanrossum]

I don't think so. It's very marginal.

--Guido (on Android)

On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" <report@bugs.python.org>
wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply
here?

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python...

[jhylton]

The patch looks pretty good.

I'd factor out the common error-checking code (common between
LOAD_DEREF and DELETE_DEREF) into a helper function.

It would also be good to add some test cases.

Jeremy

On Tue, Feb 23, 2010 at 9:38 AM, Guido van Rossum
<report@bugs.python.org> wrote:

Guido van Rossum <guido@python.org> added the comment:

I don't think so. It's very marginal.

--Guido (on Android)

On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" <report@bugs.python.org>
wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply
here?

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python...

----------
Added file: http://bugs.python.org/file16341/unnamed

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4617\>

---

[florentx]

This bug is waiting for unit tests and a small patch cleanup.
See previous message: http://bugs.python.org/issue4617#msg99950

[terryjreedy]

I have changed my mind on this issue. Since

e = 1
del e
def g(): print(e)
g()

compiles and raises a run-time name error, so should the same code embedded within a function. In either case, the premature deletion is a logic error, not a syntax error.

However, changing the language definition, even to fix what is considered a design bug, is a feature request. For both 2.7 and 3.1, section 6.5. "The del statement", says "It is illegal to delete a name from the local namespace if it occurs as a free variable in a nested block."

So this seems too late for 2.7. On the other hand, Guido has allowed it for 3.2 in spite of the moratorium, but I think it should go in the initial release.

[gvanrossum]

Yeah, please fix in 3.2, don't fix in 2.7.

[amauryfa]

Fixed in r84685, with tests and doc updates.