Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SyntaxError when free variable name is also an exception target #48867

Closed
amauryfa opened this issue Dec 10, 2008 · 20 comments
Closed

SyntaxError when free variable name is also an exception target #48867

amauryfa opened this issue Dec 10, 2008 · 20 comments
Assignees
Labels
interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@amauryfa
Copy link
Member

amauryfa commented Dec 10, 2008

BPO 4617
Nosy @gvanrossum, @warsaw, @rhettinger, @terryjreedy, @pjeby, @amauryfa, @benjaminp, @ezio-melotti, @florentx
Files
  • delete_deref.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/amauryfa'
    closed_at = <Date 2010-09-10.22:02:17.749>
    created_at = <Date 2008-12-10.12:18:42.495>
    labels = ['interpreter-core', 'type-feature']
    title = 'SyntaxError when free variable name is also an exception target'
    updated_at = <Date 2013-04-19.00:49:14.320>
    user = 'https://github.com/amauryfa'

    bugs.python.org fields:

    activity = <Date 2013-04-19.00:49:14.320>
    actor = 'barry'
    assignee = 'amaury.forgeotdarc'
    closed = True
    closed_date = <Date 2010-09-10.22:02:17.749>
    closer = 'amaury.forgeotdarc'
    components = ['Interpreter Core']
    creation = <Date 2008-12-10.12:18:42.495>
    creator = 'amaury.forgeotdarc'
    dependencies = []
    files = ['12318']
    hgrepos = []
    issue_num = 4617
    keywords = ['patch']
    message_count = 20.0
    messages = ['77536', '77691', '77693', '77696', '77703', '77704', '78434', '79228', '99247', '99855', '99866', '99880', '99911', '99915', '99918', '99950', '113312', '113335', '113340', '116048']
    nosy_count = 11.0
    nosy_names = ['gvanrossum', 'jhylton', 'barry', 'rhettinger', 'terry.reedy', 'pje', 'amaury.forgeotdarc', 'benjamin.peterson', 'ezio.melotti', 'cmcqueen1975', 'flox']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue4617'
    versions = ['Python 3.2']

    @amauryfa
    Copy link
    Member Author

    amauryfa commented Dec 10, 2008

    This issue comes from bpo-4613. The following code raises a
    SyntaxError("can not delete variable 'e' referenced in nested scope"):

    def f():
        e = None
        def g():
            e
    try:
        pass
    except Exception as e:
        pass # SyntaxError here???
    

    The reason is because of
    http://www.python.org/dev/peps/pep-3110/#semantic-changes, a "del e"
    statement is inserted.

    The above code is correct, and should work.
    I suggest that the limitation: "can not delete variable referenced in
    nested scope" could be removed.

    After all, the "variable referenced" has no value before it is set,
    accessing it raises either NameError("free variable referenced before
    assignment in enclosing scope") or UnboundLocalError("local variable
    referenced before assignment")

    The Attached patch adds a DELETE_DEREF opcode, that removes the value of
    a cell variable, and put it in a "before assignment" state.

    Some compiler experts should review it. Few regressions are possible,
    since the new opcode is emitted where a SyntaxError was previously
    raised. The patch could also be applied to 2.7, even if it is less
    critical there.

    Tests are to come, but I'd like other's suggestions.

    @terryjreedy
    Copy link
    Member

    terryjreedy commented Dec 12, 2008

    -1 as I understand the proposal. Your code is bugged and should fail as
    soon as possible.

    If I understand correctly, you agree that the SyntaxError is correct as
    the language is currently defined, but you want the definition changed.
    It is not clear if you only want implicit deletes at the end of except
    clauses to work or if you only want explicit deletes to work.

    If the latter, you want

    def f():
      e = 1
      del e
      def g(): print(e)
      return g

    to compile. I would not. Your reason "After all, the "variable
    referenced" has no value before it is set," (duh, right) makes no sense
    to me in this context. g must have a valid value of e to run. So you
    seem to be suggesting that detection of buggy code should be delayed.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Dec 12, 2008

    Not sure the "del e" idea was a good solution to the garbage collection
    problem. Amaury's code looks correct to me. Maybe the existing e
    variable should be overwritten and the left intact (as it used to be) or
    perhaps it should be made both temporary and invisible like the
    induction variable in a list comprehension.

    Phillip, any thoughts?

    @amauryfa
    Copy link
    Member Author

    amauryfa commented Dec 12, 2008

    Terry, my motivation is that the sample code above runs correctly with
    python 2.6, but python 3.0 cannot even compile it. The sample looks valid
    python code, and should run.
    Yes, the same 'e' is used both as a nested variable and as an exception
    target, but this should not matter with our dynamic language.

    First I thought to turn the implicit "del e" into something else (and change
    PEP-3110),
    but then I saw that the error "can not delete variable referenced in nested
    scope" is actually a limitation of the interpreter that is easy to remove.

    @pjeby
    Copy link
    Mannequin

    pjeby mannequin commented Dec 13, 2008

    I could argue either way on this one; it's true that deleting a
    nested-scope variable is sometimes desirable, but it also seems to me
    like closing over an except: variable is a Potentially Bad Idea.

    In neither case, however, do I think it's appropriate to drop the
    temporary nature of the variable. I could perhaps get behind resetting
    the variable to None instead of deleting it, but of course the PEP would
    need changing. There's also a question of whether we should do the same
    thing with "with ... as" variables.

    (Btw, I'm not sure why this one's assigned to me; ISTM I might have
    proposed the current except/as GC semantics, but I'm not familiar with
    the actual implementation in 2.6 or 3.0)

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Dec 13, 2008

    Guido, any thoughts?

    @rhettinger rhettinger assigned gvanrossum and unassigned pjeby Dec 13, 2008
    @loewis loewis mannequin added release-blocker and removed deferred-blocker labels Dec 20, 2008
    @benjaminp
    Copy link
    Contributor

    benjaminp commented Dec 28, 2008

    I think being able to delete free variables is reasonable and brings
    more consistency as well as solving corner cases like this.

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Jan 6, 2009

    I don't think this has much to do with try/except. That it works in 2.6
    but not in 3.0 isn't a big deal; the semantics of variables used in
    except clauses has changed dramatically.

    It has to do with deletion of a variable that's held in a cell for
    reference by an inner function, like this:

    def outer():
      x = 0
      def inner(): return x
      del x  # SyntaxError

    I suspect (but do not know for sure) that the reason this is considered
    a SyntaxError is that the implementer of cells punted on the 'del'
    implementation and inserted a SyntaxError instead. (You can tell it's a
    pass-two SyntaxError because it doesn't mention a line number.)

    I think it's fine to fix this in 2.7 and 3.1, but I don't see it as a
    priority given that this has always been this way (and despite that it
    now affects try/except). It will probably require a new opcode.

    I don't see a reason to declare this a release blocker just because the
    try/except code is affected, and I don't think try/except needs to be
    changed to avoid this.

    @gvanrossum gvanrossum removed their assignment Jan 6, 2009
    @cmcqueen1975
    Copy link
    Mannequin

    cmcqueen1975 mannequin commented Feb 12, 2010

    There's also this one which caught me out:

    def outer():
      x = 0
      y = (x for i in range(10))
      del x  # SyntaxError

    @jhylton
    Copy link
    Mannequin

    jhylton mannequin commented Feb 22, 2010

    It's an interesting bug. Maybe the compiler shouldn't allow you to
    use such a variable as a free variable in a nested function?

    On Thu, Feb 11, 2010 at 9:09 PM, Craig McQueen <report@bugs.python.org> wrote:

    Craig McQueen <python@craig.mcqueen.id.au> added the comment:

    There's also this one which caught me out:

    def outer():
     x = 0
     y = (x for i in range(10))
     del x  # SyntaxError

    ----------
    nosy: +cmcqueen1975


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue4617\>



    Python-bugs-list mailing list
    Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Feb 22, 2010

    All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

    As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.


    (*) However there's also this example:

    >>> def f():
    ...  try: 1/0
    ...  except Exception as a:
    ...   def g(): return a
    ...   return g
    ... 
    SyntaxError: can not delete variable 'a' referenced in nested scope
    >>>

    @jhylton
    Copy link
    Mannequin

    jhylton mannequin commented Feb 22, 2010

    On Mon, Feb 22, 2010 at 6:10 PM, Guido van Rossum
    <report@bugs.python.org> wrote:

    Guido van Rossum <guido@python.org> added the comment:

    All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

    There's no reason we couldn't revise the language spec to explain that
    except clauses and comprehensions are block statements, i.e.
    statements that introduce a new block. For the except case, there
    would be some weird effects.

    y = 10
    try:
      ...
    except SomeError as err:
      y = 12
    print y  # prints 10

    In the example above, y would be a local variable in the scope of the
    except handler that shadows the local variable in the block that
    contains the try/except. It might be confusing that you couldn't
    assign to a local variable in the except handler without using a
    nonlocal statement.

    As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

    The general request here is to remove all the SyntaxErrors about
    deleting cell variables, right? Instead, you'd get a NameError at
    runtime saying that the variable is currently undefined. You'd want
    that change regardless of whether we change the language as described
    above.

    hoping-for-some-bdfl-pronouncements-ly y'rs,
    Jeremy


    (*) However there's also this example:

    >>> def f():
    ...  try: 1/0
    ...  except Exception as a:
    ...   def g(): return a
    ...   return g
    ...
    SyntaxError: can not delete variable 'a' referenced in nested scope
    >>>

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue4617\>


    @gvanrossum
    Copy link
    Member

    gvanrossum commented Feb 23, 2010

    On Mon, Feb 22, 2010 at 6:51 PM, Jeremy Hylton <report@bugs.python.org> wrote:

    There's no reason we couldn't revise the language spec to explain that
    except clauses and comprehensions are block statements, i.e.
    statements that introduce a new block.

    However (even apart from the below example) it would be tough to
    implement cleanly in CPython.

    For the except case, there would be some weird effects.

    y = 10
    try:
     ...
    except SomeError as err:
     y = 12
    print y  # prints 10

    In the example above, y would be a local variable in the scope of the
    except handler that shadows the local variable in the block that
    contains the try/except.  It might be confusing that you couldn't
    assign to a local variable in the except handler without using a
    nonlocal statement.

    Yeah, there are all sorts of problems with less-conspicuous nested
    scopes like this, for a language that defaults to local assignment
    like Python. Hence the horrible hacks.

    > As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

    The general request here is to remove all the SyntaxErrors about
    deleting cell variables, right?  Instead, you'd get a NameError at
    runtime saying that the variable is currently undefined.  You'd want
    that change regardless of whether we change the language as described
    above.

    Yeah, if we could kill those SyntaxErrors we can leave the rest as is.

    @amauryfa
    Copy link
    Member Author

    amauryfa commented Feb 23, 2010

    The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply here?

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Feb 23, 2010

    I don't think so. It's very marginal.

    --Guido (on Android)

    On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" <report@bugs.python.org>
    wrote:

    Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

    The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply
    here?

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python...

    @jhylton
    Copy link
    Mannequin

    jhylton mannequin commented Feb 23, 2010

    The patch looks pretty good.

    I'd factor out the common error-checking code (common between
    LOAD_DEREF and DELETE_DEREF) into a helper function.

    It would also be good to add some test cases.

    Jeremy

    On Tue, Feb 23, 2010 at 9:38 AM, Guido van Rossum
    <report@bugs.python.org> wrote:

    Guido van Rossum <guido@python.org> added the comment:

    I don't think so. It's very marginal.

    --Guido (on Android)

    On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" <report@bugs.python.org>
    wrote:

    Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

    The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply
    here?

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python...

    ----------
    Added file: http://bugs.python.org/file16341/unnamed


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue4617\>


    @florentx florentx mannequin added interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Mar 13, 2010
    @florentx
    Copy link
    Mannequin

    florentx mannequin commented Aug 8, 2010

    This bug is waiting for unit tests and a small patch cleanup.
    See previous message: http://bugs.python.org/issue4617#msg99950

    @terryjreedy
    Copy link
    Member

    terryjreedy commented Aug 8, 2010

    I have changed my mind on this issue. Since

    e = 1
    del e
    def g(): print(e)
    g()

    compiles and raises a run-time name error, so should the same code embedded within a function. In either case, the premature deletion is a logic error, not a syntax error.

    However, changing the language definition, even to fix what is considered a design bug, is a feature request. For both 2.7 and 3.1, section 6.5. "The del statement", says "It is illegal to delete a name from the local namespace if it occurs as a free variable in a nested block."

    So this seems too late for 2.7. On the other hand, Guido has allowed it for 3.2 in spite of the moratorium, but I think it should go in the initial release.

    @terryjreedy terryjreedy added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Aug 8, 2010
    @gvanrossum
    Copy link
    Member

    gvanrossum commented Aug 8, 2010

    Yeah, please fix in 3.2, don't fix in 2.7.

    @amauryfa amauryfa self-assigned this Sep 10, 2010
    @amauryfa
    Copy link
    Member Author

    amauryfa commented Sep 10, 2010

    Fixed in r84685, with tests and doc updates.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants