compile() raises SystemError if called from except clause

[tifv]

BPO	11441
Nosy	@brettcannon, @birkenfeld, @amauryfa, @ncoghlan, @benjaminp, @tifv, @durban
Files	issue11441.patch issue11441_2.patch: New patch with test in test_compile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-01-11.17:06:19.547>
created_at = <Date 2011-03-08.16:24:43.753>
labels = ['type-bug', 'library']
title = 'compile() raises SystemError if called from except clause'
updated_at = <Date 2013-01-11.17:06:19.546>
user = 'https://github.com/tifv'

bugs.python.org fields:

activity = <Date 2013-01-11.17:06:19.546>
actor = 'brett.cannon'
assignee = 'none'
closed = True
closed_date = <Date 2013-01-11.17:06:19.547>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2011-03-08.16:24:43.753>
creator = 'july'
dependencies = []
files = ['21054', '21055']
hgrepos = []
issue_num = 11441
keywords = ['patch']
message_count = 16.0
messages = ['130342', '130343', '130346', '130374', '130378', '130380', '130386', '130431', '130434', '130436', '130453', '130454', '130515', '130517', '130518', '179704']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'georg.brandl', 'amaury.forgeotdarc', 'ncoghlan', 'benjamin.peterson', 'july', 'daniel.urban']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11441'
versions = ['Python 3.2', 'Python 3.3']

[tifv]

Normal:
>>> compile('1 = 1', '<string>', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
SyntaxError: can't assign to literal

SystemError is raised instead of SyntaxError:
>>> try: abcde
... except NameError:
...  compile('1 = 1', '<string>', 'exec')
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'abcde' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
SystemError: Objects/tupleobject.c:126: bad argument to internal function

Error can be discovered by calling dis.dis('1 = 1').

[durban]

Apparently ast_error_finish calls PyTuple_GetItem with a value that is not a tuple, but a SyntaxError instance (in Python/ast.c line 112). It seems that ast_error_finish expects that PyErr_Fetch will return the exception value as a tuple, and in some cases this seems correct (for example when not in an except clause), but not in this case. I don't know much about Python exception handling in C, but it seems to me, that it shouldn't expect always a tuple (see also http://docs.python.org/dev/py3k/c-api/exceptions.html#PyErr_NormalizeException).

[amauryfa]

Right. In most cases, "PyErr_SetObject(PyExc_SyntaxError, tuple);" will store the untouched tuple in tstate->curexc_value, *except* when "Implicit exception chaining" occurs, in which case the exception is normalized.

ast_error_finish() should not expect a tuple in all cases, and should probably normalize the exception as well.

[durban]

Here is a patch. I wasn't sure, where to put the test, so I put it in test_ast.

[benjaminp]

You can put the test in test_compile.

[durban]

Okay, here is a new patch with the test in the correct place (I hope).

[amauryfa]

Why is the exception normalized at the end? I suppose it's because when value is an exception instance, it's replaced by a tuple, but the original value has to be recreated at the end. So in some cases, the SyntaxError object is created twice...

If PyErr_NormalizeException() can't be avoided, I suggest to call it at the start, just after PyErr_Fetch, and use the PySyntaxErrorObject* structure directly to get the file name and line numbers.

[durban]

Why is the exception normalized at the end? I suppose it's because
when value is an exception instance, it's replaced by a tuple, but the
original value has to be recreated at the end. So in some cases, the
SyntaxError object is created twice...

If PyErr_NormalizeException() can't be avoided, I suggest to call it
at the start, just after PyErr_Fetch, and use the PySyntaxErrorObject*
structure directly to get the file name and line numbers.

Yeah, it is because ast_error_finish creates a new tuple to use as the exception value. (It creates a new (errstr, (filename, lineno, offset, loc)) tuple from the original (errstr, lineno, offset) tuple). And yes, in some cases the SyntaxError instance is created twice. I wasn't sure if it's okay to simply replace the args field of a PyBaseExceptionObject. I don't know, if PyErr_NormalizeException() can be avoided, you wrote, that it "should probably normalize the exception as well".

Would it be better, if we, when got an exception instance, create the new tuple from the info, and replace the args field of the instance with it? (But it also seems to me, that the SyntaxError objects have other fields as well, so probably we should modify them also. That's why I thought that calling PyErr_NormalizeException with the new tuple is the simplest thing to do, becuase I guess that'll take care of all fields automatically.)

[tifv]

There is an XXX just before the definition of ast_error. Wouldn't it be
useful?

The idea is to merge ast_error() and ast_error_finish().
This requires redefinition of most functions in ast.c, adding "const char
*filename" to their parameters.

[amauryfa]

That's why I thought that calling PyErr_NormalizeException with the new
tuple is the simplest thing to do, becuase I guess that'll take care of
all fields automatically.

You could also call PyErr_NormalizeException at the beginning, and update the fields directly in the PySyntaxErrorObject structure. No need to deal with any tuple.

[durban]

You could also call PyErr_NormalizeException at the beginning, and
update the fields directly in the PySyntaxErrorObject structure. No
need to deal with any tuple.

Sorry, but I don't really understand. If I call PyErr_NormalizeException at the beginning, the SyntaxError instance will be initialized with the wrong 3-tuple: (errstr, lineno, offset). If after that I update the msg, filename, ... fields, they will be correct, but I think the args field still will be the wrong 3-tuple. So I'll have to still create the new tuple, and replace the args field, right?

[amauryfa]

hmm, you are right, of course. I forgot that e.args is part of the SyntaxError members.

[durban]

So, I see four possible solutions:

1. If we get a tuple, create the new tuple, normalize the exception, and store it. If we get a SyntaxError instance, use its args, create the new tuple, normalize, and store. (In this case a SyntaxError instance will be created twice.)

2. If we get a tuple, create the new tuple and store it without normalization. If we get a SyntaxError instance use its args to create the new tuple and store it without normalization. (I think, that later it's still possible that a new SynaxError will be created, but we don't create it here.)

3. If we get a tuple, create the new tuple, and store it without normalization. If we get a SyntaxError, take its args, create the new tuple, and call SyntaxError.__init__ with it. I think this will set all fields properly.

4. Like 3., but if we got a tuple, store the new tuple with normalization.

My patch currently does 1.
My patch, without the PyErr_NormalizeException() call would be 2.
I think maybe 3. would be the best solution, or 4., if normalization is desired in all cases.

I can write a new patch, if the experts tell me what is the best solution from the four (or some other I didn't think of).

[amauryfa]

I'd choose solution 3, but instead of calling SyntaxError.__init__, call PyErr_NormalizeException().

[durban]

Err... sorry, I don't understand again:

If we get a tuple, create a new, store it without normalization. That's okay, I understand.

If we get a SyntaxError instance, then take its args field, create the new tuple. Then call PyErr_NormalizeException(), with:
a) the new tuple? But this will create a new SyntaxError instance...
b) the old SyntaxError instance? But this won't correct the wrong fields of the instance. (I think SyntaxError.__init__ would correct them without creating a new instance.)
c) or replace the args of the SyntaxError instance with the new tuple, then call PyErr_NormalizeException() on the instance? But I think that still won't correct the other fields of the instance...

Sorry for all these questions... I'd just like to help.

[brettcannon]

This no longer seems to be a problem in Python 3.2, 3.3, or 3.4.