Over restricted type checking on eval() function

[anonymous]

BPO	215126
Nosy	@tim-one, @arigo, @rhettinger
Files	any_locals.diff: Any object can be used a f_locals now. geneval3.diff: Patch with unittests, ready for 2nd review geneval4a.diff: Revised patch allowing stores and deletes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/arigo'
closed_at = <Date 2006-08-01.10:49:17.000>
created_at = <Date 2000-09-22.19:36:01.000>
labels = ['interpreter-core', 'type-feature']
title = 'Over restricted type checking on eval() function'
updated_at = <Date 2006-08-01.10:49:17.000>
user = 'https://bugs.python.org/anonymous'

bugs.python.org fields:

activity = <Date 2006-08-01.10:49:17.000>
actor = 'arigo'
assignee = 'arigo'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2000-09-22.19:36:01.000>
creator = 'anonymous'
dependencies = []
files = ['3', '4', '5']
hgrepos = []
issue_num = 215126
keywords = []
message_count = 24.0
messages = ['1555', '1556', '1557', '1558', '1559', '1560', '1561', '1562', '1563', '1564', '1565', '1566', '1567', '1568', '1569', '1570', '1571', '1572', '1573', '1574', '1575', '1576', '1577', '1578']
nosy_count = 7.0
nosy_names = ['tim.peters', 'arigo', 'mcherm', 'gregsmith', 'pfremy', 'rhettinger', 'aaiyer']
pr_nums = []
priority = 'high'
resolution = None
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue215126'
versions = []

[nobody]

The built-in function eval() takes a string argument and a dictionary. The second argument should allow any instance which defines __getitem__ as opposed to just dictionaries.

The following example creates a type error:
eval, argument 2: expected dictionary, instance found

class SpreadSheet:
    _cells = {}
    def __setitem__( self, key, formula ):
        self._cells[key] = formula
    def __getitem__( self, key ):
        return eval( self._cells[key], self )

ss = SpreadSheet()
ss['a1'] = '5'
ss['a2'] = 'a1*5'
ss['a2']

[tim-one]

Changed Group to Feature Request. Should be added to PEP-42 (I'll do that if nobody beats me to it).

CPython requires a genuine dict now for speed. I believe JPython allows any mapping object.

[tim-one]

Added to PEP-42.

[mcherm]

Logged In: YES
user_id=99874

I'll just add my voice as somebody who would find this to
be "darn handy" if it could ever done without noticably
impacting the speed of python code that DOESN'T use eval().

[gregsmith]

Logged In: YES
user_id=292741

This may be a compromise solution, which could also be
useful in other contexts: What if the object passed
is derived from dict - presumably that doesn't help
because the point is to do low-level calls to the
actual dict's lookup functions.
Now, suppose we modify the basic dict type, so
that before throwing
a KeyError, it checks to see if it is really a derived
object with a method __keyerror__, and if so, calls
that and returns its result (or lets it throw)?
Now you can make objects that look like dicts, and
act like them at the low level, but can automatically
populate themselves when non-existent keys are requested.
Of course, __keyerror__('x') does not have to make
an 'x' entry in the dict; it could make no change, or
add several entries, depending on the desired semantics
regarding future lookups. It could be set up so that
every lookup fails and is forwarded by __keyerror__ to
the __getitem__ of another object, for instance.

The cost of this to the 'normal' dict lookup is that
the need to do PyDict_CheckExact() each time
a lookup fails.

[mcherm]

Logged In: YES
user_id=99874

Hmm... I like this!

Of course, I am wary of adding *yet another* special double-
underscore function to satisfy a single special purpose, but
this would satisfy all of *my* needs, and without any
performance impact for lookups that are FOUND. Lookups that
are NOT found would have a slight performance degrade (I
know better than to speculate about the size of the effect
without measuring it). I'm not really sure what percentage of
dict lookups succeed.

At any rate, what are these "other contexts" you mention in
which a __keyerror__ would "also be useful"? Because if we
can find other places where it is useful, that significantly
improves the usefulness.

OTOH, if the tests can be done ONLY for eval (I don't really
think that's possible), then I'm certain no one cares about the
performance of eval.

[pfremy]

Logged In: YES
user_id=233844

I have exactly the same need as the original poster. The
only difference in my case is that I inhreted a dictionnary.

I want to use the eval() function as a simple expression
evaluator. I have the
follwing dictionnary:
d['a']='1'
d['b']='2'
d['c']='a+b'

I want the following results:
d[a] -> 1
d[b] -> 2
d[c] -> 3

To do that, I was planning to use the eval() function and
overloading the __getitem__ of the global or local dictionnary:

class MyDict( dict ) : 
	def __getitem__( self, key ):
		print "__getitem__", key
		val = dict.__getitem__( self, key )
		print "val = '%s'" % val
		return eval( val , self )

But it does not work:
d[a]:
__getitem__ a
val = '1'
-> 1

d[b]:
__getitem__ b
val = '2'
-> 2

d[c]:
__getitem__ c
val = 'e+1'
ERROR
Traceback (most recent call last):
  File "test_parse_jaycos_config.py", line 83, in testMyDict
    self.assertEquals( d['c'], 2 )
  File "parse_config_file.py", line 10, in __getitem__
    return eval( val , self )
  File "<string>", line 0, in ?
TypeError: cannot concatenate 'str' and 'int' objects

d['c'] did fetch the 'a+1' value, which was passed to eval.
However, eval()
tried to evaluate the expression using the content of the
dictionnary instead
of using __getitem__. So it fetched the string '1' for 'a'
instead of the value 1, so the result is not suitable for
the addition.

[arigo]

Logged In: YES
user_id=4771

With minimal work and performance impact, we could allow
frame->f_locals to be something else than a real dictionary.
It looks like it would be easily possible as long as we
don't touch the time-critical frame->f_globals. Code
compiled by eval() always uses LOAD_NAME and STORE_NAME,
which is anyway allowed to be a bit slower than LOAD_FAST or
LOAD_GLOBAL.

Note that PyDict_GetItem() & co., as called by
LOAD/STORE_NAME, do a PyDict_Check() anyway. We could just
replace it with PyDict_CheckExact() and if it fails fall
back to calling the appropriate ob_type->tp_as_mapping
method. This would turn some of the PyDict_Xxx() API into a
general mapping API, half-way between its current dict-only
usage and the fully abstract PyObject_Xxx() API.

This is maybe a bit strange from the user's point of view,
because eval("expr", mydict) still wouldn't work: only
eval("expr", {}, mydict) would.
which does which does
Now, there is no way I can think of that code compiled by
eval() could contain LOAD_GLOBAL or STORE_GLOBAL. The only
way to tell the difference between eval("expr", mydict) and
eval("expr", {}, mydict) appears to be by calling globals()
or somehow inspecting the frame directly. Therefore it
might be acceptable to redefine the two-argument eval(expr,
dict) to be equivalent not to eval(expr, dict, dict) but
eval(expr, {}, dict). This hack might be useful enough even
if it won't work with the exec statement.

[rhettinger]

Logged In: YES
user_id=80475

+1

Amrin's idea provides most of the needed functionality with
zero performance impact.

Also, using a local dictionary for the application variables
is likely to be just exactly what a programmer would want to do.

[rhettinger]

Logged In: YES
user_id=80475

Armin, can you whip-up a quick patch so that we can explore
the implications of your suggestion. Ideally, I would like
to see something like this go in for Py2.4.

[arigo]

Logged In: YES
user_id=4771

Quick patch attached. I didn't try to use the PyDict_GetItem trick described, but just systematically use PyObject_GetItem/SetItem/DelItem when working with f_locals.

This might confuse some extension modules that expect PyEval_GetLocals() to return a dict object.

The eval trick is now: eval(code, nondict) --> eval(code, globals(), nondict).

Besides eval() I removed the relevant typecheck from execfile() and the exec statement. Any other place I am missing?

We might want to still somehow check the type of the locals, to avoid strange errors caused by e.g. eval("a", "b"). PyMapping_Check() is the obvious candidate, but it looks like a hack.

More testing is needed. test_descrtut.py line 84 now succeeds, unexpectedly, which is interpreted as a test failure.

Needs some docs, too.