Warn that evaluate() should not be used on user input

[corgeman]

The evaluate() function eventually calls eval() on the data provided. eval() is extremely dangerous when supplied with user input and to my knowledge it isn't mentioned that the function does this. I would add a warning in the documentation about this. As a proof-of-concept, the following code should execute the command 'echo verybad' on your computer when ran:

import numexpr
s = """
(lambda fc=(
    lambda n: [
        c for c in 
            ().__class__.__bases__[0].__subclasses__() 
            if c.__name__ == n
        ][0]
    ):
    fc("function")(
        fc("Popen")("echo verybad",shell=True),{}
    )()
)()
"""
numexpr.evaluate(s)

[robbmcleod]

Yeah developers seem to be using NumExpr more and more as a parser for handling input from a UI. It would be much safer if we could use ast.parse instead. I think adding a warning to the docs is fine, but it would also make sense to me to add an optional sanitizer. We could search the string for various Python keywords that have no business being in an expression, such as:

1. import
2. lambda
3. eval
4. __ (dunder)
5. locals
6. globals

Sanitization could be enabled by default but give the user the means to turn it off.

[robbmcleod]

Alright I implemented a check in 4b2d89c that forbids certain operators used in the expression. Namely: ;, :, [, and __. As far as I can tell this defeats all of the commonly cited attack vectors against eval(). I.e. it bans multiple expressions, lambdas, indexing, and all the dunders. If we want to be very safe, we would want to ban . as a character. However, this is difficult because . is both the reference operator and the decimal operator.

If we want to ban . we could do so if we require it to be followed by a numeral. E.g. 0.1 would pass the check, os.remove would not pass the check. However, this would ban valid Python code such as a * 5. which is a shorthand for casting to double. Perhaps there is someone out their with better regex-foo than myself and can suggest a solution for a regex that bans the Python . operator but doesn't cause issues for decimal points?

Regardless I think this is a good step forward and we're at the point where we should do a new release.

[corgeman]

Looks good to me!

[robbmcleod]

Ok, made some more improvements. I can ban attribute access to everything but .real and .imag, via

_forbidden_re = re.compile('[\;[\:]|__|\.[abcdefghjklmnopqstuvwxyzA-Z_]')

I also strip the string of all whitespace beforehand.

[robbmcleod]

Closing with release of 2.8.5.

[jan-kubena]

Hi @robbmcleod, just wanted to point out that you can still access other attributes because Python translates some utf chars into ascii chars automatically (mainly greek alphabet used in math), thus:

numexpr.evaluate("(3+1).ᵇit_count()")

Results in:

array(1, dtype=int32)

It might be better to whitelist real and imag rather than blacklist chars.

[robbmcleod]

@jan-kubena ok thanks for the heads up. The trouble here is that decimal needs to work, and numbers can appear in variable names, but not at the start. Do you happen to know where the documentation for which character sets are mangled is?

The really proper way to do it would be to back-port the ast parser work I did in for my attempt at making 3.0.

[robbmcleod]

Apparently the pandas group is using dunders as private variables in some of their NumExpr calls.

pandas-dev/pandas#54449

I'm of two minds about this. I could regex against "[a-zA-Z0-9_]+" instead of just "__". But for the security conscious, it does feel to me that NumExpr shouldn't be able to access private variables in the locals and globals dictionaries.

[nicoddemus]

Hi folks,

I have a simpler example which also breaks in the new version:

import pandas as pd

name = "Mass (kg)"
df = pd.DataFrame({name: [200, 300, 400]})
df.query(f"`{name}` < 300")

ValueError: Expression (BACKTICK_QUOTED_STRING_Mass__LPAR_kg_RPAR_) < (300) has forbidden control characters.

I understand Mass (kg) is a reasonable column name, so the validation definitely needs more tuning.

[zorion]

Hi folks,
I'm not sure if it is the same issue here, we are using numexpr for calculations in a pandas dataframe and we are using double underscore in some of our calculations (for instance, a__b) but now it fails in 2.8.5 (working in 2.8.4).
I have a minimal example without involving pandas:

import numexpr
numexpr.evaluate('a__b / 2', {'a__b': 4})

Or a oneliner:

python -c "import numexpr; print(numexpr.evaluate('a__b / 2', {'a__b': 4}))"

In numexpr==2.8.4 that one works perfectly.
In numexpr==2.8.5 we have the following:
ValueError: Expression a__b / 2 has forbidden control characters.

Is it an intentional feature or something that can be workedaround easily (sending some kwarg to ignore the error)?
We will pin our numexpr requeriments to "<2.8.5" in the meanwhile.

Many thanks!

[robbmcleod]

@nicoddemus and @zorion, just waiting to hear back from Pandas' devs on the original report: pandas-dev/pandas#54449 before I do anything. I've tried in the past to establish some sort of line of communication with them and gotten crickets.