Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Support py3k kwonly args #7

Merged
merged 2 commits into from

2 participants

@eevee

They were being correctly parsed already, but a template like this:

<%def name="asdf(*qwert, zxcv='hi')">${zxcv}</%def>

would generate code like this:

def render_asdf(context,*qwert):
    __M_caller = context.caller_stack._push_frame()
    try:
        zxcv = context.get('zxcv', UNDEFINED)
    ...

which would then either TypeError when called with the required keyword argument, or (if there were also a **kwargs) have the argument overwritten by UNDEFINED right out of the gate.

Should work now. Added a few tests. Everything passes on Python 2.7, Python 3.3, and PyPy 2.2.1 (except for two tests that fail PyPy in Beaker somewhere, so not a Mako problem).

Inspired by... mitsuhiko, or python-future, or someone?, I added a compat.py2k, so code can be written as though Python 3 were the default and Python 2 were the special case.

I also made a best effort at the _ast-less code path; four tests fail, but all four of them are problems with how Pygments decided to color the syntax, so I assume that's just an inherent issue with the compiler module. (Two of them are marked as requiring Python 2.5+, so will never run in practice anyway.)

eevee added some commits
@eevee eevee Support Python 3's keyword-only arguments.
Previously, they would parse correctly in Python 3, but any keyword-only
arguments would be quietly lost, and the user would either get
`TypeError: foo() got an unexpected keyword argument...` or the
confusing behavior of having the keyword argument overwritten with
whatever's in the context with the same name.
836e5f9
@eevee eevee Make a best effort at pre-2.5 compatibility. 11a2349
@zzzeek
Owner

that is a motherlode of a pullrequest, bud, i need to take some time to see what im lookin at.

@zzzeek

random guess, x[::-1] means "reversed()". going to try that now. (tries) it does! but i know, why use plain english "reversed" when you can perl it up.... :)

iirc reverse is actually the only way to do this in perl!

but in python 3 reversed always returns an iterator, so .pop won't work. so it'd have to be list(reversed(...)) which is a bit much imo :)

Owner

damn that py3k and its iterators!

@eevee

some brief explanation is perhaps in order:

py2 has args, vararg, kwarg, defaults.

py3 adds kwonlyargs and kw_defaults, for single args that come after *args. the defaults are separate because they don't have to be clustered all at the end; def foo(*, a=1, b, c=3) is totally valid. (it also adds attributes for annotations, but i have no idea how to make those work here.)

so now positional and named args are much more similar: there can be any number of them, some of them might have defaults, and there might be a slurpy one at the end. so i split up the names and defaults and handle them separately

@zzzeek
Owner

did you mark the failing tests as skip? I'm not getting any failures even if i run nose in python 2.4. these tests only fail when pygments is installed?

@eevee

Yeah, two are 2.5+ only and two require Pygments 1.4+. Does Pygments not work on 2.4 any more? I couldn't find a minimum version mentioned anywhere,

@zzzeek
Owner

im pretty unconcerned about py2.4, supporting it is mostly just a practice exercise at this point.

@zzzeek zzzeek merged commit 11a2349 into zzzeek:master
@zzzeek
Owner

woop, thanks a bunch! maybe ill even release 0.9.2 soon.

@eevee

woo thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 11, 2014
  1. @eevee

    Support Python 3's keyword-only arguments.

    eevee authored
    Previously, they would parse correctly in Python 3, but any keyword-only
    arguments would be quietly lost, and the user would either get
    `TypeError: foo() got an unexpected keyword argument...` or the
    confusing behavior of having the keyword argument overwritten with
    whatever's in the context with the same name.
  2. @eevee
This page is out of date. Refresh to see the latest.
View
77 mako/ast.py
@@ -112,38 +112,65 @@ def __init__(self, code, allow_kwargs=True, **exception_kwargs):
if not allow_kwargs and self.kwargs:
raise exceptions.CompileException(
"'**%s' keyword argument not allowed here" %
- self.argnames[-1], **exception_kwargs)
+ self.kwargnames[-1], **exception_kwargs)
- def get_argument_expressions(self, include_defaults=True):
- """return the argument declarations of this FunctionDecl as a printable
- list."""
+ def get_argument_expressions(self, as_call=False):
+ """Return the argument declarations of this FunctionDecl as a printable
+ list.
+
+ By default the return value is appropriate for writing in a ``def``;
+ set `as_call` to true to build arguments to be passed to the function
+ instead (assuming locals with the same names as the arguments exist).
+ """
namedecls = []
- defaults = [d for d in self.defaults]
- kwargs = self.kwargs
- varargs = self.varargs
- argnames = [f for f in self.argnames]
- argnames.reverse()
- for arg in argnames:
- default = None
- if kwargs:
- arg = "**" + arg_stringname(arg)
- kwargs = False
- elif varargs:
- arg = "*" + arg_stringname(arg)
- varargs = False
+
+ # Build in reverse order, since defaults and slurpy args come last
+ argnames = self.argnames[::-1]
+ kwargnames = self.kwargnames[::-1]
+ defaults = self.defaults[::-1]
+ kwdefaults = self.kwdefaults[::-1]
+
+ # Named arguments
+ if self.kwargs:
+ namedecls.append("**" + kwargnames.pop(0))
+
+ for name in kwargnames:
+ # Keyword-only arguments must always be used by name, so even if
+ # this is a call, print out `foo=foo`
+ if as_call:
+ namedecls.append("%s=%s" % (name, name))
+ elif kwdefaults:
+ default = kwdefaults.pop(0)
+ if default is None:
+ # The AST always gives kwargs a default, since you can do
+ # `def foo(*, a=1, b, c=3)`
+ namedecls.append(name)
+ else:
+ namedecls.append("%s=%s" % (
+ name, pyparser.ExpressionGenerator(default).value()))
else:
- default = len(defaults) and defaults.pop() or None
- if include_defaults and default:
- namedecls.insert(0, "%s=%s" %
- (arg,
- pyparser.ExpressionGenerator(default).value()
- )
- )
+ namedecls.append(name)
+
+ # Positional arguments
+ if self.varargs:
+ namedecls.append("*" + argnames.pop(0))
+
+ for name in argnames:
+ if as_call or not defaults:
+ namedecls.append(name)
else:
- namedecls.insert(0, arg)
+ default = defaults.pop(0)
+ namedecls.append("%s=%s" % (
+ name, pyparser.ExpressionGenerator(default).value()))
+
+ namedecls.reverse()
return namedecls
+ @property
+ def allargnames(self):
+ return tuple(self.argnames) + tuple(self.kwargnames)
+
class FunctionArgs(FunctionDecl):
"""the argument portion of a function declaration"""
View
4 mako/codegen.py
@@ -543,7 +543,7 @@ def write_def_decl(self, node, identifiers):
"""write a locally-available callable referencing a top-level def"""
funcname = node.funcname
namedecls = node.get_argument_expressions()
- nameargs = node.get_argument_expressions(include_defaults=False)
+ nameargs = node.get_argument_expressions(as_call=True)
if not self.in_def and (
len(self.identifiers.locally_assigned) > 0 or
@@ -864,7 +864,7 @@ def visitBlockTag(self, node):
if node.is_anonymous:
self.printer.writeline("%s()" % node.funcname)
else:
- nameargs = node.get_argument_expressions(include_defaults=False)
+ nameargs = node.get_argument_expressions(as_call=True)
nameargs += ['**pageargs']
self.printer.writeline("if 'parent' not in context._data or "
"not hasattr(context._data['parent'], '%s'):"
View
1  mako/compat.py
@@ -3,6 +3,7 @@
py3k = sys.version_info >= (3, 0)
py33 = sys.version_info >= (3, 3)
+py2k = sys.version_info < (3,)
py26 = sys.version_info >= (2, 6)
py25 = sys.version_info >= (2, 5)
jython = sys.platform.startswith('java')
View
12 mako/parsetree.py
@@ -437,7 +437,7 @@ def get_argument_expressions(self, **kw):
return self.function_decl.get_argument_expressions(**kw)
def declared_identifiers(self):
- return self.function_decl.argnames
+ return self.function_decl.allargnames
def undeclared_identifiers(self):
res = []
@@ -451,7 +451,7 @@ def undeclared_identifiers(self):
).union(
self.expression_undeclared_identifiers
).difference(
- self.function_decl.argnames
+ self.function_decl.allargnames
)
class BlockTag(Tag):
@@ -502,7 +502,7 @@ def get_argument_expressions(self, **kw):
return self.body_decl.get_argument_expressions(**kw)
def declared_identifiers(self):
- return self.body_decl.argnames
+ return self.body_decl.allargnames
def undeclared_identifiers(self):
return (self.filter_args.\
@@ -524,7 +524,7 @@ def __init__(self, keyword, attributes, **kwargs):
**self.exception_kwargs)
def declared_identifiers(self):
- return self.code.declared_identifiers.union(self.body_decl.argnames)
+ return self.code.declared_identifiers.union(self.body_decl.allargnames)
def undeclared_identifiers(self):
return self.code.undeclared_identifiers.\
@@ -554,7 +554,7 @@ def __init__(self, namespace, defname, attributes, **kwargs):
**self.exception_kwargs)
def declared_identifiers(self):
- return self.code.declared_identifiers.union(self.body_decl.argnames)
+ return self.code.declared_identifiers.union(self.body_decl.allargnames)
def undeclared_identifiers(self):
return self.code.undeclared_identifiers.\
@@ -589,6 +589,6 @@ def __init__(self, keyword, attributes, **kwargs):
**self.exception_kwargs)
def declared_identifiers(self):
- return self.body_decl.argnames
+ return self.body_decl.allargnames
View
21 mako/pyparser.py
@@ -214,13 +214,25 @@ def __init__(self, listener, **exception_kwargs):
def visit_FunctionDef(self, node):
self.listener.funcname = node.name
+
argnames = [arg_id(arg) for arg in node.args.args]
if node.args.vararg:
argnames.append(arg_stringname(node.args.vararg))
+
+ if compat.py2k:
+ # kw-only args don't exist in Python 2
+ kwargnames = []
+ else:
+ kwargnames = [arg_id(arg) for arg in node.args.kwonlyargs]
if node.args.kwarg:
- argnames.append(arg_stringname(node.args.kwarg))
+ kwargnames.append(arg_stringname(node.args.kwarg))
self.listener.argnames = argnames
self.listener.defaults = node.args.defaults # ast
+ self.listener.kwargnames = kwargnames
+ if compat.py2k:
+ self.listener.kwdefaults = []
+ else:
+ self.listener.kwdefaults = node.args.kw_defaults
self.listener.varargs = node.args.vararg
self.listener.kwargs = node.args.kwarg
@@ -367,8 +379,13 @@ def __init__(self, listener, **exception_kwargs):
def visitFunction(self, node, *args):
self.listener.funcname = node.name
- self.listener.argnames = node.argnames
+ self.listener.argnames = list(node.argnames)
+ if node.kwargs:
+ self.listener.kwargnames = [self.listener.argnames.pop()]
+ else:
+ self.listener.kwargnames = []
self.listener.defaults = node.defaults
+ self.listener.kwdefaults = []
self.listener.varargs = node.varargs
self.listener.kwargs = node.kwargs
View
3  test/__init__.py
@@ -96,6 +96,9 @@ def maybe(*args, **kw):
return function_named(maybe, fn_name)
return decorate
+def requires_python_3(fn):
+ return skip_if(lambda: not py3k, "Requires Python 3.xx")(fn)
+
def requires_python_2(fn):
return skip_if(lambda: py3k, "Requires Python 2.xx")(fn)
View
19 test/test_ast.py
@@ -1,7 +1,7 @@
import unittest
from mako import ast, exceptions, pyparser, util, compat
-from test import eq_, requires_python_2
+from test import eq_, requires_python_2, requires_python_3
exception_kwargs = {
'source': '',
@@ -263,6 +263,8 @@ def test_function_decl(self):
eq_(parsed.funcname, 'foo')
eq_(parsed.argnames,
['a', 'b', 'c', 'd', 'e', 'f'])
+ eq_(parsed.kwargnames,
+ [])
def test_function_decl_2(self):
"""test getting the arguments from a function"""
@@ -270,7 +272,20 @@ def test_function_decl_2(self):
parsed = ast.FunctionDecl(code, **exception_kwargs)
eq_(parsed.funcname, 'foo')
eq_(parsed.argnames,
- ['a', 'b', 'c', 'args', 'kwargs'])
+ ['a', 'b', 'c', 'args'])
+ eq_(parsed.kwargnames,
+ ['kwargs'])
+
+ @requires_python_3
+ def test_function_decl_3(self):
+ """test getting the arguments from a fancy py3k function"""
+ code = "def foo(a, b, *c, d, e, **f):pass"
+ parsed = ast.FunctionDecl(code, **exception_kwargs)
+ eq_(parsed.funcname, 'foo')
+ eq_(parsed.argnames,
+ ['a', 'b', 'c'])
+ eq_(parsed.kwargnames,
+ ['d', 'e', 'f'])
def test_expr_generate(self):
"""test the round trip of expressions to AST back to python source"""
View
15 test/test_def.py
@@ -2,7 +2,7 @@
from mako import lookup
from test import TemplateTest
from test.util import flatten_result, result_lines
-from test import eq_, assert_raises
+from test import eq_, assert_raises, requires_python_3
from mako import compat
class DefTest(TemplateTest):
@@ -45,6 +45,19 @@ def test_def_args(self):
"""hello mycomp hi, 5, 6"""
)
+ @requires_python_3
+ def test_def_py3k_args(self):
+ template = Template("""
+ <%def name="kwonly(one, two, *three, four, five=5, **six)">
+ look at all these args: ${one} ${two} ${three[0]} ${four} ${five} ${six['seven']}
+ </%def>
+
+ ${kwonly('one', 'two', 'three', four='four', seven='seven')}""")
+ eq_(
+ template.render(one=1, two=2, three=(3,), six=6).strip(),
+ """look at all these args: one two three four 5 seven"""
+ )
+
def test_inter_def(self):
"""test defs calling each other"""
template = Template("""
Something went wrong with that request. Please try again.