Skip to content
This repository
Browse code

[svn] added c implementation of the jinja context class.

--HG--
branch : trunk
  • Loading branch information...
commit ee2c18ef16ceb0f100c5c8211002cc9c309f2944 1 parent 6d0b195
Armin Ronacher authored
6 CHANGES
@@ -22,7 +22,8 @@ Version 1.1
22 22
23 23 - some small bugfixes.
24 24
25   -- improved security system regarding function calls.
  25 +- improved security system regarding function calls and variable
  26 + assignment in for loops.
26 27
27 28 - added `lipsum` function to generate random text.
28 29
@@ -64,6 +65,9 @@ Version 1.1
64 65 - fixed a bug in the parser that didn't unescape keyword arguments. (thanks
65 66 to Alexey Melchakov for reporting)
66 67
  68 +- You can now use the environment to just tokenize a template. This can
  69 + be useful for syntax highlighting or other purposes.
  70 +
67 71
68 72 Version 1.0
69 73 -----------
8 THANKS
... ... @@ -0,0 +1,8 @@
  1 +Thanks To
  2 +=========
  3 +
  4 +All the people listed here helped improving Jinja a lot, provided
  5 +patches, helped working out solutions etc. Thanks to all of you!
  6 +
  7 +- Axel Böhm
  8 +- Alexey Melchakov
7 TODO
@@ -2,7 +2,8 @@
2 2 TODO List for Jinja
3 3 ===================
4 4
5   -- Requirements in Jinja (blocks and set directives) outside of renderable
6   - blocks should become part of the module not the generate function.
7   -
8 5 - Improve the context lookup (maybe with an optional C extension)
  6 +
  7 +- `include` and `extends` should work with dynamic data too. In order to
  8 + support this the blocks should be stored as importable functions in the
  9 + generated source.
173 docs/src/contextenv.txt
... ... @@ -1,173 +0,0 @@
1   -=======================
2   -Context and Environment
3   -=======================
4   -
5   -The two central objects in Jinja are the `Environment` and `Context`. Both
6   -are designed to be subclassed by applications if they need to extend Jinja.
7   -
8   -Environment
9   -===========
10   -
11   -The initialization parameters are already covered in the `Quickstart`_ thus
12   -not repeated here.
13   -
14   -But beside those configurable instance variables there are some functions used
15   -in the template evaluation code you may want to override:
16   -
17   -**def** `to_unicode` *(self, value)*:
18   -
19   - Called to convert variables to unicode. Per default this checks if the
20   - value is already unicode. If not it's converted to unicode using the
21   - charset defined on the environment.
22   -
23   - Also `None` is converted into an empty string per default.
24   -
25   -**def** `get_translator` *(self, context)*:
26   -
27   - Return the translator used for i18n. A translator is an object that
28   - provides the two functions ``gettext(string)`` and
29   - ``ngettext(singular, plural, n)``. Both of those functions have to
30   - behave like the `ugettext` and `nugettext` functions described in the
31   - python `gettext documentation`_.
32   -
33   - If you don't provide a translator a default one is used to switch
34   - between singular and plural forms.
35   -
36   - Have a look at the `i18n`_ section for more information.
37   -
38   -**def** `get_translations` *(self, name)*:
39   -
40   - Get the translations for the template `name`. Only works if a loader
41   - is present. See the `i18n`_ section for more details.
42   -
43   -**def** `get_translations_for_string` *(self, string)*:
44   -
45   - Get the translations for the string `string`. This works also if no
46   - loader is present and can be used to lookup translation strings from
47   - templates that are loaded from dynamic resources like databases.
48   -
49   -**def** `apply_filters` *(self, value, context, filters)*:
50   -
51   - Now this function is a bit tricky and you usually don't have to override
52   - it. It's used to apply filters on a value. The Jinja expression
53   - ``{{ foo|escape|replace('a', 'b') }}`` calls the function with the
54   - value of `foo` as first parameter, the current context as second and
55   - a list of filters as third. The list looks like this:
56   -
57   - .. sourcecode:: python
58   -
59   - [('escape', ()), ('replace', (u'a', u'b'))]
60   -
61   - As you can see the filter `escape` is called without arguments whereas
62   - `replace` is called with the two literal strings ``a`` and ``b``, both
63   - unicode. The filters for the names are stored on ``self.filters`` in a
64   - dict. Missing filters should raise a `FilterNotFound` exception.
65   -
66   - **Warning** this is a Jinja internal method. The actual implementation
67   - and function signature might change.
68   -
69   -**def** `perform_test` *(self, context, testname, args, value, invert)*:
70   -
71   - Like `apply_filters` you usually don't override this one. It's the
72   - callback function for tests (``foo is bar`` / ``foo is not bar``).
73   -
74   - The first parameter is the current contex, the second the name of
75   - the test to perform. the third a tuple of arguments, the fourth is
76   - the value to test. The last one is `True` if the test was performed
77   - with the `is not` operator, `False` if with the `is` operator.
78   -
79   - Missing tests should raise a `TestNotFound` exception.
80   -
81   - **Warning** this is a Jinja internal method. The actual implementation
82   - and function signature might change.
83   -
84   -**def** `get_attribute` *(self, obj, attribute)*:
85   -
86   - Get `attribute` from the object provided. The default implementation
87   - performs security tests.
88   -
89   - **Warning** this is a Jinja internal method. The actual implementation
90   - and function signature might change.
91   -
92   -**def** `get_attributes` *(self, obj, attributes)*:
93   -
94   - Get some attributes from the object. If `attributes` is an empty
95   - sequence the object itself is returned unchanged.
96   -
97   -**def** `call_function` *(self, f, context, args, kwargs, dyn_args, dyn_kwargs)*:
98   -
99   - Call a function `f` with the arguments `args`, `kwargs`, `dyn_args` and
100   - `dyn_kwargs` where `args` is a tuple and `kwargs` a dict. If `dyn_args`
101   - is not `None` you have to add it to the arguments, if `dyn_kwargs` is
102   - not `None` you have to update the `kwargs` with it.
103   -
104   - The default implementation performs some security checks.
105   -
106   - **Warning** this is a Jinja internal method. The actual implementation
107   - and function signature might change.
108   -
109   -**def** `call_function_simple` *(self, f, context)*:
110   -
111   - Like `call_function` but without arguments.
112   -
113   - **Warning** this is a Jinja internal method. The actual implementation
114   - and function signature might change.
115   -
116   -**def** `finish_var` *(self, value, ctx)*:
117   -
118   - Postprocess a variable before it's sent to the template.
119   -
120   - **Warning** this is a Jinja internal method. The actual implementation
121   - and function signature might change.
122   -
123   -.. admonition:: Note
124   -
125   - The Enviornment class is defined in `jinja.environment.Environment`
126   - but imported into the `jinja` package because it's often used.
127   -
128   -Context
129   -=======
130   -
131   -Jinja wraps the variables passed to the template in a special class called a
132   -context. This context supports variables on multiple layers and lazy (deferred)
133   -objects. Often your application has a request object, database connection
134   -object or something similar you want to access in filters, functions etc.
135   -
136   -Beacause of that you can easily subclass a context to add additional variables
137   -or to change the way it behaves.
138   -
139   -**def** `pop` *(self)*:
140   -
141   - Pop the outermost layer and return it.
142   -
143   -**def** `push` *(self, data=None)*:
144   -
145   - Push a dict to the stack or an empty layer.
146   -
147   - Has to return the pushed object.
148   -
149   -**def** `to_dict` *(self)*:
150   -
151   - Flatten the context and convert it into a dict.
152   -
153   -**def** `__getitem__` *(self, name)*:
154   -
155   - Resolve an item. Per default this also resolves `Deferred` objects.
156   -
157   -**def** `__setitem__` *(self, name, value)*:
158   -
159   - Set an item in the outermost layer.
160   -
161   -**def** `__delitem__` *(self, name)*:
162   -
163   - Delete an item in the outermost layer. Do not raise exceptions if
164   - the value does not exist.
165   -
166   -**def** `__contains__` *(self, name)*:
167   -
168   - Return `True` if `name` exists in the context.
169   -
170   -
171   -.. _i18n: i18n.txt
172   -.. _Quickstart: devintro.txt
173   -.. _gettext documentation: http://docs.python.org/lib/module-gettext.html
2  docs/src/designerdoc.txt
@@ -784,7 +784,7 @@ The following keywords exist and cannot be used as identifiers:
784 784 `and`, `block`, `cycle`, `elif`, `else`, `endblock`, `endfilter`,
785 785 `endfor`, `endif`, `endmacro`, `endraw`, `endtrans`, `extends`, `filter`,
786 786 `for`, `if`, `in`, `include`, `is`, `macro`, `not`, `or`, `pluralize`,
787   - `raw`, `recursive`, `set`, `trans`
  787 + `print`, `raw`, `recursive`, `set`, `trans`
788 788
789 789 If you want to use such a name you have to prefix or suffix it or use
790 790 alternative names:
5 docs/src/devintro.txt
@@ -57,6 +57,11 @@ addition to the initialization values:
57 57 syntax tree. This tree of nodes is used by the
58 58 `translators`_ to convert the template into
59 59 executable source- or bytecode.
  60 +``lex(source, filename)`` Tokenize the given sourcecode and return a
  61 + generator of tuples in the form
  62 + ``(lineno, token, value)``. The filename is just
  63 + used in the exceptions raised.
  64 + **New in Jinja 1.1**
60 65 ``from_string(source)`` Load and parse a template source and translate it
61 66 into eval-able Python code. This code is wrapped
62 67 within a `Template` class that allows you to
49 docs/src/index.txt
@@ -6,41 +6,54 @@ Welcome in the Jinja documentation.
6 6
7 7 - `Installing Jinja <installation.txt>`_
8 8
9   -- Application Developer Documentation:
  9 +- **Application Developer Documentation**:
10 10
11   - - `Quickstart <devintro.txt>`_
  11 + - `Quickstart <devintro.txt>`_ - getting started with Jinja
12 12
13   - - `Template Loaders <loaders.txt>`_
  13 + - `Template Loaders <loaders.txt>`_ - documentation for the different
  14 + loader types and how to write custom ones.
14 15
15   - - `Filter Functions <filters.txt>`_
  16 + - `Filter Functions <filters.txt>`_ - information about how to write
  17 + custom filter functions.
16 18
17   - - `Test Functions <tests.txt>`_
  19 + - `Test Functions <tests.txt>`_ - information about how to write
  20 + custom test functions.
18 21
19   - - `Global Objects <objects.txt>`_
  22 + - `Global Objects <objects.txt>`_ - information about the special global
  23 + namespace in Jinja templates.
20 24
21   - - `Streaming Interface <streaming.txt>`_
  25 + - `Streaming Interface <streaming.txt>`_ - using Jinja for big templates
  26 + by streaming the output.
22 27
23   - - `Context and Environment <contextenv.txt>`_
  28 + - `Internationalization <i18n.txt>`_ - how to internationalize applications
  29 + using Jinja templates.
24 30
25   - - `Translators <translators.txt>`_
  31 + - `Alternative Syntax <altsyntax.txt>`_ - changing the default Jinja
  32 + block / variable / comment delimiters.
26 33
27   - - `Framework Integration <frameworks.txt>`_
  34 + - `API Documentation <api.txt>`_ - API documentation for public Jinja
  35 + objects like `Environment`.
28 36
29   - - `Debugging Support <debugging.txt>`_
  37 + - `Translators <translators.txt>`_ - explanation about the Jinja template
  38 + translation interface.
30 39
31   - - `Internationalization <i18n.txt>`_
  40 + - `Framework Integration <frameworks.txt>`_ - integrating Jinja into
  41 + python frameworks.
32 42
33   - - `Alternative Syntax <altsyntax.txt>`_
  43 + - `Debugging Support <debugging.txt>`_ - debugging Jinja templates.
34 44
35   - - `Developer Recipies <devrecipies.txt>`_
  45 + - `Developer Recipies <devrecipies.txt>`_ - tips and tricks for application
  46 + developers.
36 47
37   -- Template Designer Documentation:
  48 +- **Template Designer Documentation**:
38 49
39   - - `Syntax Reference <designerdoc.txt>`_
  50 + - `Syntax Reference <designerdoc.txt>`_ - the designer documentation. Put
  51 + this under your pillow.
40 52
41   - - `Differences To Django <fromdjango.txt>`_
  53 + - `Differences To Django <fromdjango.txt>`_ - coming from django? Then this
  54 + document is for you.
42 55
43   - - `Designer Recipies <recipies.txt>`_
  56 + - `Designer Recipies <recipies.txt>`_ - various tips and tricks for designers.
44 57
45 58 - `Changelog <changelog.txt>`_
46 59
2  docs/src/objects.txt
@@ -74,6 +74,8 @@ The function is always called with the same arguments. The first one is the
74 74 current environment, the second the context and the third is the name of the
75 75 variable. In this example ``recent_comments``.
76 76
  77 +The value is cached until rendering/streaming finished.
  78 +
77 79 Unsafe Methods / Attributes
78 80 ===========================
79 81
13 docs/src/recipies.txt
@@ -169,3 +169,16 @@ Or if you use the `capture` filter in `clean` mode:
169 169 <div class="head">{{ title }}</div>
170 170 </body>
171 171 </html>
  172 +
  173 +
  174 +Vim Syntax Highlighting
  175 +=======================
  176 +
  177 +Because of the similar syntax to django you can use the django highlighting
  178 +plugin for jinja too. There is however a Jinja syntax highlighting plugin
  179 +too which supports all of the syntax elements.
  180 +
  181 +You can download it from the vim webpage: `jinja.vim`_
  182 +
  183 +
  184 +.. _jinja.vim: http://www.vim.org/scripts/script.php?script_id=1856
6 docs/src/translators.txt
@@ -5,6 +5,6 @@ Translators
5 5 Jinja translates the template sourcecode into executable python code behind
6 6 the secenes. This is done by the python translator which is currently the
7 7 only shipped translator. Because the interface isn't stable it's also not
8   -recommended yet to write other translators. However for the next Jinja version
9   -a JavaScript translator is planned which allows you to translate Jinja
10   -templates into executable JavaScript code.
  8 +recommended yet to write other translators. However for one of the next Jinja
  9 +versions a JavaScript translator is planned which allows you to translate
  10 +Jinja templates into executable JavaScript code.
91 jinja/_native.py
... ... @@ -0,0 +1,91 @@
  1 +# -*- coding: utf-8 -*-
  2 +"""
  3 + jinja._native
  4 + ~~~~~~~~~~~~~
  5 +
  6 + This module implements the native base classes in case of not
  7 + having a jinja with the _speedups module compiled.
  8 +
  9 + :copyright: 2007 by Armin Ronacher.
  10 + :license: BSD, see LICENSE for more details.
  11 +"""
  12 +from jinja.datastructure import Deferred, Undefined
  13 +
  14 +
  15 +class BaseContext(object):
  16 +
  17 + def __init__(self, silent, globals, initial):
  18 + self.silent = silent
  19 + self.current = current = {}
  20 + self.stack = [globals, initial, current]
  21 + self.globals = globals
  22 + self.initial = initial
  23 +
  24 + def pop(self):
  25 + """
  26 + Pop the last layer from the stack and return it.
  27 + """
  28 + rv = self.stack.pop()
  29 + self.current = self.stack[-1]
  30 + return rv
  31 +
  32 + def push(self, data=None):
  33 + """
  34 + Push a new dict or empty layer to the stack and return that layer
  35 + """
  36 + data = data or {}
  37 + self.stack.append(data)
  38 + self.current = self.stack[-1]
  39 + return data
  40 +
  41 + def __getitem__(self, name):
  42 + """
  43 + Resolve one item. Restrict the access to internal variables
  44 + such as ``'::cycle1'``. Resolve deferreds.
  45 + """
  46 + if not name.startswith('::'):
  47 + # because the stack is usually quite small we better
  48 + # use [::-1] which is faster than reversed() in such
  49 + # a situation.
  50 + for d in self.stack[::-1]:
  51 + if name in d:
  52 + rv = d[name]
  53 + if rv.__class__ is Deferred:
  54 + rv = rv(self, name)
  55 + # never touch the globals!
  56 + if d is self.globals:
  57 + self.initial[name] = rv
  58 + else:
  59 + d[name] = rv
  60 + return rv
  61 + if self.silent:
  62 + return Undefined
  63 + raise TemplateRuntimeError('%r is not defined' % name)
  64 +
  65 + def __setitem__(self, name, value):
  66 + """
  67 + Set a variable in the outermost layer.
  68 + """
  69 + self.current[name] = value
  70 +
  71 + def __delitem__(self, name):
  72 + """
  73 + Delete an variable in the outermost layer.
  74 + """
  75 + if name in self.current:
  76 + del self.current[name]
  77 +
  78 + def __contains__(self, name):
  79 + """
  80 + Check if the context contains a given variable.
  81 + """
  82 + for layer in self.stack:
  83 + if name in layer:
  84 + return True
  85 + return False
  86 +
  87 + def __len__(self):
  88 + """
  89 + Size of the stack.
  90 + """
  91 + return len(self.stack)
392 jinja/_speedups.c
... ... @@ -0,0 +1,392 @@
  1 +/**
  2 + * jinja._speedups
  3 + * ~~~~~~~~~~~~~~~
  4 + *
  5 + * This module implements the BaseContext, a c implementation of the
  6 + * Context baseclass. If this extension is not compiled the datastructure
  7 + * module implements a class in python.
  8 + *
  9 + * :copyright: 2007 by Armin Ronacher.
  10 + * :license: BSD, see LICENSE for more details.
  11 + */
  12 +
  13 +#include <Python.h>
  14 +#include <structmember.h>
  15 +
  16 +static PyObject *Undefined, *TemplateRuntimeError;
  17 +static PyTypeObject *DeferredType;
  18 +
  19 +struct StackLayer {
  20 + PyObject *dict; /* current value, a dict */
  21 + struct StackLayer *prev; /* lower struct layer or NULL */
  22 +};
  23 +
  24 +typedef struct {
  25 + PyObject_HEAD
  26 + struct StackLayer *globals; /* the dict for the globals */
  27 + struct StackLayer *initial; /* initial values */
  28 + struct StackLayer *current; /* current values */
  29 + long stacksize; /* current size of the stack */
  30 + int silent; /* boolean value for silent failure */
  31 +} BaseContext;
  32 +
  33 +static int
  34 +init_constants(void)
  35 +{
  36 + PyObject *datastructure = PyImport_ImportModule("jinja.datastructure");
  37 + if (!datastructure)
  38 + return 0;
  39 + PyObject *exceptions = PyImport_ImportModule("jinja.exceptions");
  40 + if (!exceptions) {
  41 + Py_DECREF(datastructure);
  42 + return 0;
  43 + }
  44 + Undefined = PyObject_GetAttrString(datastructure, "Undefined");
  45 + PyObject *deferred = PyObject_GetAttrString(datastructure, "Deferred");
  46 + DeferredType = deferred->ob_type;
  47 + TemplateRuntimeError = PyObject_GetAttrString(exceptions, "TemplateRuntimeError");
  48 + Py_DECREF(datastructure);
  49 + Py_DECREF(exceptions);
  50 + return 1;
  51 +}
  52 +
  53 +static void
  54 +BaseContext_dealloc(BaseContext *self)
  55 +{
  56 + struct StackLayer *current = self->current, *tmp;
  57 + while (current) {
  58 + tmp = current;
  59 + Py_XDECREF(current->dict);
  60 + current->dict = NULL;
  61 + current = tmp->prev;
  62 + PyMem_Free(tmp);
  63 + }
  64 + self->ob_type->tp_free((PyObject*)self);
  65 +}
  66 +
  67 +static int
  68 +BaseContext_init(BaseContext *self, PyObject *args, PyObject *kwds)
  69 +{
  70 + PyObject *silent = NULL, *globals = NULL, *initial = NULL;
  71 +
  72 + static char *kwlist[] = {"silent", "globals", "initial", NULL};
  73 + if (!PyArg_ParseTupleAndKeywords(args, kwds, "OOO", kwlist,
  74 + &silent, &globals, &initial))
  75 + return -1;
  76 + if (!PyDict_Check(globals) || !PyDict_Check(initial)) {
  77 + PyErr_SetString(PyExc_TypeError, "stack layers must be a dicts.");
  78 + return -1;
  79 + }
  80 +
  81 + self->silent = PyObject_IsTrue(silent);
  82 +
  83 + self->globals = PyMem_Malloc(sizeof(struct StackLayer));
  84 + self->globals->dict = globals;
  85 + Py_INCREF(globals);
  86 + self->globals->prev = NULL;
  87 +
  88 + self->initial = PyMem_Malloc(sizeof(struct StackLayer));
  89 + self->initial->dict = initial;
  90 + Py_INCREF(initial);
  91 + self->initial->prev = self->globals;
  92 +
  93 + self->current = PyMem_Malloc(sizeof(struct StackLayer));
  94 + self->current->dict = PyDict_New();
  95 + if (!self->current->dict)
  96 + return -1;
  97 + Py_INCREF(self->current->dict);
  98 + self->current->prev = self->initial;
  99 +
  100 + self->stacksize = 3;
  101 + return 0;
  102 +}
  103 +
  104 +static PyObject*
  105 +BaseContext_pop(BaseContext *self)
  106 +{
  107 + if (self->stacksize <= 3) {
  108 + PyErr_SetString(PyExc_IndexError, "stack too small.");
  109 + return NULL;
  110 + }
  111 + PyObject *result = self->current->dict;
  112 + struct StackLayer *tmp = self->current;
  113 + self->current = tmp->prev;
  114 + PyMem_Free(tmp);
  115 + self->stacksize--;
  116 + return result;
  117 +}
  118 +
  119 +static PyObject*
  120 +BaseContext_push(BaseContext *self, PyObject *args)
  121 +{
  122 + PyObject *value = NULL;
  123 + if (!PyArg_ParseTuple(args, "|O:push", &value))
  124 + return NULL;
  125 + if (!value) {
  126 + value = PyDict_New();
  127 + if (!value)
  128 + return NULL;
  129 + }
  130 + else if (!PyDict_Check(value)) {
  131 + PyErr_SetString(PyExc_TypeError, "dict required.");
  132 + return NULL;
  133 + }
  134 + else
  135 + Py_INCREF(value);
  136 + struct StackLayer *new = malloc(sizeof(struct StackLayer));
  137 + new->dict = value;
  138 + new->prev = self->current;
  139 + self->current = new;
  140 + self->stacksize++;
  141 + Py_INCREF(value);
  142 + return value;
  143 +}
  144 +
  145 +static PyObject*
  146 +BaseContext_getstack(BaseContext *self, void *closure)
  147 +{
  148 + PyObject *result = PyList_New(self->stacksize);
  149 + if (!result)
  150 + return NULL;
  151 + struct StackLayer *current = self->current;
  152 + int idx = 0;
  153 + while (current) {
  154 + PyList_SetItem(result, idx++, current->dict);
  155 + Py_INCREF(current->dict);
  156 + current = current->prev;
  157 + }
  158 + PyList_Reverse(result);
  159 + return result;
  160 +}
  161 +
  162 +static PyObject*
  163 +BaseContext_getcurrent(BaseContext *self, void *closure)
  164 +{
  165 + Py_INCREF(self->current->dict);
  166 + return self->current->dict;
  167 +}
  168 +
  169 +static PyObject*
  170 +BaseContext_getinitial(BaseContext *self, void *closure)
  171 +{
  172 + Py_INCREF(self->initial->dict);
  173 + return self->initial->dict;
  174 +}
  175 +
  176 +static PyObject*
  177 +BaseContext_getglobals(BaseContext *self, void *closure)
  178 +{
  179 + Py_INCREF(self->globals->dict);
  180 + return self->globals->dict;
  181 +}
  182 +
  183 +static int
  184 +BaseContext_readonly(BaseContext *self, PyObject *value, void *closure)
  185 +{
  186 + PyErr_SetString(PyExc_AttributeError, "can't set attribute");
  187 + return -1;
  188 +}
  189 +
  190 +static PyObject*
  191 +BaseContext_getitem(BaseContext *self, PyObject *item)
  192 +{
  193 + if (!PyString_Check(item)) {
  194 + Py_INCREF(Py_False);
  195 + return Py_False;
  196 + }
  197 +
  198 + /* disallow access to internal jinja values */
  199 + char *name = PyString_AS_STRING(item);
  200 + if (strlen(name) >= 2 && name[0] == ':' && name[1] == ':') {
  201 + Py_INCREF(Py_False);
  202 + return Py_False;
  203 + }
  204 +
  205 + PyObject *result;
  206 + struct StackLayer *current = self->current;
  207 + while (current) {
  208 + result = PyDict_GetItemString(current->dict, name);
  209 + if (!result) {
  210 + current = current->prev;
  211 + continue;
  212 + }
  213 + Py_INCREF(result);
  214 + if (PyObject_TypeCheck(result, DeferredType)) {
  215 + PyObject *args = PyTuple_New(2);
  216 + if (!args || !PyTuple_SetItem(args, 0, (PyObject*)self) ||
  217 + !PyTuple_SetItem(args, 1, item))
  218 + return NULL;
  219 +
  220 + PyObject *resolved = PyObject_CallObject(result, args);
  221 + if (!resolved)
  222 + return NULL;
  223 +
  224 + /* never touch the globals */
  225 + Py_DECREF(result);
  226 + Py_INCREF(resolved);
  227 + PyObject *namespace;
  228 + if (current == self->globals)
  229 + namespace = self->initial->dict;
  230 + else
  231 + namespace = current->dict;
  232 + PyDict_SetItemString(namespace, name, resolved);
  233 + return resolved;
  234 + }
  235 + return result;
  236 + }
  237 +
  238 + if (self->silent) {
  239 + Py_INCREF(Undefined);
  240 + return Undefined;
  241 + }
  242 + PyErr_Format(TemplateRuntimeError, "'%s' is not defined", name);
  243 + return NULL;
  244 +}
  245 +
  246 +static int
  247 +BaseContext_contains(BaseContext *self, PyObject *item)
  248 +{
  249 + if (!PyString_Check(item))
  250 + return 0;
  251 +
  252 + char *name = PyString_AS_STRING(item);
  253 + if (strlen(name) >= 2 && name[0] == ':' && name[1] == ':')
  254 + return 0;
  255 +
  256 + struct StackLayer *current = self->current;
  257 + while (current) {
  258 + if (!PyMapping_HasKeyString(current->dict, name)) {
  259 + current = current->prev;
  260 + continue;
  261 + }
  262 + return 1;
  263 + }
  264 +
  265 + return 0;
  266 +}
  267 +
  268 +static int
  269 +BaseContext_setitem(BaseContext *self, PyObject *item, PyObject *value)
  270 +{
  271 + char *name = PyString_AS_STRING(item);
  272 + if (!value)
  273 + return PyDict_DelItemString(self->current->dict, name);
  274 + return PyDict_SetItemString(self->current->dict, name, value);
  275 +}
  276 +
  277 +static PyObject*
  278 +BaseContext_length(BaseContext *self)
  279 +{
  280 + return PyInt_FromLong(self->stacksize);
  281 +}
  282 +
  283 +static PyGetSetDef BaseContext_getsetters[] = {
  284 + {"stack", (getter)BaseContext_getstack, (setter)BaseContext_readonly,
  285 + "a read only copy of the internal stack", NULL},
  286 + {"current", (getter)BaseContext_getcurrent, (setter)BaseContext_readonly,
  287 + "reference to the current layer on the stack", NULL},
  288 + {"initial", (getter)BaseContext_getinitial, (setter)BaseContext_readonly,
  289 + "reference to the initial layer on the stack", NULL},
  290 + {"globals", (getter)BaseContext_getglobals, (setter)BaseContext_readonly,
  291 + "reference to the global layer on the stack", NULL},
  292 + {NULL} /* Sentinel */
  293 +};
  294 +
  295 +static PyMemberDef BaseContext_members[] = {
  296 + {NULL} /* Sentinel */
  297 +};
  298 +
  299 +static PyMethodDef BaseContext_methods[] = {
  300 + {"pop", (PyCFunction)BaseContext_pop, METH_NOARGS,
  301 + "Pop the highest layer from the stack"},
  302 + {"push", (PyCFunction)BaseContext_push, METH_VARARGS,
  303 + "Push one layer to the stack"},
  304 + {NULL} /* Sentinel */
  305 +};
  306 +
  307 +static PySequenceMethods BaseContext_as_sequence[] = {
  308 + 0, /* sq_length */
  309 + 0, /* sq_concat */
  310 + 0, /* sq_repeat */
  311 + 0, /* sq_item */
  312 + 0, /* sq_slice */
  313 + 0, /* sq_ass_item */
  314 + 0, /* sq_ass_slice */
  315 + BaseContext_contains, /* sq_contains */
  316 + 0, /* sq_inplace_concat */
  317 + 0 /* sq_inplace_repeat */
  318 +};
  319 +
  320 +static PyMappingMethods BaseContext_as_mapping[] = {
  321 + (lenfunc)BaseContext_length,
  322 + (binaryfunc)BaseContext_getitem,
  323 + (objobjargproc)BaseContext_setitem
  324 +};
  325 +
  326 +static PyTypeObject BaseContextType = {
  327 + PyObject_HEAD_INIT(NULL)
  328 + 0, /* ob_size */
  329 + "jinja._speedups.BaseContext", /* tp_name */
  330 + sizeof(BaseContext), /* tp_basicsize */
  331 + 0, /* tp_itemsize */
  332 + (destructor)BaseContext_dealloc,/* tp_dealloc */
  333 + 0, /* tp_print */
  334 + 0, /* tp_getattr */
  335 + 0, /* tp_setattr */
  336 + 0, /* tp_compare */
  337 + 0, /* tp_repr */
  338 + 0, /* tp_as_number */
  339 + &BaseContext_as_sequence, /* tp_as_sequence */
  340 + &BaseContext_as_mapping, /* tp_as_mapping */
  341 + 0, /* tp_hash */
  342 + 0, /* tp_call */
  343 + 0, /* tp_str */
  344 + 0, /* tp_getattro */
  345 + 0, /* tp_setattro */
  346 + 0, /* tp_as_buffer */
  347 + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
  348 + "", /* tp_doc */
  349 + 0, /* tp_traverse */
  350 + 0, /* tp_clear */
  351 + 0, /* tp_richcompare */
  352 + 0, /* tp_weaklistoffset */
  353 + 0, /* tp_iter */
  354 + 0, /* tp_iternext */
  355 + BaseContext_methods, /* tp_methods */
  356 + BaseContext_members, /* tp_members */
  357 + BaseContext_getsetters, /* tp_getset */
  358 + 0, /* tp_base */
  359 + 0, /* tp_dict */
  360 + 0, /* tp_descr_get */
  361 + 0, /* tp_descr_set */
  362 + 0, /* tp_dictoffset */
  363 + (initproc)BaseContext_init, /* tp_init */
  364 + 0, /* tp_alloc */
  365 + PyType_GenericNew /* tp_new */
  366 +};
  367 +
  368 +static PyMethodDef module_methods[] = {
  369 + {NULL, NULL, 0, NULL} /* Sentinel */
  370 +};
  371 +
  372 +#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
  373 +#define PyMODINIT_FUNC void
  374 +#endif
  375 +PyMODINIT_FUNC
  376 +init_speedups(void)
  377 +{
  378 + PyObject *module;
  379 +
  380 + if (PyType_Ready(&BaseContextType) < 0)
  381 + return;
  382 +
  383 + if (!init_constants())
  384 + return;
  385 +
  386 + module = Py_InitModule3("_speedups", module_methods, "");
  387 + if (!module)
  388 + return;
  389 +
  390 + Py_INCREF(&BaseContextType);
  391 + PyModule_AddObject(module, "BaseContext", (PyObject*)&BaseContextType);
  392 +}
111 jinja/datastructure.py
@@ -180,107 +180,58 @@ class Flush(TemplateData):
180 180 jinja_no_finalization = True
181 181
182 182
183   -class Context(object):
  183 +# import these here because those modules import Deferred and Undefined
  184 +# from this module.
  185 +try:
  186 + # try to use the c implementation of the base context if available
  187 + from jinja._speedups import BaseContext
  188 +except ImportError:
  189 + # if there is no c implementation we go with a native python one
  190 + from jinja._native import BaseContext
  191 +
  192 +
  193 +class Context(BaseContext):
184 194 """
185 195 Dict like object containing the variables for the template.
186 196 """
187 197
188 198 def __init__(self, _environment_, *args, **kwargs):
189   - self.environment = _environment_
190   - self._stack = [_environment_.globals, dict(*args, **kwargs), {}]
191   - self.globals, self.initial, self.current = self._stack
192   - self._translate_func = None
  199 + super(Context, self).__init__(_environment_.silent,
  200 + _environment_.globals,
  201 + dict(*args, **kwargs))
193 202
194   - # cache object used for filters and tests
  203 + self._translate_func = None
195 204 self.cache = {}
196   -
197   - def translate_func(self):
198   - """
199   - Return the translator for this context.
200   - """
201   - if self._translate_func is None:
202   - translator = self.environment.get_translator(self)
203   - def translate(s, p=None, n=None, r=None):
204   - if p is None:
205   - return translator.gettext(s) % (r or {})
206   - return translator.ngettext(s, p, r[n]) % (r or {})
207   - self._translate_func = translate
208   - return self._translate_func
209   - translate_func = property(translate_func, doc=translate_func.__doc__)
210   -
211   - def pop(self):
212   - """
213   - Pop the last layer from the stack and return it.
214   - """
215   - rv = self._stack.pop()
216   - self.current = self._stack[-1]
217   - return rv
218   -
219   - def push(self, data=None):
220   - """
221   - Push a new dict or empty layer to the stack and return that layer
222   - """
223   - data = data or {}
224   - self._stack.append(data)
225   - self.current = self._stack[-1]
226   - return data
  205 + self.environment = _environment_
227 206
228 207 def to_dict(self):
229 208 """
230 209 Convert the context into a dict. This skips the globals.
231 210 """
232 211 result = {}
233   - for layer in self._stack[1:]:
  212 + for layer in self.stack[1:]:
234 213 for key, value in layer.iteritems():
235 214 if key.startswith('::'):
236 215 continue
237 216 result[key] = value
238 217 return result
239 218
240   - def __getitem__(self, name):
241   - """
242   - Resolve one item. Restrict the access to internal variables
243   - such as ``'::cycle1'``. Resolve deferreds.
244   - """
245   - if not name.startswith('::'):
246   - # because the stack is usually quite small we better use [::-1]
247   - # which is faster than reversed() somehow.
248   - for d in self._stack[::-1]:
249   - if name in d:
250   - rv = d[name]
251   - if rv.__class__ is Deferred:
252   - rv = rv(self, name)
253   - # never touch the globals!
254   - if d is self.globals:
255   - self.initial[name] = rv
256   - else:
257   - d[name] = rv
258   - return rv
259   - if self.environment.silent:
260   - return Undefined
261   - raise TemplateRuntimeError('%r is not defined' % name)
262   -
263   - def __setitem__(self, name, value):
264   - """
265   - Set a variable in the outermost layer.
266   - """
267   - self.current[name] = value
268   -
269   - def __delitem__(self, name):
270   - """
271   - Delete an variable in the outermost layer.
272   - """
273   - if name in self.current:
274   - del self.current[name]
275   -
276   - def __contains__(self, name):
  219 + def translate_func(self):
277 220 """
278   - Check if the context contains a given variable.
  221 + Return a translation function for this context. It takes
  222 + 4 parameters. The singular string, the optional plural one,
  223 + the indicator number which is used to select the correct
  224 + plural form and a dict with values which should be inserted.
279 225 """
280   - for layer in self._stack:
281   - if name in layer:
282   - return True
283   - return False
  226 + if self._translate_func is None:
  227 + translator = self.environment.get_translator(self)
  228 + def translate(s, p=None, n=None, r=None):
  229 + if p is None:
  230 + return translator.gettext(s) % (r or {})
  231 + return translator.ngettext(s, p, r[n]) % (r or {})
  232 + self._translate_func = translate
  233 + return self._translate_func
  234 + translate_func = property(translate_func, doc=translate_func.__doc__)
284 235
285 236 def __repr__(self):
286 237 """
13 jinja/environment.py
@@ -167,6 +167,17 @@ def parse(self, source, filename=None):
167 167 parser = Parser(self, source, filename)
168 168 return parser.parse()
169 169
  170 + def lex(self, source, filename=None):
  171 + """
  172 + Lex the given sourcecode and return a generator that yields tokens.
  173 + The stream returned is not usable for Jinja but can be used if
  174 + Jinja templates should be processed by other tools (for example
  175 + syntax highlighting etc)
  176 +
  177 + The tuples are returned in the form ``(lineno, token, value)``.
  178 + """
  179 + return self.lexer.tokeniter(source, filename)
  180 +
170 181 def from_string(self, source):
171 182 """
172 183 Load and parse a template source and translate it into eval-able
@@ -300,7 +311,7 @@ def call_function(self, f, context, args, kwargs, dyn_args, dyn_kwargs):
300 311 """
301 312 if dyn_args is not None:
302 313 args += tuple(dyn_args)
303   - elif dyn_kwargs is not None:
  314 + if dyn_kwargs is not None:
304 315 kwargs.update(dyn_kwargs)
305 316 if getattr(f, 'jinja_unsafe_call', False) or \
306 317 getattr(f, 'alters_data', False):
4 jinja/lexer.py
@@ -51,7 +51,7 @@
51 51 operator_re = re.compile('(%s)' % '|'.join([
52 52 isinstance(x, unicode) and str(x) or re.escape(x) for x in [
53 53 # math operators
54   - '+', '-', '*', '//', '/', '%',
  54 + '+', '-', '**', '*', '//', '/', '%',
55 55 # braces and parenthesis
56 56 '[', ']', '(', ')', '{', '}',
57 57 # attribute access and comparison / logical operators
@@ -64,7 +64,7 @@
64 64 'endfilter', 'endfor', 'endif', 'endmacro', 'endraw',
65 65 'endtrans', 'extends', 'filter', 'for', 'if', 'in',
66 66 'include', 'is', 'macro', 'not', 'or', 'pluralize', 'raw',
67   - 'recursive', 'set', 'trans'])
  67 + 'recursive', 'set', 'trans', 'print', 'call', 'endcall'])
68 68
69 69
70 70 class Failure(object):
24 jinja/nodes.py
@@ -220,6 +220,26 @@ def __repr__(self):
220 220 )
221 221
222 222
  223 +class Call(Node):
  224 + """
  225 + A node that represents am extended macro call.
  226 + """
  227 +
  228 + def __init__(self, lineno, expr, body):
  229 + self.lineno = lineno
  230 + self.expr = expr
  231 + self.body = body
  232 +
  233 + def get_items(self):
  234 + return [self.expr, self.body]
  235 +
  236 + def __repr__(self):
  237 + return 'Call(%r, %r)' % (
  238 + self.expr,
  239 + self.body
  240 + )
  241 +
  242 +
223 243 class Set(Node):
224 244 """
225 245 Allow defining own variables.
@@ -322,7 +342,9 @@ def get_items(self):
322 342 return [self.template]
323 343
324 344 def __repr__(self):
325   - return 'Include(%r)' % self.template
  345 + return 'Include(%r)' % (
  346 + self.template
  347 + )
326 348
327 349
328 350 class Trans(Node):
47 jinja/parser.py
@@ -41,6 +41,7 @@
41 41 end_of_if = StateTest.expect_name('endif')
42 42 end_of_filter = StateTest.expect_name('endfilter')
43 43 end_of_macro = StateTest.expect_name('endmacro')
  44 +end_of_call = StateTest.expect_name('endcall')
44 45 end_of_block_tag = StateTest.expect_name('endblock')
45 46 end_of_trans = StateTest.expect_name('endtrans')
46 47
@@ -77,6 +78,7 @@ def __init__(self, environment, source, filename=None):
77 78 'filter': self.handle_filter_directive,
78 79 'print': self.handle_print_directive,
79 80 'macro': self.handle_macro_directive,
  81 + 'call': self.handle_call_directive,
80 82 'block': self.handle_block_directive,
81 83 'extends': self.handle_extends_directive,
82 84 'include': self.handle_include_directive,
@@ -262,6 +264,19 @@ def handle_macro_directive(self, lineno, gen):
262 264 args = None
263 265 return nodes.Macro(lineno, ast.name, args, body)
264 266
  267 + def handle_call_directive(self, lineno, gen):
  268 + """
  269 + Handle {% call foo() %}...{% endcall %}
  270 + """
  271 + expr = self.parse_python(lineno, gen, '(%s)').expr
  272 + if expr.__class__ is not ast.CallFunc:
  273 + raise TemplateSyntaxError('call requires a function or macro '
  274 + 'call as only argument.', lineno,
  275 + self.filename)
  276 + body = self.subparse(end_of_call, True)
  277 + self.close_remaining_block()
  278 + return nodes.Call(lineno, expr, body)
  279 +
265 280 def handle_block_directive(self, lineno, gen):
266 281 """
267 282 Handle block directives used for inheritance.
@@ -305,7 +320,8 @@ def handle_extends_directive(self, lineno, gen):
305 320 raise TemplateSyntaxError('extends requires a string', lineno,
306 321 self.filename)
307 322 if self.extends is not None:
308   - raise TemplateSyntaxError('extends called twice', lineno)
  323 + raise TemplateSyntaxError('extends called twice', lineno,
  324 + self.filename)
309 325 self.extends = nodes.Extends(lineno, tokens[0][2][1:-1])
310 326
311 327 def handle_include_directive(self, lineno, gen):
@@ -313,10 +329,13 @@ def handle_include_directive(self, lineno, gen):
313 329 Handle the include directive used for template inclusion.
314 330 """
315 331 tokens = list(gen)
316   - if len(tokens) != 1 or tokens[0][1] != 'string':
317   - raise TemplateSyntaxError('include requires a string', lineno,
318   - self.filename)
319   - return nodes.Include(lineno, tokens[0][2][1:-1])
  332 + # hardcoded include (faster because copied into the bytecode)
  333 + if len(tokens) == 1 and tokens[0][1] == 'string':
  334 + return nodes.Include(lineno, str(tokens[0][2][1:-1]))
  335 + raise TemplateSyntaxError('invalid syntax for include '
  336 + 'directive. Requires a hardcoded '
  337 + 'string', lineno,
  338 + self.filename)
320 339
321 340 def handle_trans_directive(self, lineno, gen):
322 341 """
@@ -338,8 +357,9 @@ def handle_trans_directive(self, lineno, gen):
338 357 try:
339 358 gen.next()
340 359 except StopIteration:
341   - #XXX: what about escapes?
342   - return nodes.Trans(lineno, data[1:-1], None,
  360 + # XXX: this looks fishy
  361 + data = data[1:-1].encode('utf-8').decode('string-escape')
  362 + return nodes.Trans(lineno, data.decode('utf-8'), None,
343 363 None, None)
344 364 raise TemplateSyntaxError('string based translations '
345 365 'require at most one argument.',
@@ -560,6 +580,7 @@ def parse(self):
560 580 'as identifier.' % node.name,
561 581 node.lineno, self.filename)
562 582 node.name = node.name[:-1]
  583 + # same for attributes
563 584 elif node.__class__ is ast.Getattr:
564 585 if not node.attrname.endswith('_'):
565 586 raise TemplateSyntaxError('illegal use of keyword %r '
@@ -567,6 +588,18 @@ def parse(self):
567 588 node.name, node.lineno,
568 589 self.filename)
569 590 node.attrname = node.attrname[:-1]
  591 + # if we have a ForLoop we ensure that nobody patches existing
  592 + # object using "for foo.bar in seq"
  593 + elif node.__class__ is nodes.ForLoop:
  594 + def check(node):
  595 + if node.__class__ not in (ast.AssName, ast.AssTuple):
  596 + raise TemplateSyntaxError('can\'t assign to '
  597 + 'expression.', node.lineno,
  598 + self.filename)
  599 + for n in node.getChildNodes():
  600 + check(n)
  601 + check(node.item)
  602 + # now set the filename and continue working on the childnodes
570 603 node.filename = self.filename
571 604 todo.extend(node.getChildNodes())
572 605 return nodes.Template(self.filename, body, self.extends)
113 jinja/translators/python.py
@@ -44,6 +44,12 @@ class GeneratorExit(Exception):
44 44 """For python2.3/python2.4 compatibility"""
45 45
46 46
  47 +try:
  48 + set
  49 +except NameError:
  50 + from sets import Set as set
  51 +
  52 +
47 53 def _to_tuple(args):
48 54 """
49 55 Return a tuple repr without nested repr.
@@ -173,6 +179,7 @@ def __init__(self, environment, node):
173 179 nodes.Cycle: self.handle_cycle,
174 180 nodes.Print: self.handle_print,
175 181 nodes.Macro: self.handle_macro,
  182 + nodes.Call: self.handle_call,
176 183 nodes.Set: self.handle_set,
177 184 nodes.Filter: self.handle_filter,
178 185 nodes.Block: self.handle_block,
@@ -386,10 +393,13 @@ def handle_template(self, node):
386 393
387 394 # bootstrapping code
388 395 lines = [
  396 + '# Essential imports\n'
389 397 'from __future__ import division\n'
390 398 'from jinja.datastructure import Undefined, LoopContext, '