Skip to content
This repository

Stream operation for pystache #108

gonvaled opened this Issue · 5 comments

2 participants

gonvaled Chris Jerdonek

This is what I usually do to render a file with pystache:

txt = open(filename, 'r').read()
print pystache.render(txt, context).encode('utf-8')
("context" is my mapping)

But now I have to render very big templates, which causes memory problems. It would be great if pystache could operate on streams.

Chris Jerdonek

Thanks for the report. For future reference, using memory mapped files might be another possible approach to dealing with large files, but I'm not sure.


That is a good idea. For the input data this would probably work, but (and I am talking here without knowing the details of the pystache implementation) I guess this would not completely solve the issue, since the output data would still be in memory


And it seems not even input mmap is supported:

import mmap
import pystache
tmp_file = '/tmp/tmp_file'
template_text = 'Hello {{name}}'
d = {'name' : 'Steve'}
f = open(tmp_file, 'w')
f = open(tmp_file, 'r+b')
mymmap = mmap.mmap(f.fileno(), 0)
print pystache.render(mymmap, d)
» python 
Traceback (most recent call last):
  File "", line 11, in 
    print pystache.render(mymmap, d)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/", line 8, in render
    return Template(template, context).render()
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/", line 176, in render
    result = self._render_tags(template)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/", line 119, in _render_tags
    template = template.replace(tag, replacement)
AttributeError: 'mmap.mmap' object has no attribute 'replace'
Chris Jerdonek

Just to clarify, I didn't mean to suggest that mmap would work without making additional changes. :) Work would need to be done there, too (but probably not as much). And yes, you are right. If support for mmap were added to the API, we would not want those methods to return the output as a string, etc.

Chris Jerdonek

Could you describe what your large files "look" like? For example, are they deeply nested with sections, is it long strings of plain text in between a few scattered tags, or is it equally dense with both plain text and tags? Also, are there section tags that contain huge strings, etc?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.