Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Stream operation for pystache #108

Open
gonvaled opened this Issue · 5 comments

2 participants

@gonvaled

This is what I usually do to render a file with pystache:

txt = open(filename, 'r').read()
print pystache.render(txt, context).encode('utf-8')
("context" is my mapping)

But now I have to render very big templates, which causes memory problems. It would be great if pystache could operate on streams.

@cjerdonek
Collaborator

Thanks for the report. For future reference, using memory mapped files might be another possible approach to dealing with large files, but I'm not sure.

@gonvaled

That is a good idea. For the input data this would probably work, but (and I am talking here without knowing the details of the pystache implementation) I guess this would not completely solve the issue, since the output data would still be in memory

@gonvaled

And it seems not even input mmap is supported:

import mmap
import pystache
tmp_file = '/tmp/tmp_file'
template_text = 'Hello {{name}}'
d = {'name' : 'Steve'}
f = open(tmp_file, 'w')
f.write(template_text)
f.flush()
f = open(tmp_file, 'r+b')
mymmap = mmap.mmap(f.fileno(), 0)
print pystache.render(mymmap, d)
mymmap.close()
» python test_pystache_mmap.py 
Traceback (most recent call last):
  File "test_pystache_mmap.py", line 11, in 
    print pystache.render(mymmap, d)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/__init__.py", line 8, in render
    return Template(template, context).render()
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 176, in render
    result = self._render_tags(template)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 119, in _render_tags
    template = template.replace(tag, replacement)
AttributeError: 'mmap.mmap' object has no attribute 'replace'
@cjerdonek
Collaborator

Just to clarify, I didn't mean to suggest that mmap would work without making additional changes. :) Work would need to be done there, too (but probably not as much). And yes, you are right. If support for mmap were added to the API, we would not want those methods to return the output as a string, etc.

@cjerdonek
Collaborator

Could you describe what your large files "look" like? For example, are they deeply nested with sections, is it long strings of plain text in between a few scattered tags, or is it equally dense with both plain text and tags? Also, are there section tags that contain huge strings, etc?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.