Stream operation for pystache #108

Open
gonvaled opened this Issue Apr 20, 2012 · 6 comments

Comments

Projects
None yet
3 participants
@gonvaled

This is what I usually do to render a file with pystache:

txt = open(filename, 'r').read()
print pystache.render(txt, context).encode('utf-8')
("context" is my mapping)

But now I have to render very big templates, which causes memory problems. It would be great if pystache could operate on streams.

@cjerdonek

This comment has been minimized.

Show comment Hide comment
@cjerdonek

cjerdonek Apr 23, 2012

Collaborator

Thanks for the report. For future reference, using memory mapped files might be another possible approach to dealing with large files, but I'm not sure.

Collaborator

cjerdonek commented Apr 23, 2012

Thanks for the report. For future reference, using memory mapped files might be another possible approach to dealing with large files, but I'm not sure.

@gonvaled

This comment has been minimized.

Show comment Hide comment
@gonvaled

gonvaled Apr 27, 2012

That is a good idea. For the input data this would probably work, but (and I am talking here without knowing the details of the pystache implementation) I guess this would not completely solve the issue, since the output data would still be in memory

That is a good idea. For the input data this would probably work, but (and I am talking here without knowing the details of the pystache implementation) I guess this would not completely solve the issue, since the output data would still be in memory

@gonvaled

This comment has been minimized.

Show comment Hide comment
@gonvaled

gonvaled Apr 27, 2012

And it seems not even input mmap is supported:

import mmap
import pystache
tmp_file = '/tmp/tmp_file'
template_text = 'Hello {{name}}'
d = {'name' : 'Steve'}
f = open(tmp_file, 'w')
f.write(template_text)
f.flush()
f = open(tmp_file, 'r+b')
mymmap = mmap.mmap(f.fileno(), 0)
print pystache.render(mymmap, d)
mymmap.close()
» python test_pystache_mmap.py 
Traceback (most recent call last):
  File "test_pystache_mmap.py", line 11, in 
    print pystache.render(mymmap, d)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/__init__.py", line 8, in render
    return Template(template, context).render()
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 176, in render
    result = self._render_tags(template)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 119, in _render_tags
    template = template.replace(tag, replacement)
AttributeError: 'mmap.mmap' object has no attribute 'replace'

And it seems not even input mmap is supported:

import mmap
import pystache
tmp_file = '/tmp/tmp_file'
template_text = 'Hello {{name}}'
d = {'name' : 'Steve'}
f = open(tmp_file, 'w')
f.write(template_text)
f.flush()
f = open(tmp_file, 'r+b')
mymmap = mmap.mmap(f.fileno(), 0)
print pystache.render(mymmap, d)
mymmap.close()
» python test_pystache_mmap.py 
Traceback (most recent call last):
  File "test_pystache_mmap.py", line 11, in 
    print pystache.render(mymmap, d)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/__init__.py", line 8, in render
    return Template(template, context).render()
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 176, in render
    result = self._render_tags(template)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 119, in _render_tags
    template = template.replace(tag, replacement)
AttributeError: 'mmap.mmap' object has no attribute 'replace'
@cjerdonek

This comment has been minimized.

Show comment Hide comment
@cjerdonek

cjerdonek Apr 27, 2012

Collaborator

Just to clarify, I didn't mean to suggest that mmap would work without making additional changes. :) Work would need to be done there, too (but probably not as much). And yes, you are right. If support for mmap were added to the API, we would not want those methods to return the output as a string, etc.

Collaborator

cjerdonek commented Apr 27, 2012

Just to clarify, I didn't mean to suggest that mmap would work without making additional changes. :) Work would need to be done there, too (but probably not as much). And yes, you are right. If support for mmap were added to the API, we would not want those methods to return the output as a string, etc.

@cjerdonek

This comment has been minimized.

Show comment Hide comment
@cjerdonek

cjerdonek May 5, 2012

Collaborator

Could you describe what your large files "look" like? For example, are they deeply nested with sections, is it long strings of plain text in between a few scattered tags, or is it equally dense with both plain text and tags? Also, are there section tags that contain huge strings, etc?

Collaborator

cjerdonek commented May 5, 2012

Could you describe what your large files "look" like? For example, are they deeply nested with sections, is it long strings of plain text in between a few scattered tags, or is it equally dense with both plain text and tags? Also, are there section tags that contain huge strings, etc?

@markfink

This comment has been minimized.

Show comment Hide comment
@markfink

markfink Aug 21, 2015

1++
I would love to see the streaming, too. I create large output files similar to sql format with multiple entities. Basically the rendering processes one list with 500k entries. Currently it fits all in memory but the rendering takes ~5 Minutes. I could split the file up to multiple files but this makes the handling much more complex.

1++
I would love to see the streaming, too. I create large output files similar to sql format with multiple entities. Basically the rendering processes one list with 500k entries. Currently it fits all in memory but the rendering takes ~5 Minutes. I could split the file up to multiple files but this makes the handling much more complex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment