RFC: Dropping "compiled." subpackage prefix for compiled templates

[pfalcon]

The idea was to organize files more cleanly, but it only complicates matters, because it's unclear where that subpackage should reside: as the top-level subpackage, or in a particular template subdir, etc.

Dropping it, there should not be such ambiguities.

Another reason is to have cleaner mapping from template names to module names and to avoid fragmenting memory by constantly concatenating "compiled." in loader. So, with the new scheme, if there's a template

templates/group/my_temp_late

it will map to Python module name

templates.group.my_temp_late

To extend this a bit further, now "native" mode for naming templates are defined, where template name can't contain a dot (thus, no file extensions), ditto for directory components of a template. And component separator used then is ".". In other words, the native template name matches Python module name for the template. By following native naming, memory needed to look up template is minimized.

Then there's "compatibility" mode, where template names may contain dot/have extension, and they're addressed with directory separators. But looking up such templates may require extra memory.

[pfalcon]

@cefn : Comments welcome. I would set this as a prerequisite for implementing #7

[cefn]

Update: OK, I misread the suggestion. The below is what you're already doing in code which I didn't originally adopt - having just cherry picked the utemplate Compiler and rolled my own strategies elsewhere.

Definitely having compiled. as a prefix can be dropped.

I had anticipated managing my own mapping between templateIds and loaded modules, and that's what ended up happening. In my case, for example there is a (storyUid,nodeUid,fieldName) tuple which dictates uniqueness through concatenation with underscores, and a prefix to allow numbers to be used in the module names (otherwise this generates invalid module names). Probably there should be logic in my code to force escaping of other characters in uids as people could adopt other illegal symbols very easily.

I hadn't understood the indirection through compiled.Loader vs source.Loader until I inspected it a bit further. The idea of switching the Loader allows resources to be saved which I haven't worried about as much, since the boilerplate.Compiler definition is in a frozen module and hitting the cache just bypasses any use of them.

I could have made this "{}/{}/{}".format(storyUid,nodeUid,fieldName), had it create a few extra directories and adopted your framework more extensively, as well as getting the potential benefit of not loading any source logic when templates were known to be pre-compiled.

Incidentally I ended up authoring a top-level routine which force-compiled all the templates, and also console-dumped the module name qstrings meaning this overhead could also be pushed into frozen bytecode.

Is there a reason not to use templates/group/my_temp_late.py ? I've been building logic which have python code for the generator factory serialised to .py on the fly, then reloaded, with the subsequent loads hitting the cached .py. This then means I can heave the whole python path contents over into modules and include them as frozen bytecode and the load routine is seamless either way.

See loadTemplateGeneratorFactory in https://github.com/cefn/avatap/blob/master/python/boilerplate.py

I know it can be more efficient (write directly to files rather than strings to files) but this happens on the desktop in python3 right now anyway so I haven't bothered with that efficiency gain, given the stress I have with efficiency actually on the board.

[cefn]

Something which is emerging from my experiments is the need to
a) shorten template names (these are qstrings and eventually overload the irom space)
b) lazy-compile templates (if you've changed the source, this would ideally be known dynamically, else a special update-templates routine needs to be triggered somehow)
c) de-duplicate templates (if your source is identical to another, this should be served by a single loaded generatorFactory module)

One way to address this which is looking promising for my use case is to name the modules t_{hash}.py where the hash is calculated from the text of the source using uhashlib.sha256 and shortening the result with h.digest().hex()[:8]. Not sure yet if micropython's support for hashing, byte manipulation will make this a pain to deliver on ESP8266.

Using the hash for identification, whenever the source changes, it 'points' to a different cached module, which can then be lazily created only if that source string was never compiled into the templates directory before.

This might be incompatible with your use case, where the templateId may be intended to explicitly eliminate the need to access the source strings. However, I thought I'd share it as it's likely to be baked into my own use of the framework, and presumably if the source strings are in a filesystem, it's fairly cheap and low RAM to traverse them for hash calculations anyway if only when originally loading. This suggests the need for a dict() to index templateIds to templateModule names, though to save re-running the hash all the time.

Not a feature request, yet, just an idea to throw into the mix.

[pfalcon]

Update: OK, I misread the suggestion. The below is what you're already doing in code which I didn't originally adopt - having just cherry picked the utemplate Compiler and rolled my own strategies elsewhere.

Thanks. Note that wearing my utemplate's maintainer hat, I'm interested in maintaining and developing it. I appreciate your giving it a try, putting to use, and communicating bugs/issues, but for any use going beyond what's supported out of the box with utemplate, you generally would need to figure out a way to do it. I'm just trying to keep you in loop on the changes which may affect you.

[pfalcon]

This was implemented now.