Skip to content

Loading…

Support template precompilation using Y.Template #11

Closed
rgrove opened this Issue · 40 comments

10 participants

@rgrove

It'd be super awesome if shifter supported precompiling templates using Y.Template. Here's how I imagine it working.

Let's say I'm building a new widget and I want to keep its templates out of my JS during development. I'd start with a directory structure like this:

  • component/
    • js/
      • component.js
      • component-templates.js
    • templates/
      • one.micro
      • two.micro
    • build.json

The files one.micro and two.micro are arbitrary templates. I can name them whatever I want. The .micro extension indicates that they should be compiled with Y.Template.Micro. A .handlebars extension would indicate they should be compiled with Y.Template.Handlebars.

For example, one.micro might look like this:

<% if (this.visible) { %>
    Hello world!
<% } %>

In component-templates.js, I'd create a namespace to hold my precompiled templates, and insert placeholders where the precompiled template source should go after the file is built.

var Templates = Y.namespace('Component').Templates = new Y.Template(Y.Template.Micro);

Templates.one = Templates.revive("@template:one.micro@");
Templates.two = Templates.revive("@template:two.micro@");

In my build.json file, I'd create the following config:

{
    "name": "component",

    "builds": {
        "component": {
            "jsfiles": ["component.js"],

            "config": {
                "requires": ["component-templates"]
            }
        },

        "component-templates": {
            "jsfiles": ["component-templates.js"],

            "config": {
                "requires": ["template"]
            },

            "templates": true
        }
    }
}

The "templates": true setting tells shifter to look for template files in the ./templates directory with an extension matching any supported template engine (.micro for Micro, .handlebars for Handlebars).

Shifter would then scan all the JS files for component-templates looking for template placeholders, extract the names of the referenced templates, precompile the templates, and replace the placeholders with the precompiled templates.

For example, "@template:one.micro@" (including the quotes, which are there to prevent lint errors on the source file) would be replaced in the built file with the precompiled template code for ./templates/one.micro. The source JS would remain untouched.

If there's a template file in ./templates that is never referenced in a placeholder, shifter would skip precompiling that template for the sake of efficiency.

@rgrove

Addendum: relative paths should be supported in template placeholders. So "@template:foo/bar/baz.micro@" should refer to ./templates/foo/bar/baz.micro. Shifter should ignore .. in template paths to prevent traversal above the component's root directory.

@ericf
YUI Library member

Seems like a great proposal to me. I like that component authors are still in control of their "component-templates" module, this seems important to support referencing a template at some alias for back-compat, etc.

@davglass
YUI Library member

This seems pretty simple to do, however Shifter has no requirement on YUI at the moment. So I have to figure out a good way to get to the compilers without requiring all of YUI just to do it..

@davglass davglass was assigned
@evocateur

I managed to avoid pulling in all of yui in mustache-wax by just pulling in handlebars directly. Perhaps Template.Micro could have a YUI-independent core (via gear-lib?)for its precompilation bits?

@rgrove

Addendum #2: Shifter should also support an optional templateOptions property, which may be set to a hash of options to be passed to Y.Template#precompile().

@rgrove

@evocateur: Not quite so easy in this case. We'd also have to create a YUI-independent Y.Escape.html(), and trying to do this just so shifter won't need to require YUI seems unnecessarily complicated.

@davglass
YUI Library member

I'm kind of stuck in a catch 22 at the moment, since it's not in master yet, I can't npm install it, so I can't use it.. Ugh!

@rgrove

Yeah, I know. This change can wait until Y.Template is in master, no rush.

@evocateur

This would seem to be a deficiency in Y.Template's compile method, then (and its internal dependency on this.revive(), regardless of whether or not we're precompiling or not).

Contrast with Handlebars, whose precompiled output is a raw function that is passed directly to Handlebars#template(), which provides the escaper and whatnot.

I'll comment on the lines I'm talking about in yui/yui3#230

@davglass
YUI Library member

This just gives me time to create the packer (custom YUI package) I've been talking about :)

@rgrove

@evocateur: Let me clarify. The code generated for a precompiled template has zero YUI dependencies. You could use that code anywhere, even without YUI. But generating that precompiled code does require YUI, because Y.Template.Micro is a YUI module. YUI isn't strictly required to revive the precompiled code, but shifter won't be doing the reviving, so that's a moot point.

Strictly speaking, Y.Escape.html() isn't required for the precompilation step, but it is a dependency of template-micro because it's needed for the revive step (it's also needed in order to revive precompiled Handlebars templates).

The precompilation step in Y.Template.Micro could be broken out into code that has zero YUI dependencies, but this would make maintenance a pain, and it doesn't seem like it's all that necessary.

@evocateur

@davglass A custom packer would be awesome!

@rgrove I was writing on the other thread: https://github.com/yui/yui3/pull/230/files#r1490895

Without a viable CLI utility for precompiling Y.Template.Micro templates, Y.Handlebars will remain the only viable option for folks like me who want this and also want to avoid bloating shifter with the entire library just to use one function.

@evocateur

Ack, I shouldn't have forked the thread. :/

@rgrove

@evocateur: Nothing's stopping anyone from writing a viable CLI utility for precompiling micro-templates, nor would such a CLI utility need to load the "entire library" just to use Y.Template.Micro.precompile(). It would need to load template-micro and its dependencies (which are escape and yui-base). That's not what I'd consider bloated.

If you're referring to the fact that YUI modules aren't available as individual npm modules, that's a completely separate issue, and not one that should dictate whether or not it's okay for Node.js-friendly YUI components to depend on YUI.

@evocateur

Anyway, @rgrove, you're correct, my comment on the Y.Template.Micro pull request was orthogonal to this discussion, strictly speaking.

That being said, is it an express goal of shifter to avoid pulling in yui as a dependency? My perception is that there is a fair amount of YUI-specific assumptions already in shifter, and depending on yui would only really be a threat in the Inception, "yo dawg"-recursion sense.

Edit: I need to learn how to type faster. :P

@evocateur

Nothing's stopping anyone from writing a viable CLI utility for precompiling micro-templates [...]

I thought that's what this thread was about? Unless there's some other interface for shifter that I'm not aware of?

I think the yui npm package is awesome, but struggle with the lack of parsimony in using it here. If it's perfectly reasonable to write an independent CLI micro-template precompiler (without depending on the entire yui package?), shouldn't it be possible to use that independent utility in shifter?

@rgrove

@evocateur: Clearly there's a misunderstanding here. Let me break it down:

  • Y.Template.Micro will be part of YUI.
  • Y.Template.Micro will not be a standalone JavaScript library, because it's part of YUI.
  • If you want to use Y.Template.Micro, you need YUI.
  • If you don't want to use YUI, don't use Y.Template.Micro.

Could I (or someone else) write a Template.Micro that doesn't use or require YUI? Yes. Am I interested in doing it? No, because I need it in YUI.

Currently, using a YUI module in a Node.js app means you must have an npm dependency on all of YUI, but it does not mean you must load all of YUI into memory. The lack of npm granularity is unfortunate, and I'd love to see individual npm modules for individual YUI modules, but that's way outside the scope of this ticket.

There are many problems in the world. I'm trying to fix one. I can't fix all the others first, but I can fix this one.

@evocateur

If you don't want to use YUI, don't use Y.Template.Micro.

I'm not questioning the need, design, or usefulness of Y.Template.Micro. I think it's swell. I think precompiling templates via shifter would rock. I'm not demanding you fix all the problems in the world. I appreciate all the work you and many others do to make YUI great.

Fundamentally, I am questioning the dependency graph of a YUI build tool that includes the built asset itself. It's not even an issue of implementation, because obviously the yui tree in node_modules wouldn't be touched by the operation of shifter. It's a code smell in a build tool, not an existential attack on the Y.Template module itself (and certainly not it's dashingly handsome author).

I am also willing to accept the distinct possibility that I am a bikeshedding worrywart who needs to get some actual work done today. :)

@rgrove

In the compiler world, a common benchmark of a compiler's completeness is whether it can be used to compile itself. It seems reasonable to me that a build tool could be used to build itself (or some component of itself).

But we definitely agree on one thing: I am dashingly handsome. ;)

@ziemkowski

Wouldn't it be more appropriate for shifter to keep it light and just do something similar to the CLI replacers, by just creating a replacement string composed of the file contents with single quotes escaped and new lines removed? That way it's totally independent of what template engine or framework somebody wishes to use.

So the Github App example might do something like this:

template: Y.Handlebars.compile('@template:user.handlebars@'),

Is the compile process for Micro/Handlebars really heavy enough to justify the complexity (and presumably larger file size) of precompiling them at build?

If that is the case, maybe this should be two separate but related features. 1) The simple stringifying of templates files when template is truthy. 2) The template build.json property accepting a compiler name as a value to customize the replace value of those files using the specified compiler. Seems more pluggable that way and having framework dependencies on those compilers would be less of an issue in that case.

@evocateur

If all user agents were Chrome, I wouldn't bat an eye and compiling the templates anew every page load. As we live in the real world (hello, mobile!), precompilation remains a vitally important step in the use of client-side templates in JS. The time spent compiling in a single-threaded environment dwarfs any minor increase in absolute file size.

Myself, I'm envisioning some sort of custom script that I can run postbuild to accomplish my goals. I will probably be borrowing the templates subdirectory pattern, I think that's great. grifter is going to be sooo cool...

@rgrove

The compilation step for Handlebars is very expensive, even in Chrome. Template.Micro's compilation step is much faster than Handlebars, but is still significantly slower than just rendering a precompiled template.

@ericf
YUI Library member

Independent of the speed of compilation, Handlebars' compiler is 9.5KB minified and gzipped. Saving those bytes becomes significant.

@benjamind

Has any progress been made on this?

I need a similar setup and have tried post-build scripts to modify the output files of the shifter build process. The problem is that replacing the compiled templates into the output files isn't ideal in that the '-min.js' and '-coverage.js' versions of my modules aren't really good candidates for a simple replace. While I could minify the precompiled template code before injection into the '-min.js' file, doing something similar for the coverage file is extremely hard.

Ideally I need to hook into the shifter build process at the stage where it has read the content of my files, but before its done any processing on them. That way I could replace in the compiled templates before the minification and coverage steps are performed.

For now I'm going to simply copy my source tree before compilation, replace in the templates, then run my shifter build. Not ideal.

Any thoughts?

@davglass
YUI Library member

I have made 0 progress on this as the new templates just landed in YUI proper.

The YUI Community still needs to determine how this should work, I'm not going to create a solution to something with no data ;)

@benjamind

Fair point. :-)

Thankfully the copy process is pretty fast so I'm not in too much of a hurry to figure this one out.

@rgrove

@davglass My original description in this issue was pretty detailed and specific about how I think things should work. What's missing?

@benjamind

I guess its the details of how to do this without having to include all of YUI into shifter?

I'd be tempted to keep shifter clean of this, and just add more external exec hooks at various stages of shifters process. For instance a pre-compress-exec which calls out to my external script passing the contents of the buffer through stdin, and reads back the the result from my script as the new buffer would let me hook in the Y.Template functionality.

This would keep the core of shifter clean of YUI, but make it pretty easy to add this template handling functionality.

@ericf
YUI Library member

Let's roundtable this. Or if our standard time doesn't work we can schedule a G+ Hangout.

@davglass
YUI Library member

Yes, I'm not including all of YUI in shifter just to compile templates.

@ericf
YUI Library member

@davglass If that's the case, then we'll need some way for a Shifter task to communicate with another process that does require YUI.

@rgrove

If shifter can't compile YUI micro-templates, then I guess shifter can't compile templates. That's too bad.

I understand not wanting to include all of YUI in shifter just to get Y.Template.Micro. But this speaks to a bigger problem: why should it ever be necessary to include all of YUI in a Node.js project? This makes YUI very unattractive in Node.

Every YUI module should be an npm module. Instead of requiring a single yui package that contains all of YUI, I should be able to require yui-template-micro and load only the modules needed by YUI's template-micro module.

@davglass
YUI Library member

You beat me to my next comment, the packaging issue is on my plate after the new year starts :P

@benjamind

@ericf agreed on the interprocess communication. Would make shifter very powerful if we can plugin additional functionality at different stages of the build process.

@rgrove and @davglass I'm personally not too bothered with having to include all of YUI on my node implementations as i'm largely using it server side and as such all those files aren't necessarily much of a hindrance. Its not like I have to re-download all of YUI every time I want to use it on the server.

@demonkoryu

I'd love to see progress here.
I'm building a PHP/Assetic-based ZF2 asset module that can currently consume shifter build.json-files and output YUI modules + loader data. I'm looking for a build.json precedent for handlebars template precompilation.

@earnubs

I wrote a grunt task to precompile using Y.Template, doesn't answer the issue but might be useful to someone:
https://github.com/earnubs/grunt-yui-template-compile

@VincentVToscano

Any movement here?

@caridy
YUI Library member

this tool is now in maintenance mode, no new features will be added.

@caridy caridy closed this
@VincentVToscano

Thanks for the update and passing along the article as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.