Asset pipeline

[skade]

There. I said the ugly word. I think its both the most hated and most powerful feature of Rails, so we should tread carefully here. I heard from multiple people that the absence of sprockets and ability to use something else is one of the reasons they really enjoy Padrino.

So, I'd personally prefer an approach similar to what we did before:

padrino gen project my_project --pipeline $my_favourite_asset_pipeline

Why do we need some kind of asset pipeline? In short:

• setting up LESS/SCSS, uglify and all the rest every time you start a project is a waste of time and frustrating
• it solves a very real problem with multiple apps that gets only harder with Apps in gems #1007: apps may want a different set of javascripts/css etc. compiled together while still reusing some parts of other packages.
• properly done, they can also act as a test harness.

In our proper fashion, implementing one API and then trying to shoehorn another component in is not a way to go, so I went ahead and picked up some projects that can qualify as an "asset pipeline":

• sprockets: the classic, as used in rails. A bit unwieldy to configure, but it has a track record. Written in Ruby.
• rake-pipeline with rake-pipeline-web-filters: used by e.g. ember.js. Much simpler, basically a set of filters that transform directory trees. Written in Ruby.
• grunt with bower: A task set to compile and manage assets, especially js libraries. A huge advantage is the integration in bower for library management. Written in Javascript.
• sinatra-assetpack: Easy to integrate, dependencies differ from whats used. Written in Ruby.

There is a very good article about implementing the first two into sinatra and their respective benefits here. AFAIK, @dariocravero has been working on a grunt integration.

There are, in my opinion, multiple things that are crucial to a successful implementation:

• the adoption of a default source file organization that works with all three options
• a generic interface for the most default options (compile...)
• a way for padrino to provide flexible path management, e.g. loading assets from gems. This should mostly be a problem with Grunt.

Another problem, in my opinion, is ease of development. Considering how much pain therubyracer, therubyrhino and other things have given me (as a developer and as someone that has to assist developers, especially on windows), I'd vastly prefer to take the high road and require node.js to be installed for all options for the first implementation. Its also a much better environment to actually get started with JS testing. This might be added setup effort for every developer, but provides a lot of smooth sailing from there on, especially as it ensures that you don't accidentally load a javascript runtime in you development system :). @ujifgc wouldn't want that.

So much for the general writeup. Any comments?

P.S.: I had a look at a few packaging engines using python. But 3 runtime systems would have been a bit inconvenient...

[ujifgc]

What do you think of sinatra-assetpack? I use it in production and development for serving and compressing js and css.

[skade]

Sounds reasonable as well.

[nesquena]

Only for sprockets but https://github.com/Cirex/padrino-assets or https://github.com/nightsailer/padrino-sprockets looked cool.

I think having access to an asset pipeline is helpful especially if tied in well with multiple apps. FWIW I have sometimes used https://github.com/railsjedi/jammit-sinatra (jammit] as well which provides some but not all of the features of a full pipeline.

Personally on a lot of my simple projects I think the asset pipeline is actually too complicated so I wouldn't really want it enabled by default but maybe:

1. Maintain a padrino-assets official gem that people can plugin to padrino via the use of a plugin recipe (i.e padrino plugin asset-pipeline).

2. As you mentioned on project generation, having a flag (by default off probably) that let's you pick a pipeline: padrino gen project my_project --pipeline grunt

Either these would support sprockets, grunt, etc via a common shared API for common tasks (i.e compiling). Curious to hear what the rest of you guys think.

[dariocravero]

I agree with @nesquena up to some point that the asset pipeline isn't really necessary for all projects -think APIs.

However, it should be recommended for users starting any app that will have a browsable frontend of some kind. IMHO, most -if not all- apps working in production mode should be built thinking of performance and resource optimisation from day one. That's where a mix of a good set of guidelines explaining what the concept is, why you should care about it, etc., and providing different alternatives as @skade suggests fits in really well. That's why I'd be inclined with going with 2), i.e., padrino gen project my_project --pipeline grunt, disabled by default.

We should also think of a CLI-enabled way for changing this choice, however that's topic for some other issue, perhaps #677?

Coming back @skade's implementation proposal, I reckon it will provide a great way of separating concerns and letting us (or others) add whichever pipeline they consider necessary in the future.

For the default organisation, I think we could name it client or assets and it should live inside app. I like client or something more descriptive better because real apps are living on the frontend these days!... We could also go app/js, app/css, app/images. I'd be inclined to hide them from public because most times you want to serve customised versions of them -think fingerprinted assets to wisely use caching or gziped versions when it makes sense. Thoughts?

I've had experiences with jammit-sinatra, sprockets, grunt and sinatra-assetpack. As @nesquena says, jammit-sinatra is good but it doesn't cover the whole lot. However, it might still be ok with some users' needs. sprockets and sinatra-assetpack are very good too, they both provide a great code organisation and amazing support for a bunch of coding alternatives. Grunt itself is actually more like rake, it's API isn't fully stable yet but that doesn't prevent people from adopting it because it's really good!! I've been mainly using grunt-bbb as a boilerplate and came up with a Gruntfile.js for a project I was working on. It did all I wanted from an asset pipeline but uploading resources to a CDN. I did that through a custom rake task because it was the easiest at the time. Anyway, coming back to the implementation with Grunt, I was serving the files in dev mode using this implementation. Everything in production mode was referencing a CDN asset with its fingerprinted version being set by the Gruntfile and the asset on the HAML templates were using the helpers provided on that implementation -it could definitely be better.

Bower is amazing! And the good news is that it isn't mutually exclusive with the pipelines -think about it as bundler for the web. Perhaps we can look at implementing a Ruby client to leverage dependencies on JS and make it available to the whole community standalone?

For the API, I think it might be best if we analyse what steps should be present in an asset pipeline. Of course, we should take into account what the current implementations do to achieve this. I reckon this will put us in a better position to be able to tackle the problem in a more comprehensive way, still allowing for libraries that don't support the whole lot to be used, should the user want.

We also have to be conscious of people using a CSS preprocessor and the pipeline together since they may step over since most pipeline managers already handle them. E.g., what we do with the generator if both are chosen?

On the CDN bit, if people don't need/want to use them, although it's a bit out of the scope of the framework, it might be a good idea to provide guidelines to configure webservers (apache, nginx, etc.) to serve this more efficiently.

Another thing we should be aware of, and that might be a big plus, is the ability to allow templates to be reused on both: backend and frontend. I don't know how much work is involved in doing that but it could be a great feature. E.e.g, think about SEO and, say, Backbone enabled apps that do loads of things on the browser. Yes, I know, search engines will become more intelligent at some point but they aren't there yet. At the end of the day, templates should be compiled before they're used on the browser to maximise performance. In any way, this might be out of the scope so let me know if you want me to elaborate on the topic.

Well, this was quite a post hehe... Thanks @skade for bringing it up, it was about time we discussed it!.. :D

[dariocravero]

For reference when we tackle this: sinatra-asset-pipeline

[Ortuna]

@padrino/core-members does padrino-pipeline fit the bill for this? It's still in an early state, but since it's relying on very well done gems it can do a lot.

I can keep it as a separate gem, but I wouldn't mind seeing it incorporated into Padrino once it's ready. I'd like some help on it to build up the API and add more pipelines.

😸

[DAddYE]

I agree with u @Ortuna

[ghost]

I forked padrino-assets several months ago for use in my own projects.

Its main issue was a lack of support for multiple mounted apps, which essentially calls for namespacing of asset files and directories, to prevent overwriting. I added in this functionality, but never got around to adding tests for it, but it worked in the padrino project I was using it for.

I considered unofficially taking over future development of the gem, since it seems like the creator is no longer around.

[DAddYE]

@ELeo have you tried to write him an email and ask to add you to the repo and rubygems?

[skade]

@ELeo Is there anything in padrino core we could adopt to drive the development of similar things?

I am slowly moving to the opinion that we should provide necessary hooks and infos (where the files are, where they should be), but maybe not ship one within padrino itself.

[ghost]

@skade it's hard for me to say since I only added to the gem rather than built it from the ground up. From what I saw, at least in this case, there wasn't much that I found myself in need of beyond Sinatra's extension documentation. I could probably answer this better if I had worked on the gem more recently; I'm glancing at the code and don't see anything jumping out at me.

@DAddYE haven't tried that yet, I'll see if he responds.

[nesquena]

I am slowly moving to the opinion that we should provide necessary hooks and infos (where the files are, where they should be), but maybe not ship one within padrino itself.

+1 I think I am of a similar opinion. Provide necessary extensibility without baking the pipeline into the core.

[Ortuna]

@nesquena @skade, could you give an example of those hooks? Working with padrino-pipeline I've come to realize that the underlying systems are different(namely packages(asset_pack) vs directives(sprockets)). So a unified config gets a bit nasty. I can't envision a different API than what I have in padrino-pipline. Perhaps we can merge it in with core? I'd be more than willing to work on this(merge or new API)!

[dariocravero]

+1 on @skade's point as @nesquena suggests.

@Ortuna I reckon that the first step would be to compare the existing solutions. This will give us both: what's common and what needs to be "polyfilled". @padrino/core-members what do you think?

[ujifgc]

I like my assets to be served with rack middleware. I started to write this little thing to do it and planning to add compiling and compressing without requiring gem-bundled dependencies like sprockets or sinatra-assetpack.

[wakatara]

I have to admit to not being a big fan of the Rails asset pipeline. I've used sinatra-assetpack to good effect in a lot of sinatra apps and was actually trying to figure out how to add it into my current padrino app (which is mostly /admin at the mo).

I do think @ujifgc's rack middleware looks really interesting, though not sure how far it needs to go to be on a feature comparison level (ie. compiling, compressing) with say sinatra-assetpack or jammit/sprockets/bower etc etc...

[dariocravero]

For the curious, here's an implementation of an asset pipeline/process convertor using make. Interesting read :)

[ujifgc]

Yes, very interesting implementation of combining files.

[joefiorini]

Since people seem interested in this, I wanted to throw out an alternative to the Asset Pipeline. I have a gem called Half Pipe that uses a Rails generator to setup some convenient grunt tasks. There is no reason this couldn't work just as well with Padrino, the only changes needed would be a new generator template and firing up a padrino server in the background instead of a rails server. I'm happy to accept pull requests or have some discussions around what it would take to implement this. Would anyone be interested in helping out?

[dariocravero]

Hey @joefiorini, thanks for taking your time to comment here :). I like the approach you're following, I also believe that frontend tooling is better off being managed by anything in the JS world since it's built for it. Asset management with tools like bower is something everybody should do these days if they want their project dependencies to grow out nicely :). Have you explored the possibility of integrating yeoman directly instead of building your own grunt workflow? That should bring it a lot of goodies and release you from many responsibilities too :). Thoughts?

[joefiorini]

@dariocravero Interesting points, I'd like to be able to open this conversation up to other half-pipe users. would you mind opening up an issue on half pipe with your thoughts on this?

[fgarcia]

Has anyone already considered the functionality conflict of an asset pipeline with the stylesheet component? If a pipeline sees app.css.scss it will be expected to deal with Sass, although IMHO, most people using Sass also want Compass, and in that case, there will be some type of relationship with the pipeline anyways.

People are mentioning Bower, and I believe it already makes a much better job than the script component... but that requires a npm install instead a plain gem... maybe that is another story

Regarding Grunt it overlaps much more with existing functionality, like code reloading, stylesheet auto-generation, etc. Basically it should have some connection with padrino start

[ahacking]

IMO it is important to separate client and server development. The tools can be completely different and even your teams may be different. I personally don't want to share my server code base with client developers who I may hire for specific features so I keep the code bases separate, but I do want to be able to hook client build and test with my server development testing and deployment workflow.

The way I have been doing client side development is with brunch and I highly recommend its relative ease of setup and compilation speed compared to anything else. It is node based. Having your server configured to support CORS makes separation of client and API very straight forward. Being able to develop client independent of the back end is very important. I am using a non standard rails backend (sequel and my own high performance json serialization gem) for performance and I am looking to Sinatra/padrino/grape as a possible request latency reduction. If I couldn't develop the client standalone and it had to be infected with padrino concerns I simply would not use or consider migrating to padrino.

Whilst I am now doing client side developmemt using the brunch toolchain I took quite a path to get there, using yeoman, grunt, and browerisfy in anger and can truely say it was an aweful experience compared to brunch.

I have also taken a good look at rake-pipeline and iridium but its nowhere close to the workflow and capabilities of brunch and no direct support for bower.

All these pipelines work very differently, some do advanced live reload, some require manual refresh, some support source maps whilst others require development assets to NOT be combined (and experience long browser reload times as the number of files increase), some support environment specific vendor sources/variants whilst others don't.

Whilst I use bower I find it also has a lot of short comings and I've also used asset gems in the past too which are largely sprockets based but I don't want to go back to sprockets. At some point I may end up using or building something else that better serves my needs.

The point is client/asset pipelines will change and evolve and everyone has their favorite tools that solve their problems.

Padrino should avoid getting tied up in this mess as much as possible but still support developer workflows.

What I want is just some hooks for development setup (a command or task that does bundle install/update plus an 'install/update' for whatever client/asset pipeline toolchain I use)

I want a task/command for building assets for test and production environements and hooking running of client tests.

So really just some hooks to support the tasks developers have to do so we can all use our favorite tools whilst still supporting separate client and server development.

[jikkujose]

Can someone tell me the current state of the development? Is this being built? Or any pointers to an implementation that can be tried now?

[dariocravero]

@jikkujose no, we aren't building this yet.

I can personally recommend you to leave the task to the JS land and use gulp to manage your assets transformations (concat, minify, etc.) and Bower for your front-end dependencies if any. JavaScript tooling is proving to be very good at it and its development is much more active than anything its counterparts in Ruby.

Here's a very basic gist to get you started... It's meant to process your JS and CSS and push them to a CDN (in this case, Rackspace but you can either choose not to use it or to use something else like an rsync alternative that's commented out and KeyCDN uses).
Assets will be compiled to public/dist and to run it you have to provide the version to compile to as follows: gulp assets --version 0.1, our build system isn't based in md5 hashes but versions because it helps us target users better but that's your choice :). We've made a simple wrapper around our gemified app's version to be able to run gulp assets --version ./version`` instead.

#!/usr/bin/env ruby
require_relative 'lib/app/version'
print App::VERSION

# lib/app/version.rb
module App
  VERSION = '0.0.1'
end

This process isn't fully streamlined and that's why it needs to be revised and hasn't been proposed for Padrino. There's also grunt-semantic-release that goes a bit further and has a more holistic approach to front-end application management but I haven't got around using it yet but it looks very promising.

[jikkujose]

Hey @dariocravero thanks for such a detailed write up. Apologies for late reply. I actually read quite a bit about various build systems and it seems we can achieve all this through Guard and its plugins? Being Ruby, I am very comfortable using and configuring it. Any reason it may fall short?

[Rendez]

I've been pretty close to this discussion since I started using Padrino. There are really good points here, I personally was looking for the next:

• Use Bower since it's the best and easiest package manager for the web.
• Sprockets is a solid solution for precompiling assets, it's straightforward and avoids having to precompile assets with grunt.

Using Grunt/Gulp to run precompiling tasks requires to either, a) having to include compiled files in the repo, b) Needing node.js in the deployment machine.

This is why after finding rails-assets, and consequently sinatra-asset-pipeline, I decided to give the latter a go with the intent of making it work for sub-apps.

The daemon of rails-assets searches for packages in Bower and wraps them in a GEM, so they can be included like you normally would with sprockets' directives.

I've got a working version of this adaptation in production, and I'm particularly very happy with the solution: it's clean, has sprockets, uses Bower, and doesn't need Node.js.

I've pushed the repository live today, the gem is obviously called padrino-asset-pipeline. Feel free to try it out and report back issues, etc. If we find it fit, I propose we consider this as a valid alternative to Padrino-assets.

A last point: I will write tests and soon make it into a GEM, depending on your approval and review process.

Cheers!

[jikkujose]

Seems I am failing to follow much of the steps to get this working; seems the discussion assumes Rails experience or a previous exposure with asset compilation. Is there any non rails pointers anyone can suggest to get started?

[Rendez]

@jikkujose I've added more detail to the readme of my repo, so you can see how to use it.
@skade Reading your initial comment above, I think you might be interested in the approach I took.

Again, it needs feedback, but I think the solution covers the basic problem from most of the angles.

Feedback welcome :)

[jikkujose]

Wow, that was fast. Will check it out.

[dnesteryuk]

You can get Asset pipline (with sprockets) in your Padrino app without any additional gems. I described steps which you need to do here. I hope it will help you.

[namusyaka]

@dnesteryuk Nice!