Move our build script over to Rake

[paulirish]

I'm thinking we should move from the Ant build script to something else..

why?

• XML is gross. Therefore the build.xml is less than sexy.
• ant build files are a pain to read and learn from
• ant build files are not very easy to maintain. we have a lot of unsolved issues and such
• using java hasnt been the cross-platform magic that i was hoping for.

However, what is does is hugely useful.

We're thinking about using Sass and Compass in the build cycle in order to get @import inlining for our stylesheet optimization. This introduces a Ruby dependency. (However I will note that I have nearly successfully campaigned a YUI Compressor (which we're currently using) developer to add the @import inlining feature)

So if we need ruby in the build process.. Let's just go for it. Rakefiles are wayyy better and more readable than Ant build files.

Things to be done:

• Find experts
• Get an estimate on level of difficulty on porting the script
• Determine a lead
• Port it
  • Hopefully breaking this step up into a few developers, if possible
• ???
• Profit? Nah, bro we're public domain. Benefit!!!

Relevant links:

• the build.xml file
• build script docs.. what it does in English
• existing open buildscript tickets

Guys that are interested in helping (and have experience with Rake):

• @joshuaclayton
• https://twitter.com/#!/KushalP
• https://twitter.com/#!/AlexGraul
• @markupboy
• https://twitter.com/#!/halfcube
• @akahn

(alternatively if anyone reading this thinks we should stick with Ant and wants to take the lead on iterating on it... by all means please speak up! :)

[paulirish]

Also it may be worth using Thor instead of Rake. Probably.

[KushalP]

Cool! I'm up for this.

It would be a good idea to get the discussion around rake vs. thor out of the way first. To start off with I'll run through the major differences between the two and then I'll provide my take on which I feel better suits h5bp. Sorry if a lot of this seems obvious, but it doesn't look like there's been much discussion over which is better in specific situations (context of use).

Rake:
A general build script tool that is project-specific. In other words, you put your Rakefile in your project folder and in your project's source control, and you create build and other automation tasks that are specific to your project in that Rakefile. rake requires a Rakefile to run.

Thor
A general purpose command-line scripting tool that makes it very easy to re-use scripts across many projects and to do project setup. Thor sits in the global namespace and becomes executable much like calling ruby, gem or rake in the terminal. However, thor's scripts are more suited to general purpose, cross-application automation because the thor script does not rely on a file sitting in your project specific folder — no Rakefile required. A Thor script is the entire script, packed and installed for re-use anywhere.

My thoughts between the two
The choices for both of these heavily depends on how people are using, or want to use, h5bp. My assumption is that each time they're downloading a fresh folder and editing for requirements. If this is correct, then it's better suited to a Rakefile. However, if what users want is something similar to rails, where they can run h5bp generate controller ... then Thor may be better.

As a comparison, Rails 3 uses Thor for everything that's not project-specific, but it still uses Rakefiles for things like rake test:units and rake db:migrate. I follow h5bp a bit, but don't know enough about how others use it past my own use-case, so would appreciate any info that can be given on how you expect this to move forwards.

[akahn]

I agree with @KushalP. Thor is a tool for building command line apps. Rake is a build tool. I think Rake would be more appropriate here. Rake has the additional advantage that it comes with Ruby and is thus already installed on any system that has Ruby (e.g. OS X).

[KushalP]

However... we should probably organise the rake tasks into relevant (sub) namespaces so that users can easily override or add in their own. This way, we can make the main Rakefile a set of docs explaining how best to add in new tasks for specific needs. This might be overkill though, realistically need some input from @paulirish on how he wants this to pan out.

[paulirish]

Kusha, really appreciate the detail on thor vs rake. Sounds like rake is a better direction to go in. Make in 1 year when there is heavier use of the build script would it be worth generalizing things out to be more global in thor.

your last comment i'm totally on board with. Just like the markup and CSS, I want the source of the rakefile to be readable and learnable. def.

@KushalP, what do you think are the next steps?

[akahn]

I agree with keeping the Rakefile minimal. I think the Rakefile could take on the role of the .properties files in the current build file. The user can modify the Rakefile to suit their needs, and the build library, which could live in build, is customized by the user's options. If we want, the new build tool could implement a task library (see: http://onestepback.org/articles/buildingwithrake/tasklibraries.html).

[KushalP]

@paulirish I'm an HCI geek, so I'd like to begin by getting the main groups of tasks outlined; these should act as nice namespaces for tasks in the Rakefile.

Then directly import all tasks (verbatim) into the Rakefile, putting them within the relevant task groupings (namespaces).

Finally, look at what we've got and then if it's logical and the groupings meet the needs of the project, I'd split it out into files, e.g. html.rb, css.rb ... etc. The aim for this is that we can do something like rake css:minify and rake images:publish.

However, you may feel it's better to do something like rake minify:css, but this is just for the fine-grained task view. Odds are that people will just want high-level tasks like rake minify and rake production. Your call.

P.S. I'm actually in the middle of exams at the moment, so won't be ready to fully hit this until end of May. In the mean time, if someone could go through and make a checklist — in plain english — of all the tasks that need to be converted, that would be grand. Just so we can keep track of what's done and what needs to be done.

[joshuaclayton]

I think Rake's the best way to go about this currently as well.

I haven't run the build script but I see a few things happening here:

1. Add a lib directory for the ruby code. We'll want to come at this correctly and designing this entirely in rake is a bad idea. Obviously, lib/tasks will separate rake tasks depending on their responsibility.
2. Add a Gemfile for Ruby gem dependencies.
3. Use cucumber + aruba to test output (I'm assuming there is some currently from the build script) and the generated files.
4. If it makes sense, unit-test individual segments of the generator. I'm more of the opinion that really awesome integration coverage would be better, but if there's a tricky piece of code (which there shouldn't), unit tests would be really helpful. I prefer RSpec over Test::Unit (@akahn and @KushalP, thoughts?); as an added bonus, there won't be folder collisions since the test directory is already taken by the boilerplate itself.
5. Add a Rakefile (within the top level) that loads all the lib/tasks files. This may occur automatically if we require 'rake', but if not, we'll need to require the tasks in lib/tasks in this file.

There are probably other things I'm forgetting, but that's what I've got so far.

@akahn As for the properties file, what about storing all the settings in a json file? We could parse it, validate it, ensure certain attributes are there if we deem them necessary, etc. It would also keep users in a friendlier format as well as keep them out of the Rakefile or rake tasks. My original thought was to use Ruby globals or a class, but that's more of a pain and easier to mess up, in my opinion. JSON is well-documented, easy to use, and we can check it quickly.

The biggest thing I want to stress here is integration testing the shit out of this thing. I didn't integration test the Blueprint generator and finally spent some time during Railsconf getting it working well. I used cucumber, aruba, capybara-webkit and jQuery to not only test the generator but test the actual CSS. The work relevant to integration testing is here: joshuaclayton/blueprint-css@bd1f0b5. It's not hard to do and will help a lot with the whole process, and it'll ensure we don't mess anything up. This project gets a lot of usage and tests will help the whole process of porting to Ruby and Rake.

[akahn]

Josh, I'm fine with whatever testing framework. Happy to let you get the suite up and running. ;)

I'm inclined to use build/ for the build rather than lib/ (more intention revealing).

As for JSON, sure, people are most familiar with it, but it would introduce an additional dependency (Cucumber, Aruba, RSpec would only be dependencies for contributors) and I'm not sure some foo = bars would be that hard for people to handle. I like the idea of being able to run the build on a bare OS X system without installing any gems, etc. So I'm okay with either way, but it's a tradeoff.

[roblarsen]

Just a generic comment about the switch. Porting it to Rake sounds like a great idea. I always envisioned build scripts for several different build systems- much like the various server configuration files. Making Rake the default build script sounds like it would solve a lot of the serious, elegant automation people want with this tool, so I can't argue with that either. The current build script is bursting at the seams as it is and Java/XML are clearly not the tools of choice for folks around here, so there's no sense trying to push it into being something it's not.

That being said, I'd hate to see the Ant script dropped entirely in favor of Rake. While it's clearly not a common tool amongst folks who make up the community here, Ant is deeply embedded in a lot of projects outside of this particular slice of the developer world (for better or worse.)

Personally, while I'm interested in the possibilities that Rake offers, the reality is that it does nothing for me in my day-to-day work because (as @paulirish knows well) we don't do a lot of Ruby work at my job.

We do a hell of a lot of Java.

This is evidence of both our and our client's preferences. In the larger corporate world, we're not the oddballs- we're an up-to-the-minute reflection of what goes on inside the walls of big companies.

So will we use anything that comes out of this Rake port? Nope. I guess anything is possible, but it's highly unlikely. Will I be pulling some of the tasks from the ant script into a current project over the next few weeks? Indeed I will. Ant is baked right in so it's a no-brainer.

So, to me, keeping it around and maintained at current levels* would still be providing a valuable service to the many, many people out there who are deeply wedded to Ant as a tool.

Failing that, I'd be happy to maintain a separate repository for an Ant build tool based on the h5bp structure. I'd keep something for my own use anyway, so I'd be happy to keep it going in a public way.

*or potentially even simplified. Giving up on being the "one build tool to rule them all" and just catering to people who are used to Ant would solve a lot of problems. I feel like whenever I've documented a fix for a request on the mailing list it's just a simple, but inelegant, edit to the build.xml itself. For people used to Ant, the suggested edits I suggest aren't a big deal. They're definitely not the elegant, push button solution sought after here.

[craigbarnes]

It doesn't matter which platform is chosen, it's going to alienate someone. Wouldn't it just make sense to separate the build script from this project and avoid having a "default" altogether?

It seems like a good idea to just pick a default and run with it but like @roblarsen said, people who have an opinion on build tools and people looking to use it with existing projects aren't going to want to introduce a dependence on Ruby (or whatever else) into their existing tool stack.

People will probably pick their own "defaults" from what's available anyway.

[roblarsen]

I think the Ant script is too big, too. I know Ant, wrote the initial version of this script and I get confused in it now. I don't use the latest version. It's too baroque for my tastes.

Which is why I talked about simplifying it earlier.

Trying to make it so smart that no one would ever have to touch XML was ambitious, to say the least. And really, if you take the approach that Ant requires some tweaking of actual XML, I think it becomes a much more sane thing to manage. I know that the reaction here has been "wow, what the hell is this Ant thing?" but outside of this very precise slice of the technology landscape Ant is a known quality, so saying something like

<-- add CSS files here, line by line, for concatenation -->

In the build script itself is a perfectly acceptable solution to the question of concatenating multiple CSS (or javascript) files together. I mean, while the h5bp project as a whole has been searching around for a no-xml, elegant, scripted @import solution for multiple CSS files for months, people used to Ant have been able to create the same end result using just core Ant (even skipping the ant-contrib foreach) by skipping over the .properties file and just explicitly listing the files that needed to get concatenated directly in the build script. Not beautiful, but it gets the job done.

I'll reiterate what I think the best solution is- multiple build scripts, either core to the project or as separate, linked repositories. As @craigbarnes said one will get the most love as the default, but as I've pointed out if the one getting the most love isn't going to fit in with the stack someone is working with it doesn't really matter how much love it's going to get. This proposed Rake script could do everything we've ever dreamed of in the build script AND make me espresso in the office every day at 2:00 and I would still get laughed out of any meeting where I suggested we patch some ruby into the project as an extra dependency to use it. It's just not going to happen.

We've got Ant. I'd be happy to scale it back to make it sane to manage (with the idea that anyone using it has to edit XML) and easier to hack/steal tasks (which is the way it's going to be used by most bigger projects anyway.) Rake looks like it has the momentum to be an aggressively developed default option*. Make sounds like another perfectly sensible option. Beyond that? See what comes out of the community.

*in researching other build systems, I saw that the the Rake wikipedia page is slapped with the "may not meet the general notability guideline" warning. Maybe setting it as the default option and getting Paul to screencast it will smash down the doors to notability at wikipedia \o/

[paulirish]

Thanks everyone for dropping knowledge and thoughts. I'm happy to see what community feedback is and personally I'm not dropping responsibility for maintaining the Ant script.. I just want the build script to go places beyond where it's at today. I'm happy to see how things suss out within the community. :)

Anyhow!

@joshuaclayton, can you start up a fork and lay down the scaffold of the tools you mentioned?

Once that's in place, hopefully @kushaip is done with his exams and we can tackle porting the tasks. :)

[joshuaclayton]

@paulirish, definitely. I think getting a basic implementation working without striving towards being feature-complete is the best route. The users of the build script will chime in with the stuff they want/need (or patch the project, if they're comfortable with Ruby).

Additionally, it may make sense to set up an online form with predefined answers asking what their "required" features are for the generator to be useful. I've caught bugs in the Blueprint generator that never got reported that I thought were going to be widely used; it turns out a handful of the features there don't get touched. Asking beforehand will help narrow scope to something that we can knock out over the course of a week instead of a month or more.

[edbo]

Our team is actually working on integrating the html5 boilerplate into our workflow and my only gripe was going to be that it didn't use rake. We were actually at the point of porting the build scripts we need to rake for our buildserver ourselves.

In summary we would love to help out with this endeavor! Let me know if you need more contributors...

[paulirish]

@edbo -- awesome!

looks like joshua has started some scaffolding over here https://github.com/joshuaclayton/html5-boilerplate/commit/960a7bdd9

i'd say watch that branch and as it begins to take shape i'd love to see you add some of your work in. :)

[joshuaclayton]

@paulirish Is there a list of basic features for an MVP for the rake task(s)? If not, can we set up a Wufoo form and get some community involvement on a poll to help drive development? As I said before, I'd rather write the tasks that people absolutely need so we don't go down rabbit holes or spend too much time in one area. Getting a basic outline for what's necessary will be really helpful moving forward.

[KushalP]

I'm finally free of exams! Woohoo! So I can get going on some bits of this.

@joshuaclayton the MVP tasks list sounds like a good idea. @paulirish can you command your Twitter followers accordingly please?

Also, can we get a rake or similarly-named branch going where everyone comments on issue #516 please? This will make tracking it a lot easier and we can just do network diffs from the main rake branch @ html5-boilerplate. This will also streamline getting updates on what everyone's working on with regards to this issue.

Sorry if I'm stating the obvious, but you can write the issue number in the commit. This is what triggers it to email everyone with an update so we can handle code review and discuss ideas easier. There's a better run down in the blog post from the @github folks here: https://github.com/blog/831-issues-2-0-the-next-generation

Just make sure not to write fix and the issue number, because having that merged will just be a pain 💣

[stereobooster]

https://github.com/paulirish/html5-boilerplate/wiki/rake

[stereobooster]

I can't tell about pros and cons of switching to rake. But I know there are projects that will benefit from this work. For example jekyll by @mojombo (actually any ruby static site generator).

[paulirish]

Preparing a wufoo form to get feedback on the buildscript and was able to wrangle some details of the build script... It's evolved via many people so I think I'm starting to see some parts have a bit too much wtf..

We currently let you do a subset of tasks (as a build target).

• build - minor html optimizations (extra quotes removed). inline script/style minified (default)
• buildkit - same as first but html comments retained
• basics - same as first minus the basic html minification
• minify - same as first plus aggressive html minification
• text - same as first but without image (png/jpg) optimizing

Then we also currently let you target three different environments.

• production - do everything specified in your build target (default)
• test - we dont have profiling or strip console.log anymore so this is actually identical to production
• dev - only copy over the new files and compress images. (assuming your build target does images..)

I think we can kill one of the built targets, but maybe add an additional one for copy+compressimages, that way we can also kill the environments.. Regardless, the above is now documented a little more clearly and up to date.

I'll post the poll here before it goes live so we can revise if neccessary

[paulirish]

okay!

poll: https://paulirish.wufoo.com/forms/h5bp-build-script-survey/

results: https://paulirish.wufoo.com/reports/h5bp-build-script-survey/

I tried to word it in a way that gets the top priorities for folks.. and also targeted the things we do and have been thinking about.

Please take a look comment here if I'm missing something or lacking clarity! :)

[roblarsen]

Does it make sense to open a separate issue to incorporate feedback for the the ant script?

[joshuaclayton]

@roblarsen - if it's regarding a bug or issue with the existing build script, it should be in a separate issue. If it's functionality that you feel is missing currently and would like to see in the rake script, please fill out the poll that @paulirish posted above (https://paulirish.wufoo.com/forms/h5bp-build-script-survey/) and leave it in the comments area (if it doesn't sound like the script will handle what you're trying to do). Thanks!

[roblarsen]

I should have directed the question @paulirish specifically. I get the sense, from Paul's comment that there's room to pull out some of the more baroque features of the ant script. If that's the case I would be very happy to take part in that discussion.

[paulirish]

rob, i im hoping we'll get feedback on the ant script via the text field.. do you have any ideas for other questions/checkboxes that would help?

[roblarsen]

I'll give it a little thought over lunch.

[joshuaclayton]

After looking at the results of the survey, I think hitting 70%+ of users' requirements should be a great start. Based on the current results of the survey, that's three big-ticket items: combining and minifying CSS, combining and minifying JS, and image optimization. Based on the subsets, I think we should move forward with just the defaults for now too.

This will leave the work on this script highly targeted and pretty straightforward for an MVP. Thoughts?

[paulirish]

sounds groovy to me.

[edbo]

I think that would be great and it definitely fits what I was looking for. If someone takes up the first or perhaps implements a portion of one I think that would be great to use as a template to make sure we all follow the same testing conventions etc... anyway, just to re-iterate you still have my support if you need help.

[meleyal]

Jammit solves a lot of the same problems, might be good for reference (specifically the configuration options):

http://documentcloud.github.com/jammit/
http://documentcloud.github.com/jammit/#configuration

[KushalP]

Of the two branches that are on the wiki, I've decided to build from what @joshuaclayton started. Purely because it has a basic unit testing structure.

@joshuaclayton I'm curious why you decided to ignore the Gemfile.lock file? It could be extremely handy in helping users debug the Rakefile locally if something's not working. Running bundle would immediately tell them what gems are different from the gems we've been using to test the build script. In addition, we could also add this as a quick test/check to begin with.

Here's my branch. Once I've got the first feature passing unit tests I'll send a pull request through.

[joshuaclayton]

@KushaIP I'd been ignoring Gemfile.lock files in gems based on http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/, although there's been some discussion at Thoughtbot and it's more up-in-the-air now. I'm not horribly opposed to checking it in.

[stereobooster]

It seems to me that we need slightly different approach. These ruby specific stuff (lib, feature, spec, Gemfile) should not be placed in hb5p project. We need separate project for a gem. This gem can expose rake tasks or executable. So end users will do something like:

gem install h5bp
h5bp optimize js

[KushalP]

@stereobooster I think this was discussed earlier in the thread with thor vs. rake. There are certain nuances of a project that @paulirish stated where users would like to quickly hack in tasks they required. Switching to rake facilitated this better than the current ant script.

Of course, keep up with what you're doing as I agree that this is likely to be where the final Rakefile will end up: A set of common tasks for h5bp and then the custom ones for users. I think the only issue (currently) is adding an extra dependency to h5bp (the gem). By having a separate gem it's not as contained anymore.

[KushalP]

Just to add to this discussion that there is a dedicated rake branch to the project here: https://github.com/paulirish/html5-boilerplate/tree/rake

Feel free to fork it and add as you please. The more the merrier!

[7fe]

Has it been considered to use JavaScript for the build script? .... Don't laugh, I'm serious :D

[roblarsen]

Yeah, there's an open ticket about it. https://github.com/paulirish/html5-boilerplate/issues/523

[AD7six]

FWIW I think whatever build tool is chosen, it's the wrong choice. I'd much prefer for the build scripts to be written in the lowest common denominator: bash.

Maybe I just don't get it, but the reason being: if it's not the build tool to be used with whatever project you're trying to use html5boilerplate with - it's code that can't be used, and probably written in a language you're not familiar with. A project such as html5boilerplate shouldn't dictate what build tool is used, only provide the building blocks to use in any built tool (yes, all the jars are provided, and you can 'easily' run help on each one to see what it does etc).

If it were written in a build-tool-agnostic manner, that makes it easy to integrate with whichever is your build tool of choice, instead of having a (however well functioning[1]) build process you can't use. I'd say it's definitely a barrier to some users that call this program is re-written in a different format which takes time to read and decypher (if it's possible for them) to realize that the build script is 'just' running call this program. Using a language that most users are familiar with (who, that uses a build process or wants/is capable to do so, is not familiar enough to at least read a bash script) will surely aide both the users and the project.

I'd be curious to know what % of users of this project use the ant task as is.

[1] I've never actually gotten the ant task to do what I want, e.g. ran on a jekyl project it missed optimizing almost all files.

[craigbarnes]

@AD7six - that's the beauty of make. It's just a wrapper around bash to provide a command-line interface, dependencies, macros etc.

[AD7six]

make would be my choice, but it's still a /slightly/ alien syntax if you're unfamiliar.

Edit: I somehow missed your messages in the thread, basically: I agree.

[akahn]

Cool bikeshed, bro.

[AD7six]

@akahn - k, anything except rake get's the vote.

[rigelglen]

How about Phing instead of Rake ? A lot of people know PHP compared to Ruby

• Simple XML buildfiles
• Rich set of provided tasks
• Easily extendable via PHP classes
• Platform-independent: works on UNIX, Windows, MacOSX
• No required external dependencies
• Built & optimized for ZendEngine2/PHP5
• A lot of people know PHP as compared to Java ( Ant ) and Ruby ( Rake )

[andyinabox]

FWIW I found this thread after trying to mod the build script for a Capistrano deploy, and thinking to myself "man I REALLY wish this thing was built using Rake." Glad to know there's progress on that front...

[paulirish]

from what it looks like, the energy has shifted over to a Cake script (which can use both coffeescript and JS)..

https://github.com/h5bp/html5-boilerplate/tree/cake/build/cake see the extensive readme.

Most of the functionality was already put into place by @mklabs so we're looking to fold this in sometime. Would love others' feedback and eyes.

[necolas]

Closing. This work is already underway and will soon be in a dedicated repo.

[stereobooster]

One more alternetive grunt

[necolas]

@stereobooster See #523 (comment)