If any of the cucumber rails tasks fail, they'll return a non-zero exit code. When running "all", all tasks will run, and if any don't succeed, the whole rake task will return non-zero.
As of this commit, the heroku gem can't be installed or one of the cucumber features breaks for versions >3. Still figuring that one out. 2.2.2 to 3.0.1 rake completely green.