small brain teaser #207

MarcWeber opened this Issue Jul 3, 2013 · 8 comments


None yet
2 participants
rake -m  -j 4 -t abc

How long is this going to run?

multitask "a" do
  sh "echo a > a; sleep 2"

multitask "b" => "a" do
  sh "echo b > b; sleep 1"

multitask "c" => "b" do
  sh "echo c > c; sleep 1"

multitask "z" do
  sh 'sleep 1'
  rasie "bad"

multitask "abc" => ["c", "z"] do

Seriously: Why does'nt it run z and a in parallel, then stopping after 2 secs because "z" fails?

I mean devs want both: "--keep-going-in-case-of-failures" and "--stop-on-first-failure" like behaviour, fail early is more important. Am I missunderstanding anything this time?


jimweirich commented Jul 3, 2013

I haven't reviewed the code, but I suspect that the exception is only transmitted to the main task when it waits for the result of task :z. Since task :z is run in a separate thread, exceptions thrown in that thread are not seen in the main thread.

rake's documentation says "rake's version of makefiles" - as dev - I think its not enough. Worse: I somehow feel that it would have taken me less time to rewrite something "rake" like from scratch than trying to understand why -j 1 silently behaves like -j 2, why I should bother the user to lookup whether -m is ok or not etc. You know the rake community better than I do. Is it worth fixing? If so in which way? I'm not happy with drake either (without -j it behaves similar to rake, and with -j 4 it takes too long till it starts a build for whatever reason) Is there yet another alternative which is more dev and user friendly? Is there yet another library I may want to look at?

Thanks for your support. Its appreciated. I just don't think that the design of rake fits my needs very well.


jimweirich commented Jul 3, 2013

Thanks for the feedback, but I'm not sure what you are asking to be fixed. The -j1/-j2 issue? Removing -m? The fail early issue? User friendliness issues?

If you wish to make a concrete proposal, I would be happy to evaluate it.

FYI: the -j1/j2 issue is already fixed in HEAD.

There should be 2 ways to run rake:

  • fail as early as possible whenever a task failed (keep running tasks running)
  • fail as late as possible - try to run as many tasks as possible successfully (thus continue till no tasks are left which could be run because prerequisites failed)

Imagine task a and b taking 2min to run. But you have to rerun them (for whatever reason, eg because you use clean because something rake doesn't track has changed). Then why run b if z failed? If you're in a console the difference may not be that "big", because you can abort by ctrl-c. If you have to use editors loading the result into an error list aborting manually is not that nice. Fail early is what I want then.

So I want rake to abort. And I want a way to specify "-m" in the Rakefile or have a nice way to make a task a "multitask". The user should just run "rake target" without bothering thinking about -m or parallel jobs.

I'm aware than "-m" cannot (and should not) be removed for compatibility reasons.


jimweirich commented Jul 3, 2013

Go ahead an make a feature request for the fail early part (i.e. open a specific issue for just that feature).

Then review issue #205 and see if that meets your desire for an easy way of making any task or file a "multitask". Feel free to add comments there.

Feel free to close this discussion thread once you've done that.

Fixing "rake" and testing would have taken too much time.

Instead I've implemented a minimal alternative make like system for Ruby which does exactly what I need:

There is yet another new feature: It builds the dependency chain which will take longest first.
Eg if there are 100 tasks not depending on anything and 20 task depending on each other
one thread will start with the 20 chain first.

Rake users can give it a try to find out how much value this provides to them for their projects.
Then it can be discussed whether those features are worth integrating into rake.

You may want to keep this issue open for a while or add this as "future ideas" to
your README to gather more feedback about whether users (beside me) want such.


jimweirich commented Jul 8, 2013

Thanks for the feedback. I'll keep this request in mind, but I'm going to go ahead a close it for now.

jimweirich closed this Jul 8, 2013

small update for those who are following this thread: I implemented "eta" feature and redoing a task if the command line changes. eta and previous commands are stored in a cache file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment