Implement parallelism/thread-safety

[bitprophet]

Description

Fabric currently uses the simplest approach to the most common use case, but as a result is quite naive and not threadsafe, and cannot easily be run in parallel even by an outside agent.

Rework the execution model and state sharing to be thread-safe (whether by using threadlocals or something else), and if possible go further and actually implement a parallel execution mode users can choose to activate, with threading or multiprocessing or similar.

---

Morgan Goose has been the lead on this feature and has a in-working-shape branch in his Github fork (link goes to the multiprocessing branch, which is the one you want to use). We hope to merge this into core Fabric for 1.1.

---

Current TODO:

• Anal retentive renaming, e.g. s/runs_parallel/parallel/
• Code formatting cleanup/rearranging
• Mechanics/behavior/implementation double-check
• Linewise output added back in (may make sub-ticket)
• Paramiko situation examined re: dependency on 1.7.7.1+ and thus PyCrypto 2.1+
  • Including documenting the change in the install docs if necessary
• Pull in anything useful that Morgan hadn't pushed at time of my merge
• (if not included in previous) Examine logging support and decide if it's worth bumping to next release
• Test, test, test

---

Originally submitted by Jeff Forcier (bitprophet) on 2009-07-21 at 02:52pm EDT

Attachments

• multiprocessing.patch

Relations

• Related to Rework output mechanisms #20: Rework output mechanisms
• Related to Make execution model more robust/flexible #21: Make execution model more robust/flexible
• Related to Handle running without any controlling tty #197: Handle running without any controlling tty

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

For some discussion concerning users' needs along these lines, see this mid-July mailing list thread.

---

on 2009-08-09 at 11:31am EDT

[bitprophet]

Wes Winham (winhamwr) posted:

---

My 2 cents:

Reworking the execution model and implementing thread-safe state sharing is probably going to be really hard. It'll touch a lot of code, be subject to a lot of hard to track bugs, and might require backwards-incompatible changes. It would, however, be very useful for some people (probably a minority though).

To me, this makes it a candidate for either looking at when some of the more high-use issues are implemented or for work by someone else (some person or group of people who need it) in a fork. Things like more example documentation, easier output control, improved prompt handling, and #23 generic persistence handlers (so I can do everything in my virtualenv easily) are all things that would benefit everyone substantially and also not be so big on time commitment. From a cost/benefit, I'm thinking that those are much bigger wins. Not at all saying that thread-safety wouldn't be awesome, just saying that you (Jeff) have limited time.

I might be stating the obvious though and as someone who could benefit from thread-safety (I manage 6 machines in my case), but doesn't see it as a huge personal priority, of course I might be biased.

---

on 2009-08-09 at 12:50pm EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

Don't know if this is off topic for this thread, but I've made a small change in the main.py, with a command flag, that'll leverage the multiprocessing module to fork out processes for each host. It is a bit messy with stdout, but seems to work fine for my test fabfile.

It'll use the normal execution model by default, but if someone wants to run all of their hosts in parallel -P/--parallel will switch into using the Process() for each host.

One thing to note is that it won't run tasks simultaneously. Those will still be processed sequentially, while every host on a task will be run in parallel.

---

on 2010-03-11 at 04:27pm EST

[bitprophet]

Morgan Goose (goosemo) posted:

---

As that patch stands, it expects one to have the multiprocessing module. This is a non issue for python 2.6, but for 2.5/2,4 the backported module needs to be grabbed from http://code.google.com/p/python-multiprocessing/

---

on 2010-03-17 at 10:25am EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

Here is an updated patch that will check if the multiprocessing module should be imported, and will allow for python versions (2.4/2.5) w/o said module to not error out when attempting to run.

---

on 2010-03-17 at 04:52pm EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

Just as an update, this is the tip commit for the multiprocessing branch I made for this issue.
http://github.com/goosemo/fabric/commit/3406723d2f2f9fc41de2a5f064da46b460ff35a3

It adds in a cli flag -P for forcing parallel. It also adds two decorators, @runs_parallel and @runs_sequential, that will force those options on a fab task regardless of the parallel cli flag.

If anyone has issues, or requests, let me know.

---

on 2010-04-01 at 10:09am EDT

[bitprophet]

Jeremy Orem (oremj) posted:

---

It would also be beneficial to be able to specify how many parallel commands to run at once e.g., I have 100 systems and I only want to run service httpd reload on 20 at a time.

---

on 2010-05-13 at 05:12pm EDT

[bitprophet]

Joshua Peek (josh) posted:

---

This is a feature that I look forward to having in 1.0. Thanks for your work.

---

on 2010-05-20 at 07:56pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Just a note, the work done over in #7 now means that remote stdout/stderr are printed on a byte-by-byte basis instead of line-by-line (fabfile-local print statements/interactions with sys.stdout remain line-buffered as usual). This may impact any parallel solution more negatively than the previous line-buffered behavior.

Whether we solve that for this ticket by adding back in some sort of "legacy" line buffered, no-stdin-available mode (gross, lots of potential for code duplication, still lots of potential for confusing output) or taking a harder look at what sorts of output people expect when running jobs in parallel (I haven't seen whether Morgan's code handles this at all -- doesn't it get pretty confusing no matter what, unless you simply hide all but your own print statements? would we want to just be autologging each server to its own log file and expect folks to multitail or something?) isn't something I will be looking at right now, but it will need to be tackled before this ticket here can be considered done.

run-on sentences yaaaaay

---

on 2010-06-18 at 03:13pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Note that came up via IRC while Morgan was testing out a merge of his branch and the #7 branch -- because the MP solution runs a bunch of Fabric "workers" headless without actual TTYs involved, this impacts a lot of assumptions Fabric makes , though primarily the ones made by #7 (so far, the code that forces stdin to be bytewise is what errors out). So basically, the previous comment but writ large.

Keep the overall ramifications of this in mind when integrating this work. Will be trying to update #7 with a fix for this soon.

---

on 2010-07-21 at 12:39pm EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

So just today I got what I feel is an acceptable version of my multiprocessing
branch. To test it I suggest you make use of virtualenv (and virtualenvwrapper)
to keep your mainline Fabric install separate. But the code is up to date with
the HEAD of the fabric repo, and is able to be grabbed on github here: [[http://github.com/goosemo/fabric]]

I changed the readme some to explain the options I've added. Feel free to
peruse the code, and let me know if you find anything wanting in this
implementation.

---

on 2010-07-21 at 10:13pm EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

With the newest updates I am able to merge in the multiprocessing branch I have, and have no issues when running my personal scripts that utilize parallel execution, and a simple test script of some run|get|put|cd()s. People can grab commit 2f56f612846c31c94c77 and test it out some more. I'll be working on trying to make unit tests to verify parallel execution, and since it's not in the main branch people can use http://github.com/goosemo/fabric/issues for tickets. (Unless Jeff wants/prefers issues be placed in this ticket.)

---

on 2010-10-07 at 11:56am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

As mentioned on IRC, I'm fine with people using fork-specific GH issues pages, as long as any outstanding issues at merge time get ported over here. (Basically, code in the core Fab repo should have its issues documented here.)

Excited that it seems to "just work" with the latest IO changes in master. As discussed elsewhere I think we'll want to update things so that we have more bulletproof IO settings for this mode (I'm 99% sure that with nontrivial commands, users would run into garbled output between processes in the bytewise mode we use now) but if that's not resolved when I have time to look at this myself, I'll take care of it.

---

on 2010-10-07 at 12:04pm EDT

[bitprophet]

Morgan Goose (goosemo) posted:

---

Can you think of a good test command that'd give some output that you'd expect to be garbled? Also what do you mean by that, an interweaving of output lines, or an issue with an output line itself having the output of another command break it up?

---

on 2010-10-08 at 03:58pm EDT