External fabric utility library

[wraithan]

There has been lots of discussion about this, but nothing that has become a canonical standard or even a listing of all interest(ed|ing) libraries.

A goal of this is to enumerate all of the libraries, then the features of said libraries. Finally, we will try to come to a consensus on a library to use as a launch board and pump the willing talent of the community that uses Fabric into to hopefully come out with a library that we can use as a base instead of copying and pasting out of each other's scripts or reinventing the same wheel over and over again.

If you have a library of fabric utils, or know of one that isn't listed here, please comment and add it to the list. If you are the owner of one of these libs and would like to work together on this project, regardless of if your lib is the chosen golden child (though participation certainly makes it more likely) please join us on irc.freenode.net in the #fabric channel.

• https://github.com/mlavin/argyle
• https://github.com/bretth/woven
• https://bitbucket.org/kmike/django-fab-deploy
• https://github.com/hbussell/django-fab
• https://github.com/duointeractive/django-fabtastic
• https://github.com/tav/pylibs/tree/master/fabric
• https://github.com/ronnix/fabtools
• https://github.com/ddbeck/bolt
• https://github.com/sebastien/cuisine
• http://bitbucket.org/petersanchez/djeploy
• http://heynemann.github.com/provy/
• https://github.com/tav/bolt

[bitprophet]

This is somewhat related to #99. Go go Github issue mention linking!

insert "we are the #99 percent" joke here

[robmchardy]

There's Ronan Amicel's library at https://github.com/ronnix/fabtools

Recently discussed on Fab-user mailing list.

[wraithan]

Added fabtools to the list.

[ddbeck]

I've been putting my Fabric utilities in https://github.com/ddbeck/bolt (docs).

[wraithan]

Bolt added

[dillongreen]

yet another one:

• https://github.com/sebastien/cuisine
• http://www.slideshare.net/ffunction/fabric-cuisine-and-watchdog-for-server-administration-in-python starts at page 42

[petersanchez]

Can also include,

http://bitbucket.org/petersanchez/djeploy

[goosemo]

Don't exactly fit, but they use Fabric:
http://heynemann.github.com/provy/
http://fabric-provision.readthedocs.org/en/latest/index.html

[wraithan]

Updated the list with the additional mentions.

[bitprophet]

Here are my initial thoughts from reviewing these; they're colored a bit by my current needs at work but I'm trying to keep a general eye too. Specifically, my goal is to have a contrib/external contrib lib with very platform/intent agnostic primitives like Chef's "Resources". Install package, create/tweak file, upload rendered template, etc.

Things like "install an Apache/Gunicorn/Celery stack" are IMO better left out of any "blessed" library, probably instead on a recipe/snippet site a la #99.

I'm doing the usual -1/-0/+0/+1 voting on these.

I will probably update/edit this comment as I go, apologies if any responses become outdated.

• https://github.com/mlavin/argyle
  • Has some basic Debian-specific primitives which look OK.
  • Also has more complex recipes for Apache/Celery/etc
  • +0
• https://github.com/bretth/woven
  • Covers a wide range of targets: user/package management, state
    introspection (open ports, etc), virtualenv/pip stuff, Django-specific
    stack tasks (which is the stated goal), "projects" etc.
  • The lower level primitives look at least moderately fleshed out and
    stand-alone.
  • +1
• https://bitbucket.org/kmike/django-fab-deploy
  • Feels like Capistrano-level opinionation re: deploying Django apps (for
    better or worse)
  • Has some primitives which do a lot but are also somewhat opinionated /
    tied into the rest of the system
  • Also has mid-level constructs for running funcs as a given user,
    virtualenv/pip etc
  • +0
• https://github.com/hbussell/django-fab
  • Doesn't seem to be much "there" there. Basically some Django oriented
    util funcs & light VCS support. Maybe I'm missing something? Feels like an
    honest "some stuff useful for me in >1 project, on github" project. Great
    for them, not too useful for us though.
  • -1
• https://github.com/duointeractive/django-fabtastic
  • Has some primitives but they are relatively tightly integrated with the
    system
  • Stupid nitpick but the file names are very silly IMO, c_*, ft_* etc.
    Directories, how do they work?
  • -1
• https://github.com/tav/pylibs/tree/master/fabric
  • This is superseded by Tav's "bolt" link below.
• https://github.com/ronnix/fabtools
  • Looks pretty good -- generic, agnostic low level primitives. Not an
    enormous number but a moderate amount.
  • Has declarative Chef-like layer, with the unfortunate name of "icanhaz".
  • +1
• https://github.com/ddbeck/bolt
  • Tiny, similar to django-fab re: not really for building on.
  • -1
• https://github.com/sebastien/cuisine
  • Description sounds exactly like what we want/need
  • Actual code seems more basic than description implied, but still in line
    with it.
  • Lots of good low level primitives, with only minor tie-ins to rest of
    system (which isn't huge anyway)
  • Minor verbosity in API (lots of similar _suffixes)
  • +1
• http://bitbucket.org/petersanchez/djeploy
  • Similar to a few of the others -- common, somewhat agnostic primitives,
    with minor/moderate tie-in to some global utils re: env tickling.
  • +1
• http://heynemann.github.com/provy/
  • It's got its own pretty integrated setup/flow, so not good from a "grab & modify" perspective re: an unopinionated library of Fab using primitives.
  • That said, it does have a smallish set of the type of primitives we're looking for, re: files/directories/packages
    • Including for a few different target platforms
  • API is too verbose, IMO.
  • -0
• https://github.com/tav/bolt
  • Doesn't actually have any primitives -- it's basically a bunch of extra
    Fab core features implemented as a (now impossible to pull patches from :()
    fork.
  • -1

---

So the +1'd items IMHO deserving of further scrutiny:

• Woven
• Fabtools
• Cuisine
• Djeploy

[ronnix]

Due to popular demand, the Chef-like layer in fabtools has been renamed "require", deprecating the original "icanhaz" name... :)

[wraithan]

Fantastic, Fabtools has some really good stuff in it. I am glad it will have a more official looking API, I mean, we aren't Ruby hackers... :)

[bitprophet]

Nothing wrong with Ruby per se, of course. I love Ruby*, I just find the average Rubyist's overt emphasis on zany bullshit to be off-putting at times. LOL GUYS LOOK HOW WACKY WE ARE !!!!1!11!!one ZOMG WE ARE ARTISTES

---

On track: I'm going to start doing Actual Crap at work this week which means I'll be taking some of these out for a spin. I'm still not sure if I'm going to start some new repo that liberally grabs bits n pieces from all over, or take one of them and use that as a base or what.

Also not sure what @wraithan had in mind on his side, I'm just being motivated by pragmatism over here.

---

* seriously, I actually find it a moderate amount more "fun"/speedy-to-dev in than even Python. If the community were a little more grown up (and actually maintained any software past its OOH SHINY period) I'd probably have jumped ship instead of remaining a polyglot.

[bitprophet]

Wanna make a repo for this and start hacking on it. So we need a name. Oh boy! I love/hate naming shit.

A brainstorm, focusing on "stuff you use fabric to create" (versus the "stuff used to make fabric" ideas I had for what became boringly-but-appropriately ssh):

• clothes (only half serious, a bit too Ruby-esque)
  • Also sewing, sewingkit
• bolt ("a collection/group/chunk of fabric"; but already used elsewhere x2, so probably not.)
• kite (something you make with fabric, with bonus 'flying' connotations linking to deploying/etc)
• weave (something you do with fabric; close to, but not the same as, the already-taken woven. Also close to, but not the same as, python-weave which is a client for the mozilla service of the same name.)
• patchwork (a collection of pieces of fabric; fits well; not taken!)
• tapestry (ditto, though there is an Apache Java project with that name)

Alternately, "fabric patterns" or "types of fabric", such as the already-taken argyle:

• brocade ("forming patterns in cloth")
• damask (similar; also, the real thing looks cool.)

---

My personal faves: patchwork, tapestry, damask.

[goosemo]

tartan
bobbin
tailor

but I like brocade too since it'd be amusing to just put how bro'ey we
are right in the name.

• goose

On Thu, Dec 15, 2011 at 1:26 PM, Jeff Forcier
reply@reply.github.com
wrote:

Wanna make a repo for this and start hacking on it. So we need a name. Oh boy! I love/hate naming shit.

A brainstorm, focusing on "stuff you use fabric to create" (versus the "stuff used to make fabric" ideas I had for what became boringly-but-appropriately ssh):

• clothes (only half serious, a bit too Ruby-esque)
     * Also sewing, sewingkit
• bolt ("a collection/group/chunk of fabric"; but already used elsewhere x2, so probably not.)
• kite (something you make with fabric, with bonus 'flying' connotations linking to deploying/etc)
• weave (something you do with fabric; close to, but not the same as, the already-taken woven. Also close to, but not the same as, python-weave which is a client for the mozilla service of the same name.)
• patchwork (a collection of pieces of fabric; fits well; not taken!)
• tapestry (ditto, though there is an Apache Java project with that name)

Alternately, "fabric patterns" or "types of fabric", such as the already-taken argyle:

• brocade ("forming patterns in cloth")
• damask (similar; also, the real thing looks cool.)

---

Reply to this email directly or view it on GitHub:
#461 (comment)

[bitprophet]

patchwork it is, I (and the twitters) have spoken.

@tswicegood mentioned he feels a meta-package importing a number of other independent packages works best. I assume because it would make it easier to target changes (i.e. it's an extension of the same reason this package is intended to be distinct from Fabric proper.)

I disagree, mostly on the grounds that that would create a really big headache/overhead re: project maintenance, compatibility, release schedules, etc; and that such a headache would outweigh the benefits of being able to release patches for the individual components. I also am not sure that it would be easy to decide how to split things up.

Will wait for Travis' actual comment though.

[tswicegood]

FYI, I've said this one Twitter and I'll say it here. Unless this package is a meta package for installing specific other packages, I'm -1 on this.

I shouldn't be forced install pyyaml, jinja2, django, or anything else just to get at one part of this that adds a few simple tasks for restarting Nginx on remote machines. Going the meta package route means that those who are ok installing the kitchen sink can easily do that, but those who want more granular control have that capability. Without that, you either install fabric
and patchwork|tapestry|damask and get everything or go the copy/paste route that this ticket was started to stop.

Fabric has a lot of external dependencies already, creating a secondary package for extra tasks that follows that pattern is wrong and harmful toward the community as a whole. Developers new to Python are going to follow the patterns they see successful projects using. Django, Fabric, and company are promoting the idea that you should have a large package that installs everything. Or at least several larger packages instead of dozens of smaller ones.

Regarding actual headache involved with releases, that would be minimal. All of the packages are developed independently, and the <insert X name here> library is released on a timed schedule. Each release, packages that are already included are updated to their latest stable release.

As for figuring out what gets in, it's simple: pull requests on the main file that change the setup.py. Discussion happens around the PR with voting (if its not an obvious yes).

[bitprophet]

@tswicegood: my intent was never for the additional dependencies to be required; they should be like how they are in Fabric, where attempts to import are wrapped in try/except such that you only need to install them when you are using those optional features. It should ideally only require fabric.

Let me know if something in the plans above contradicts this, I might be forgetting something. But right now the Jinja and Django integration in fabric.contrib are both optional / runtime-only imports and that's the pattern I expect to follow.

So somebody wanting to use a single simple function in this package would pip install patchwork and end up with patchwork, fabric, ssh, pycrypto and that's it.

---

On the general "large package vs smaller packages" front, I acknowledge your point for how the smaller-packages-plus-meta-package setup could work. However I'm still unconvinced that the additional overhead and complexity is worth it:

1. Given the 1st half of this comment, installation would not be a Big Deal in terms of dependencies.
2. I also don't expect this to become a large library in terms of raw size; look at the number of Resources that Chef ships with, it's maybe a couple dozen functions. That's roughly what I am aiming at, personally. Not something e.g. Twisted/Django sized.
3. Even if we assume the extra development complexity is worthwhile I do not think it's worth the mental complexity for the userbase.
   • New/intermediate users would have to learn about how the sub-packages are organized, to make intelligent installation decisions. This is a burden I don't want to foist on the userbase without good reason.
   • And I'd expect the average user to "just install the metapackage" 99% of the time, at which point it feels moot. Having a highly different development model solely to cater to advanced users doesn't feel good to me.
4. I don't think the extra development/release overhead is worthwhile.
   • Fabric hasn't had the most rapid release history, but I think especially this year it's gotten better, despite it having a lot of moving parts and different chunks of features. I imagine this new lib would be similar re: feasibility of releasing various different included patches.
   • The mental user overhead of "what is in which package?" applies to devs too -- where was feature X? Where should I put patch Y? Granted, this applies to a single codebase too, just s/package/module/, but it's easier to change things around within a package IMO.
   • It's hard enough just managing multiple release lines -- multiplying that by N packages strikes me as a real headache, even with more folks pitching in. Again, I don't see that there is sufficient need to be worth this.

I also think that we could always revisit this decision later if it does get "too" large or unwieldy to develop/release. It feels easier to go in that direction, than to start out with the complex option and try to consolidate if my fears became reality.

---

Travis, I think it would help if you articulated exactly what is wrong with the single-package approach, aside from the dependencies issue which I addressed up top. That sounded like your main gripe but I suspect it may be more general?

If anybody else wants to pitch in, please do.

[josharian]

My 2c as an end user: I'd rather have something batteries-included. If it includes a little extra crap, so be it. I'd rather know it is going to just work.

[bitprophet]

Discussion ended up back in IRC again, whoops. @tswicegood's main points:

• try/except is an antipattern because it either ignores, or reinvents, pip style version/compatibility resolution. In other words, setup.py is the only good way of specifying that e.g. the Jinja2 integration requires jinja2 >= X.Y, and setup.py does not currently allow for "optional" requirements.
• If we then assume a use of setup.py for dependency resolution, multiple subpackages allows a user to limit the hard requirements they truly need for a project; a single package would require them to install all dependencies. There's currently no either/or -- loose/missing version requirements, or multiple subpackages, pick one.
• Multiple subpackages, in general, is more flexible and should be used to set a good example (cited was how eg Pinax 2 is moving to a multiple-subpackages model.)

My rebuttal:

• That's all entirely accurate.
• However, I don't think there are (or will be) enough dependencies, or enough potential problems with dependency versions, for this to be a deciding factor for Patchwork, and as above that the added complexity costs too much re: speed, user/dev mental models of the project, etc.
  • The jinja2 dependency for Fabric has never been a problem re: versions. Partly because of its stability, but still.
  • The only other optional dependency I can think of is Django, and the level of its API used is extremely high level, making it also not a contentious versioning problem.
  • My vision for patchwork does not include any other obvious dependencies -- it's mostly to be focused on the remote shell commands needed to provision/manage servers (think Chef/Puppet's low level resources.) Meaning a dependency on Fabric alone.

Travis' personal interest in this kind of project appears to be largely aligned with Fabric as a local task runner, which is where many more of the local Python dependencies come into play. I think that would work great as another third-party library, a sibling to Patchwork.

---

So I am overriding his complaint for now, and totally open to refactoring the project at a later date if the dependencies/versioning problem becomes obviously painful.

---

I've registered patchwork on PyPI and will make a repo under fabric/ on Github soon. Need to examine the various "atoms" defined in the 4 "good" libraries mentioned above, plus my current and previous work codebases, to see how they could be organized and/or merged.

Then the hacking begins!

[wraithan]

I was too busy earlier to take part in the discussion, but I am also +1 on a non-metapackage. I don't care about having to install a few dependencies. The significant projects where I would really like a tool like this already have 20-40 dependencies. Likely with overlap with what this would depend on if we add in any additional dependencies.

Once this is up I'll try to pull out all of the good tidbits from my work code base to improve it with anything we have that is useful.

[wraithan]

Wrong button...

[kmike]

Hi there, great discussion! Author of one of the +0 packages is here.

Before releasing django-fab-deploy I also examined most of other packages that existed at that time (about a year ago) and none of them had clear compatibility promises + if my memory does not fail only 1 of them had some tests (unit-style, not integrational-style) - this was woven. For such critical task as deployment this was a non-starter. Even if django-fab-depoy has API that is not very polished (it is originally based on older pre-1.0 version of fabric and so there is a lot of bikeshedding + API is intentionally function-based and this can turn some developers away) I know that it works (and not only for me) because it is tested before each release against VMs with all of supported OS versions and has 90%+ test coverage: this indeed helped to find a bunch of subtle bugs and wrong implications, especially in how commands work together and how OSes differ from each other.

So in my experience the hard part is not the API and abstractions but making scripts reliable, portable, independent but compatible with each other; one must trust her deployment tool and I think that's why there are so many custom scripts and everyone is reinventing the wheel again and again - without clear contract, compatibility information (with OSes and other scripts) and tests it is very often easier to start from scratch and build custom script that will definitely work in given environment.

btw, as for mentioned Django optional dependency - django-fab-deploy doesn't require local django and doesn't use it, it just have a django-minded docs and some defaults that make sense for django projects so I don't think requirements bloat would be the serious issue.

[veselosky]

It's not on par with these packages, but my https://github.com/veselosky/otto-webber might have some useful tidbits. Feel free to pilfer from it if you find anything useful. Unfortunately I have not had time to work on it lately due to $work (which is forcing me to learn Puppet, so I don't know when I'll get back to it either).

[KayEss]

I think that profab is probably too specific to of much interest in the list you're compiling, but here's the link anyway.

https://github.com/Proteus-tech/profab

[bitprophet]

Another challenger has appeared! Specifically, Revolver which attempts to wrap both Fabric and Cuisine and (in the really quick glance I gave it) put a relatively granular high level API on top, including smoothing over some of Fabric's design warts.

So based on that it's probably competition/inspiration for Patchwork; it should probably be added to the list-of-four we have going right now re: stuff to look at when coming up with Patchwork's API.

[bitprophet]

Don't think I actually mentioned here but I stuck up a GH repo for this and it has some real basic proto stuff in it. Currently using Fab 1 to get actual work done, but my hope is to get Fab 2 in shape so Patchwork 1.0 can use that instead. We'll see. Might end up branching it.

https://github.com/fabric/patchwork.

[georgepsarakis]

fabtools seems an excellent abstraction layer for common tasks on top of Fabric. Nevertheless, wouldn't be also nice to have a complete, more complex recipes (I am lending a Chef term here) repository e.g. "LAMP server" etc? Plus, it would probably require an appropriate recipe manager (like apt, pip, npm etc). These user-contributed out-of-the-box solutions would probably make Fabric a more attractive deployment solution.
PS: Needless to say that Fabric is absolutely amazing 👍 .

[bitprophet]

@georgepsarakis That's like a stage 2/3 goal - though to be honest I've never wanted Fabric to actually get too far into config management territory because there's a plethora of other tools that do that much better (Chef, as you mentioned, also Puppet/CFengine/Ansible/Salt). Orchestration type tasks remains the niche Fabric excels at, IMO.

My hope is that Fab 2 will be extensible enough folks can still make such recipe-like structures and share them easily, at any rate.

[andrewsomething]

Any updates on this or pointers to relevant discussion? I'd love to see some relatively "lower" level stuff get included in contrib or a more standardized 'extras' library.

For instance, package management stuff is a place where I end up repeating myself all over the place. It would be great to have fabric/contrib/package_managment/apt.py with stuff like

from fabric.api import sudo
from fabric.context_managers import shell_env

def install(packages, assume_yes=True):
    """
    Install packages via apt.
    """
    if not isinstance(packages, str):
        packages = ' '.join(packages)

    with shell_env(DEBIAN_FRONTEND='noninteractive'):
        if assume_yes:
            sudo('apt-get install -y %s' % packages)
        else:
            sudo('apt-get install %s' % packages)

def update():
# And so on...

Would pull requests for something like that be accepted for contrib? Is something more abstract desirable?

[indera]

@andrewsomething 👍

[bitprophet]

This is still on my mind, especially as Invoke and Fabric 2 develop - e.g. Fabric 2 does away with the contrib tree from 1.x, and given that both local and remote use cases have the same needs re: code sharing, that needs figuring out too.

Presently, there's just two repos for the two, https://github.com/fabric/patchwork and https://github.com/pyinvoke/invocations but there is certain to be issues re: overlap (e.g. running things like aforementioned package management locally vs remotely).

Ideally those will all be solvable with the solution to #98 in 2.0, which right now looks like "rely on the abstract base command-running API, then hand it the appropriate local-or-remote context". Needs hands-on work to solidify, which again may be soon.

[Azulinho]

Come across this thread, while the discussion has taken a different path from the initial issue, I though it would still be relevant to dump this one to the initial cake -> https://github.com/pyBookshelf/bookshelf