Split out non-ssh-dependent features into separate lib

[bitprophet]

Things are coming to a head and it'd be good to split out Fabric's task execution stuff into its own "third party" tool/library so it can be used/referenced independently of our SSH functionality.

Right now, anybody wanting to use Fab-as-runner must still install ssh and PyCrypto, which sucks.

And if we're splitting it between task running and SSH, having "Fabric" be "SSH + dependency on new runner tool" makes much more sense (both re: backwards compatibility, and overall usefulness) than vice versa.

Speaking of backwards compat, I am marking this 2.0 because it makes more sense to do it at a 2.0 backwards incompat barrier (since at the very least it adds a new install dependency to Fabric), but doing the split in, say, 1.6 or 1.7 should also be quite possible if the timing is better.

---

To be clear, this new tool would:

• Maybe, possibly, but probably not just be us glomping onto an existing tool like Paver
  • Paver tries to do too much and I've never been a big fan of how its API feels
  • Really not aware of any other tools that are at all well known and fit the use case any better
  • EDIT: Baker actually looks half decent, though it's obviously not a perfect match (nothing would be, anything would require some tweaks.)
• Have a distinct identity from Fabric, while probably remaining "affiliated"
  • Name brainstorm incoming.
• Encompass the "run Python callables as tasks from the CLI with args" functionality that currently exists within Fabric
• Likely entail some refactoring of how that machinery works, if only just to make post-ripout integration easier
• Probably get some of the remaining big task-runner "missing features" implemented right off the bat (really just Task dependencies / before/after hooks #452)

[kennethreitz]

🍰

[kennethreitz]

A totally new tool that would fit the use case well is always an option too, of course.

[bitprophet]

Updated description a bunch.

@kennethreitz: this really would kinda be a new "thing", just based on the existing needs/feature set of that half of Fabric. The idea is for it to be its own entity re: installation, development schedule etc, but still controlled by the Fabric devs/community.

[dstufft]

Ideas:

• tony - as in Tony Masters aka The Taskmaster (comic books~)
• CEO - Runs things, (or czar or something similar along that lines)

Sorry I suck at names :(

[dstufft]

Also things like "kirk" (as in captain kirk).

I tried to think of something fabric related that has to do with running tasks but I can't think of one that ties in with tasks + fabric.

But Loom isn't a bad name either I don't think.

[bitprophet]

Initial name brainstorm.

Concepts

• Running/jogging/etc
• Execution (of orders, people perhaps though that's a bit macabre, etc)
• Tasks (as in things-to-do, errands etc)
• Maybe some of the rejected or similar names from External fabric utility library #461, insofar as the idea of weaving or putting things together from fabric also kind of applies to tasks

Names not taken on PyPI

• runner
• executor
• go
  • Confusion with the programming language and/or board game, esp if the former has a binary specifically called go
  • For better or worse, has many potential silly task names such as away or fuck_yourself or bother_somebody_else
• weaver

Binary names

Thanks, Oxford American Writer's Thesaurus as presented by OS X's Dictionary application!

• run
• execute
• go
  • See above
• create (like make)
• fabricate (too long, too easily confused with fab)
• fab (i.e. have this new lib named differently but still installing a fab binary; if both present on system the Fabric one takes precedence? Seems like a dumb idea.)
• generate
• produce
• fashion (meh)
• build
• devise
• shape
• setup
• weave

---

Not in top brainstorming form today. Want moar.

[bitprophet]

@dstufft Loom's an OK name, and CEO is a mostly-ok concept (not a big fan of the comic book reference, if only because it seems obscure, I'd never heard of the character ;)). Main problem is coming up with a good binary name, I feel like a verb is pretty much a must for it to work well. Not sure what would work for either of those.

[kennethreitz]

Boss: Runs things.

[kennethreitz]

partake. Kinda like Make, but for Python.

[kennethreitz]

Executive.

[kennethreitz]

I like the idea of it orienting around tasks. Hmm..

[bitprophet]

@kennethreitz I like Boss, other than the potential confusion with a certain New Jersey native (sorry, just not a fan, bad roommate experience in college)

As I said to @dstufft, binary names are definitely key here, so if you can, please try to keep those in mind too. A great project name doesn't help if the project-name-as-binary sounds kinda dumb ;)

[kennethreitz]

Maid is interesting. Performs your tasks for you. Similar to "Make".

Butler too. We could pick a random british-sounding butler name.

[kennethreitz]

maid clean, maid release, maid test. I like it.

[bitprophet]

Aside from impinging upon my (boundless, naturally) masculinity, maid is pretty good, even though it doesn't fit the verb pattern. (I.e. it is one of the better non-verb binary names.) (EDIT: and what do butlers do? Butle?)

[kennethreitz]

I just squatted the name, just in case. I like it a lot so far.

[dstufft]

Boss as a library name is already taken FWIW

On Tuesday, February 21, 2012 at 8:38 PM, Jeff Forcier wrote:

Aside from impinging upon my (boundless, naturally) masculinity, maid is pretty good, even though it doesn't fit the verb pattern. (I.e. it is one of the better non-verb binary names.)

---

Reply to this email directly or view it on GitHub:
#565 (comment)

[kennethreitz]

intern is also an incredibly awesome choice.

[bitprophet]

@dstufft So it is, and I couldn't think of a good binary name for that either, boss docs doesn't really fit well.

[bitprophet]

@kennethreitz mentioned bake on IRC, which does fit the make/rake theme, and also kind of fits as an invoke verb (eg $ bake docs, though thematically it's a little off, feels like it'd fit better in Chef or something.

EDIT: also deliver. My initial reaction to this one was somewhat positive ($ deliver build $ deliver docs) but there's no great project name to go along with it that isn't a little too silly, and it's a little long.

[bitprophet]

$ invoke taskname (project names: invoker, invoked, ???)

[kennethreitz]

Invoked.

[dstufft]

I really like Invoked and invoke docs invoke tests invoke publish etc.

[dstufft]

Project name could just be "Invoke" too, does the binary and the project need different names?

[kennethreitz]

I think 'invoked' has a bit more pizzazz to it than 'invoke', but either could work.

[bitprophet]

Name doesn't have to differ, no, it's just that many times it makes more sense to.

---

I had a thought while walking to the train (on said train now :P go go smartphone tethering!) which is that we could possibly skip all this and simply move the entire fab + fabfile.py side of things to this "new" library (and call it, say, fabricator).

In other words, having one callable for the standalone lib and a different one for the SSH-using setup doesn't necessarily have to happen, and could even be a detriment.

Pluses:

• No need for a new binary name
• Having two binary names would be confusing anyways
• If the new lib didn't use "fabfiles", we'd have two different "task file" types too, again confusing/extra code/etc

Minuses:

• Potential confusion between the two projects due to similar names
• New project would have to be extra extensible to allow all the Fabric-isms (e.g. almost all of the CLI flags, etc) to continue working as-is
  • I.e. pip install fabricator gives us fab, which would have some smallish set of flags
  • Then if you pip install fabric, suddenly the same fab has to exhibit all the additional Fabric CLI flags, if not other things too
  • OTOH it's not like allowing "client" code to extend CLI parsing is a bad thing, it's just more work that has to be done up front.

[dcolish]

You can have a command line client become extensible. Look at how nose allows plugin entrypoints to be expressed from its command line tool

[dstufft]

I think the extra extensibility would be a good thing regardless of which way it goes.

The name confusion would be an issue I think.

This is a personal thing but I like the look of invoke test invoke docs better than fab test.

As far as the different binaries go, I would just make sure the core binary (invoke) is extensible to allow all the fabric things to happen, and then have fab just call the same entry point as invoke (i.e. the actual fab/invoke binaries would just be ultra minimalistic callers of a main() (or whatever).

I'd think if you go the invoke + fabric route that pre 2.0 both fabfiles and "Invokefiles"? would be supported, and then with 2.0 just Invoke files?

So essentially i'm +1 for splitting it into distinct "task runner" and "remote/ssh stuff", putting in compat shims for 1.x, and deprecating stuff for 2.x. This split I think would make both libs better, the task runner would support some really nice extensability (and probably external tasks that can easily be depended on, ran etc). And the remote/ssh stuff would benefit from having a smaller set of responsibilities allowing it to be more focused/

My cents anyways.

[dstufft]

I should clarify I don't hate the other way, I'm happy that this discussion is happening at all :)

[bitprophet]

Well it's not like you'd have to type it very frequently, eh? ;) But yes, tasks/ or tasks.py is a bit more globally obvious.

[kennethreitz]

taskfile.py

[dstufft]

you invoke tasks(.py|/), the wording is nice anyways :)

[bitprophet]

Yea, the "these are tasks which you invoke" setup works very nicely from the perspective of tying the keywords in with English descriptions of how it works. It's obvious, no frills, no conceit, etc.

@kennethreitz I've never really been a big fan of XYZFile as a naming convention, and especially since these days it's frequently not a single file, I see no need to continue the convention outside of Fab 1.x :)

[askedrelic]

I would totally love to see this happen. I've always viewed Fabric as a generic Pythonic way to store "tasks" for a project, whether they are local or remote based, something closer to Rake/Make. However, when trying to intro to new people, several people were confused on this viewpoint and viewed it only as a deployment tool.

I would second the module/folder aspect, toward having it be easier to make generic tasks that are drop-in able. New style tasks definitely went a good way toward that.

[bitprophet]

@askedrelic Yup, having it be two-things-in-one has always been a bit confusing unfortunately.

---

Unrelated: I summon @tswicegood to this thread since he and I were discussing the issue a few weeks ago and I figure he might want to weigh in. As usual I do reserve the right to disagree ;)

[tswicegood]

I was summoned? :-)

I like the idea of moving over to tasks as the module rather than fabfile. Makes more sense and is shorter. I, for one, hate having to install the full dep stack just to be able to run fab test inside Armstrong, so anything that could be done to extract that I'm a big plus one on.

As far as naming, what about shifting from having an external executable that's installed to having people add this to the bottom:

if __name__ == "__main__":
    <some_mod>.main()

Then you just use Python. I'm ok with an executable, just don't know if its necessary.

FWIW, I started a project a few weeks back for handling package creation called flunky. Sticking with that attendant theme, footman or lacky are both available (the latter has a conflict with lackie -- a Ruby project). Another one would be Friday, as in girl/man friday. It passes the GitHub/PyPI/Homebrew test in that there's nothing named that. Actually, now that I think about it that would be my first choice.

Side bet, I wonder how long it would be until @niran submitted a PR updating the README to include the obvious, albeit horrible theme song.

[bitprophet]

I think having an on-your-$PATH "binary" (esp. given that it's currently not a real binary but just a setuptools-created stub that calls (invoked|fabric).main()) is the most user-friendly, and I don't see any benefits to moving the contents of that stub to be autogenerated inside the tasks module (and in fact see a number of downsides to it.)

And yes, the entire point of this is to end up with a lib that lets you get the tasky stuff without needing any SSH/network/crypto deps :)

[dstufft]

I completely agree with @bitprophet on the executable. It makes it way easier to run. If someone wants to use something other than invoke / fab on their personal project it's just a stub so you can easily create your own binary (and it could easily live at finalname.__main__.main, in which case you could also just python -m finalname if you wanted too.

[kennethreitz]

+1.

[tswicegood]

Just throwing an idea out there, I think it's more convenient too. FWIW, I registered Friday on PyPI if you want to use it.

[dcolish]

@bitprophet Sorry, I figured you knew about how to use entrypoints. I was voting, in my own way, for doing a frontend in a fashion similar to nose and keeping the name as fab.

[bitprophet]

Ideas are great, I love ideas :)

Friday's a little too on the cutesy side, though I do like the pop culture reference. That song is so great!

---

I think for now I'm going with straight up invoke as both the package/project, and binary, name. Yoink! Invoker or Invoked were other possibilities for the former (i.e. still with an invoke binary) but I couldn't think of any good reason not to go the straightforward route.

The main downside of this name is the generic-ness, but that's a problem with every project I've ever taken over or created, and in this case none of the more specific options felt right to me. /justification

[bitprophet]

@dcolish Acknowledged, thanks!

---

I've not made a final decision yet, but I think at this point the sensible approach is:

• Invoke is its own thing, own new identity/"brand", uses invoke binary, looks for tasks module, etc.
  • This is clean, straightforward conceptually for users only using Invoke and not Fabric, removes historical baggage, etc
• Invoke exposes any shared code that Fabric (or other tools) might want to use, as public APIs
• Fabric continues to use fab and look for a fabfile module, while adding an install-time dependency on Invoke, and uses Invoke's APIs to call that shared code
  • This does mean that installing Fabric results in two binaries, but from the perspective of users focusing on Fabric, they don't need to know that invoke/tasks exist/are options, because they'd never be using them -- fab is still a superset of functionality.
  • Users who want to use both Invoke for its own sake (e.g. in other projects which have no Fabfiles) and Fabric are the sticking point, re: how invoke and fabric relate to one another and behave. These folks might have some interest in invoke and fab being simple aliases of each other (i.e. having Fabric's install mutate how invoke behaves, like how Nose plugins work.)
  • However, I think it's best to have the two behave differently -- fab must differentiate at least in the "what name of task module do I look for?" sense, at which point we may as well have it be its own creature, and the relation between Fabric and Invoke is only in how Fabric uses Invoke's codebase.

EDIT: I should take a look at how this will work, quickly, before making anything final. Practice vs theory and all that. May run into angles I am not thinking about here.

[bitprophet]

Invoke has a beta release now (after lots of recent work building it up). Already in:

• Better, more explicit but still almost zero-boilerplate namespacing
• "Real" CLI flag parser, no more foo:bar,lol, including auto translation of function argspec into CLI spec
• Attempted to design task execution bits to be extensible/overrideable re: parallelization
• Which was done to support: pre-tasks (indicate tasks to be run before other tasks)
• capture-and-print capable local task runner run (albeit probably not functional on Windows)
• Might be forgetting another couple bits?

Sprinting on it and hoping to start prototyping how Fab 2 will bolt onto it, this week.

[steverweber]

ensure clean
ensure built
"ensure" "state"

[mvaled]

What's the status of fabric 2? Is there a fork out there that is covering this? I mean the stuff that fabric2 should cover that is not invoke.

[bitprophet]

@mvaled It'll appear shortly, has been in a private 'clean' branch of the fabric repo.

[dmr]

Sorry if this if off topic but I sort of have to ask again:
Where can I find fabric 2?
I really like invoke and it feels like a huge improvement, but what is missing for an invoke 1.0 release? (described as the basis for fabric 2)

Thanks for all the great work on invoke and fabric!

[bitprophet]

@dmr I'm in the middle of getting a Fabric 2 working on top of Invoke. A bunch of features to be ported over from Fabric 1 will inform design decisions for Invoke (for new features or existing ones - e.g. the recent config overhaul was one of these), so I don't want to label Invoke 1.0 until it's been "battle tested" sufficiently.

My plan is to have an alpha of Fabric 2 ready for the public by PyCon at the latest (ideally much sooner) and that will lead into Fabric 2.0 + Invoke 1.0.

[dmr]

Thank you for the clarification, where can I help?

[bitprophet]

I'm working on it in a private branch for the short term until I get something I'm happy with and will be announcing on twitter/mailing list/etc once I'm ready for feedback, so please stay tuned.

[dmr]

I started switching to invoke in combination with shell commands and some Paramiko because fabric is the last package without python3 support in my project

[tebanep]

Any ETA for Fabric 2.0 and Invoke 1.0 as Python 3.5 was released yesterday and will be the default in Ubuntu 16.04 LTS?