Support for shared processors in repos

[timsutton]

Some people are writing more general-purpose processors like importers, downloaders, etc. We don't currently have a mechanism to have autopkg install and read these processors from some common area, so that somebody could maintain a processor outside of the AutoPkg core and yet still have it easily usable by others' recipes. Decided to open an issue to try and propose a solution we can test out.

One idea discussed today in IRC: support the placing of custom processors in a special directory like Processors (?) within a recipe repo which can contain these, and they would be loaded as additional Python modules available to AutoPkg. If someone wanted to keep a separate repo for just for one or more custom processors that should also be fine.

Currently the autopkglib and a recipe's folder are searched for processors, and so this would need to extend to searching a certain reserved location within each item in RECIPE_SEARCH_DIRS.

The issue of namespacing came up - there would be a conflict if more than one person writes a processor with the same name and both try to get imported.

Since we now use a Process step's Processor key to take the module name, perhaps we support namespacing within this repo, in a form like: com.github.autopkg.timsutton-recipes:MyGreatProcessor. The actual module to load is prefixed by the dirname of the recipe repo.

I'm not exactly sure how this should work if I were giving the --search-dir options, however. The --search-dir given could be a repo or not, and if it is, we could only possibly infer that it's the same repo by actually and reading the repo config in .git/config, which seems a bit clunky.

We could also still support searching all RECIPE_SEARCH_DIRS if a non-namespaced value is given for Processor, with log feedback about thedir (if any) in which the processor was found, but as most people using third-party shared processors should be namespacing them, it means we need to have a reliable way to determine whether a search path corresponds to this namespace. What I just outlined seems like it could work, but perhaps there are more elegant, cleaner ways.

All of this would also mean we should have some kind of RequiredRepos-like key with which a recipe can specify other repos that it requires in order to run - but that is an issue that can be addressed in parallel.

[jessepeterson]

Also note #38 for some (short) historical thoughts on this.

[timsutton]

Implemented for 0.4.0.

Relevant discussion here:

https://groups.google.com/d/msg/autopkg-discuss/Wi867GXaOl8/l7mrnt51yRQJ