I perhaps named this branch badly. Basically, it's a replacement of https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh as a crontabber app.
The diff is large because with this it adds a SubprocessMixin class which is handy for all work with command line things. I like its API. exit_code, out, err = self.run_process(command, input=None)
exit_code, out, err = self.run_process(command, input=None)
I have NOT tested this manually. I don't have pig or hadoop installed and I guess I could make fake programs but that faking would just be as solid as the unit tests.
Please scrutinize the way the backfill date is converted to yesterdays date and inserted into the various commands.
There is a trivial conflict in crons/base.py. git pull rebase -i should be quick.
git pull rebase -i
This is deliberately held back since we don't want to run this on crontabber when crontabber is still synchronous.
@peterbe maybe we should land it but not use it until we have the async stuff figured out?
Uh? You mean to land it but back out this change?
I don't know. By the time we have async crontabber, smedberg will probably have a jydoop replacement for pig ready for production.
fixes bug 848680 - add pig support to crontabber
I only checked for what was blocking merge; I didn't evaluate the content of this. Maybe rhelmer is better suited. I don't know enough about jobs.
@peterbe - this should be suitable for jydoop pretty much as-is so I wouldn't worry too much about that.
Looks like it's good to merge now? I think the proposal is to just land this so we can close it out, but probably won't actually use it until async crontabber is figured out. Running something other than pig should be super trivial and is definitely something we need so there's no reason not to land now AFAICT.
As you can see, it's commented out
@rhelmer please merge