Skip to content

bug 848680 - add pig support to crontabber #1121

Merged
merged 1 commit into from May 14, 2013

3 participants

@peterbe
peterbe commented Mar 8, 2013

I perhaps named this branch badly. Basically, it's a replacement of https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh as a crontabber app.

The diff is large because with this it adds a SubprocessMixin class which is handy for all work with command line things. I like its API. exit_code, out, err = self.run_process(command, input=None)

I have NOT tested this manually. I don't have pig or hadoop installed and I guess I could make fake programs but that faking would just be as solid as the unit tests.

Please scrutinize the way the backfill date is converted to yesterdays date and inserted into the various commands.

@rhelmer rhelmer was assigned Mar 27, 2013
@lonnen
Mozilla member
lonnen commented Apr 3, 2013

There is a trivial conflict in crons/base.py. git pull rebase -i should be quick.

@peterbe
peterbe commented Apr 10, 2013

This is deliberately held back since we don't want to run this on crontabber when crontabber is still synchronous.

@rhelmer
Mozilla member
rhelmer commented May 3, 2013

@peterbe maybe we should land it but not use it until we have the async stuff figured out?

@peterbe
peterbe commented May 3, 2013

Uh? You mean to land it but back out this change?
https://github.com/mozilla/socorro/pull/1121/files#L1R51

I don't know. By the time we have async crontabber, smedberg will probably have a jydoop replacement for pig ready for production.

@lonnen
Mozilla member
lonnen commented May 14, 2013

I only checked for what was blocking merge; I didn't evaluate the content of this. Maybe rhelmer is better suited. I don't know enough about :pig: jobs.

@rhelmer
Mozilla member
rhelmer commented May 14, 2013

@peterbe - this should be suitable for jydoop pretty much as-is so I wouldn't worry too much about that.

Looks like it's good to merge now? I think the proposal is to just land this so we can close it out, but probably won't actually use it until async crontabber is figured out. Running something other than pig should be super trivial and is definitely something we need so there's no reason not to land now AFAICT.

@peterbe
peterbe commented May 14, 2013

As you can see, it's commented out
https://github.com/mozilla/socorro/pull/1121/files#L1R52
@rhelmer please merge

@rhelmer rhelmer merged commit 2d1f2ea into mozilla:master May 14, 2013

1 check passed

Details default Jenkins build 'socorro-github' #796 has succeeded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.