Expand regex to account for INFO: being prepended to logs #421

Merged
merged 1 commit into from Apr 9, 2012

Conversation

Projects
None yet
2 participants
@danielhfrank
Contributor

danielhfrank commented Mar 30, 2012

In hadoop.py, HADOOP_JOB_TIMESTAMP_RE is used to parse output from hadoop to determine the job's timestamp and step number. In my usage, I found that "INFO: " was being prepended to hadoop's log output (possibly due to my own failure to set up the logs correctly), and the regex was failing to pick the line up. This resulted in mrjob raising an error despite the job apparently running fine otherwise.

I simply added an optional check for "INFO: " in this regex. It's possible that something more generic would be desirable, but this seems to fix things for me and I don't think it would screw things up for anyone else.

(fwiw, I am using hadoop 0.20.2 and python 2.5, but the original regex seemed to fail with python 2.7 as well)

@irskep

This comment has been minimized.

Show comment Hide comment
@irskep

irskep Mar 30, 2012

Contributor

Thanks, and you added the r so the escapes are correct! Dave: recommend I pull this into the release_033 branch.

Contributor

irskep commented Mar 30, 2012

Thanks, and you added the r so the escapes are correct! Dave: recommend I pull this into the release_033 branch.

irskep added a commit that referenced this pull request Apr 9, 2012

Merge pull request #421 from danielhfrank/hadoop_job_timestamp_re
Expand regex to account for INFO: being prepended to logs

@irskep irskep merged commit dcfb30b into Yelp:master Apr 9, 2012

@irskep

This comment has been minimized.

Show comment Hide comment
@irskep

irskep Apr 9, 2012

Contributor

Merged, thanks!

Contributor

irskep commented Apr 9, 2012

Merged, thanks!

@danielhfrank

This comment has been minimized.

Show comment Hide comment
@danielhfrank

danielhfrank Apr 9, 2012

Contributor

sweet!

Contributor

danielhfrank commented Apr 9, 2012

sweet!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment