New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors occur when using setup commands #1607

danielchamberlain opened this Issue Jun 14, 2017 · 1 comment


None yet
2 participants

danielchamberlain commented Jun 14, 2017

When I run the job using the following command, my job completes successfully:
python cc_mrjob/ -r emr warc_lists/debug_CC-MAIN-2017-13

However, when I modify the command to include a setup option, it fails:
python cc_mrjob/ -r emr warc_lists/debug_CC-MAIN-2017-13 --setup 'sudo pip install langid'

This error is not because of pip, the following command fails as well:
python cc_mrjob/ -r emr warc_lists/debug_CC-MAIN-2017-13 --setup 'export PYTHONPATH=$PYTHONPATH:cc_mrjob/#'

The stderr for one of my containers is shown below. As best I can tell, each container has either this message or a subset of these lines, sometimes only showing the first line.

/bin/sh: run_prestart: line 1: syntax error: unexpected end of file
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.


This comment has been minimized.


davidmarin commented Jun 15, 2017

Yes this is #1548; basically EMR messed up sh -e on AMI 5.2.0 and later, requiring a special workaround. Upgrading to the latest version of mrjob should fix this issue.

@davidmarin davidmarin closed this Jun 15, 2017

@davidmarin davidmarin added the Bug label Jun 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment