setup wrapper scripts don't work on AMI 5.2.0+

[coyotemarin]

This command:

python -m mrjob.examples.mr_word_freq_count LICENSE.txt -r emr --image-version 5.2.1 --setup true

fails with the following task error:

/bin/sh: run_prestart: line 1: syntax error: unexpected end of file

This doesn't happen without the setup command, so I'm guess it's something to do with /bin/sh?

[coyotemarin]

Huh, simply SSHing in and running the offending mapper command (/bin/sh -ex setup-wrapper.sh python2.7 mr_word_freq_count.py --step-num=0 --mapper) doesn't cause this problem.

Maybe there's a bad .bashrc file or something? /bin/sh is bash on the 5.2.1 AMI, at least.

[coyotemarin]

Here's the error if you use --bin-sh 'sh -vex':

run_prestart () {  su -s /bin/bash $SVC_USER -c "cd $WORKING_DIR && $EXEC_PATH --config '$CONF_DIR' start $DAEMON_FLAGS"
/bin/sh: run_prestart: line 1: syntax error: unexpected end of file

[coyotemarin]

run_prestart() is defined not in .bashrc, but in several /etc/init/hadoop-*.conf files. Basically, they aren't counting on us trying to get bash to work in sh emulation mode. Using bash as our shell should work as a workaround.

[coyotemarin]

Huh, no, using /bin/bash doesn't solve it. Bourne shell emulation is pretty limited (mostly not loading .bashrc, I think).

[coyotemarin]

--sh-bin '/bin/sh -x (no -e option) fixes it.

[coyotemarin]

It looks like there is something about the definition of run_prestart that angers bash -e. I'm trying to figure out if there's a way to blank it out. --cmdenv run_prestart= doesn't seem to do it, and there doesn't seem to be a way to instruct bash to ignore exported functions.

By the way, it looks like /bin/sh is bash on the 4.7.1 AMIs too. It's not a bash issue, it's a config issue.

[coyotemarin]

I think the right approach is to patch set -e back in by prepending it to the setup commands. For example, this command (with --setup false) fails:

python -m mrjob.examples.mr_word_freq_count LICENSE.txt -r emr --image-version 5.2.1 --setup 'set -e' --setup false --sh-bin '/bin/bash -x'

but this command (with --setup true) succeeds:

python -m mrjob.examples.mr_word_freq_count LICENSE.txt -r emr --image-version 5.2.1 --setup 'set -e' --setup true --sh-bin '/bin/bash -x'

Currently, we reference self._opts['sh_bin'] directly; we'd have to reference it through a method instead.

[coyotemarin]

Sent an email to the EMR product team so that there might eventually be an AMI version where we don't have to apply a workaround.

[coyotemarin]

I think this affects tasks and master node setup scripts, but not the bootstrap script (since that's not run inside Hadoop). Probably easier to just implement this for everything. There's little practical difference between running /bin/sh -ex and running /bin/sh -x on a script where the first line is set -e.

[coyotemarin]

Finished writing a complex but largely transparent workaround. sh_bin is not terribly well-tested, so it's going to take a little while to add proper unit tests.