Invalid Certificate For EMR job result of different domain

[ocrickard]

First off, thanks for the work on this! Really looking useful to us.

We've had a problem getting our MRJob onto EMR recently due to this somewhat cryptic error:

Creating Elastic MapReduce job flow
Traceback (most recent call last):
  File "mr_extractor.py", line 32, in <module>
    MREmailExtractor.run()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/job.py", line 483, in run
    mr_job.execute()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/job.py", line 501, in execute
    super(MRJob, self).execute()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/launch.py", line 146, in execute
    self.run_job()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/launch.py", line 207, in run_job
    runner.run()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/runner.py", line 449, in run
    self._run()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/emr.py", line 843, in _run
    self._launch_emr_job()
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/emr.py", line 1435, in _launch_emr_job
    persistent=False, steps=steps)
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/emr.py", line 1200, in _create_job_flow
    self._job_name, self._opts['s3_log_uri'], **args)
  File "/Library/Python/2.7/site-packages/mrjob-0.4_dev-py2.7.egg/mrjob/retry.py", line 80, in call_and_maybe_retry
    return f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/boto/emr/connection.py", line 379, in run_jobflow
    'RunJobFlow', params, RunJobFlowResponse, verb='POST')
  File "/Library/Python/2.7/site-packages/boto/connection.py", line 1048, in get_object
    response = self.make_request(action, params, path, verb)
  File "/Library/Python/2.7/site-packages/boto/connection.py", line 974, in make_request
    return self._mexe(http_request)
  File "/Library/Python/2.7/site-packages/boto/connection.py", line 880, in _mexe
    raise e
boto.https_connection.InvalidCertificateException: Host elasticmapreduce.us-west-1.amazonaws.com returned an invalid certificate (remote hostname "elasticmapreduce.us-west-1.amazonaws.com" does not match certificate): {'notAfter': 'Oct  7 23:59:59 2013 GMT', 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Washington'),), (('localityName', u'Seattle'),), (('organizationName', u'Amazon.com Inc.'),), (('commonName', u'us-west-1.elasticmapreduce.amazonaws.com'),))}
make: *** [emr] Error 1

We eventually figured out that the mapping from our declared "aws_region" in our .mrjob.conf file was being mapped to elasticmapreduce.us-west-1.amazonaws.com, while the certificate was declaring itself us-west-1.elasticmapreduce.amazonaws.com. This mapping is made at:
https://github.com/Yelp/mrjob/blob/master/mrjob/emr.py#L127

We changed this line to read:

    'us-west-1': 'us-west-1.elasticmapreduce.amazonaws.com',

and magically, our jobs are running. I do not know why this is happening, if Amazon has changed their certificates or domain management.

[coyotemarin]

Fixed by #705 (though we're going to fix it even better with #706).

[coyotemarin]

Based on a loose reading of this thread, this is an issue that happens on Python 2.6 but not Python 2.7. I've been able to duplicate this bug with boto 2.11.0 on Python 2.6.7 but not Python 2.7.5.

It looks like the boto version doesn't really matter. The "magic" happens in boto.https_connection.GetValidHostsForCert which reads the `subjectAltName' and 'subject' fields on the certificate; this code hasn't significantly changed since it was added in April 2011 (mrjob requires boto 2.2.0 which was released in Jan 2012). I'm guessing the SSL library on Python 2.6 just doesn't know how to access those fields.

The simple solution would be to just pass validate_certs=False to the EmrConnection constructor when we're on Python 2.6 But that keyword option isn't available until boto 2.5.2. It's also a bit of a blunt instrument; we'd really rather do some SSL validation.

So it sounds like further wrapping the EMR connection is the way to go. sigh At least I'm halfway done coding it. :)