find_probable_cause_of_failure() is bad at fetching logs

[coyotemarin]

We currently grab EMR logs from S3. This only works for job flows that shut down after running your job. Technically, it's not supposed to work at all; according to (http://developer.amazonwebservices.com/connect/entry.jspa?externalID=3938&categoryID=265), logs aren't copied to S3 until they've been untouched for 5 minutes.

Rather than grabbing the logs from S3 directly, we need to download the relevant logs via ssh if the job flow is still running, and S3 if it's not, and parse the log files locally.

[irskep]

Would it be more appropriate to call out to the ssh utility or to add a dependency on something like paramiko?

[irskep]

I'm refactoring the S3 and SSH log fetcher functionality to subclass LogFetcher in a new submodule.

from mrjob.logfetch.ssh import SSHLogFetcher
# etc.

This will probably also involve breaking a lot of S3-related code out of EMRJobRunner, which probably isn't a bad thing since that class is currently a couple thousand lines long.

[coyotemarin]

Yup, that sounds good. Another good way to approach this is to start out by building a standalone utility (in mrjob.tools.emr) that fetches and analyzes logs, and then patch it into EMRJobRunner.

And please, use scp; don't add another library dependency. :)

[irskep]

Can do. My current strategy is to copy any relevant functions (ls/get from S3, local, and SSH + dependencies) into instance methods and helpers for fetchers so that logfetch can be used independently. Then I will write a tool around it, verify by hand on various cases, add mocking for SSH + automated tests, and finally insert it into EMRJobRunner, removing redundant functions.

[coyotemarin]

Sounds like a good plan.

[irskep]

New info: logs have slightly different paths on S3 vs local. Here's a quickref I'll put in the comments:

S3 location             Local location
/daemons                / (root)
/jobs                   /history
/node                   <not present>
/steps                  /steps
/task-attempts          /userlogs

[irskep]

I believe this can be closed unless it also encompasses a log fetching/parsing refactor.

[coyotemarin]

Yup, thanks!