jobs that hit timeout dont return a job.failed=True

[kevinkle]

for example:

 modules.amr.amr.amr('/datastore/2017-07-05-03-40-32-228811-GCA_900016125.1_EF467_contigs_genomic.fna') from backlog_multiples
e582dbda-e5e6-45b8-91f0-b376298158e7
Failed 35 minutes ago
Traceback (most recent call last):
  File "/opt/conda/envs/backend/lib/python2.7/site-packages/rq/worker.py", line 700, in perform_job
    rv = job.perform()
  File "/opt/conda/envs/backend/lib/python2.7/site-packages/rq/job.py", line 500, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "./modules/amr/amr.py", line 21, in amr
    '-o', outputname])
  File "/opt/conda/envs/backend/lib/python2.7/subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/opt/conda/envs/backend/lib/python2.7/subprocess.py", line 1073, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/opt/conda/envs/backend/lib/python2.7/subprocess.py", line 121, in _eintr_retry_call
    return func(*args)
  File "/opt/conda/envs/backend/lib/python2.7/site-packages/rq/timeouts.py", line 51, in handle_death_penalty
    'value ({0} seconds)'.format(self._timeout))
JobTimeoutException: Job exceeded maximum timeout value (600 seconds)

But the if job.failed: doesnt evaluate to true as normally is the case if the actual job fails.

Will have to look for a way to check that job hits timeout.

[kevinkle]

This isn't quite correct. It looks like RQ doesn't check for failure in dependencies when getting a job status.

[kevinkle]

this will be fixed by addressing #146

[kevinkle]

#146 is closed. Closing this as well.