In DataImportTask properly catch exceptions raised by util.has_url_been_updated_since_start_of_celpp_week()

[coleslaw481]

A celpp run failed on production. Output from celpprunner.log:

2017-10-20 21:24:08,210 DEBUG d3r.celpp.dataimport Try #1 http://www.wwpdb.org/files/new_release_structure_nonpolymer.tsv has not been updated. Sleeping 1200 seconds
2017-10-20 21:44:48,354 ERROR d3r.celpp.dataimport Caught Exception trying to download file(s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/d3r/celpp/dataimport.py", line 361, in _download_files
    self._wait_for_url_to_be_updated(nonpolyurl)
  File "/usr/lib/python2.7/site-packages/d3r/celpp/dataimport.py", line 315, in _wait_for_url_to_be_updated
    val = util.has_url_been_updated_since_start_of_celpp_week(url)
  File "/usr/lib/python2.7/site-packages/d3r/celpp/util.py", line 243, in has_url_been_updated_since_start_of_celpp_week
    resp = urllib2.urlopen(req)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -2] Name or service not known>
2017-10-20 21:44:48,469 ERROR d3r.celpprunner Error caught exception
Traceback (most recent call last):
  File "/usr/bin/celpprunner.py", line 182, in run_tasks
    task.run()
  File "/usr/lib/python2.7/site-packages/d3r/celpp/dataimport.py", line 483, in run
    if self._download_files(url) is False:
  File "/usr/lib/python2.7/site-packages/d3r/celpp/dataimport.py", line 408, in _download_files
    url + ' to ' + download_path)
UnboundLocalError: local variable 'download_path' referenced before assignment
2017-10-20 21:44:48,469 DEBUG d3r.celpprunner Task dataimport has finished running  with status start
2017-10-20 21:44:48,469 ERROR d3r.celpprunner Error running task dataimport Caught Exception running task: local variable 'download_path' referenced before assignment

Looking at code it turns out the exception was raised from within DataImportTask._wait_for_url_to_be_updated() method and within that method the method
util.has_url_been_updated_since_start_of_celpp_week() had raised the URLError.

This is a bug. The code in _wait_for_url_to_be_updated() should catch this exception and continue with the while loop assuming the update is false.

[coleslaw481]

Fixed in 6b8aa77..048f62b chrisdev -> chrisdev
by adding try catch around util.has_url_been_updated_since_start_of_celpp_week call.