timeout exceptions

[craigbox]

11:01 < alastair> sigh :(
11:01 < alastair> timeoutexception
11:01 < alastair> should really fix that now

(Note: seem to commonly get URLError: <urlopen error timed out> and IOError: [Errno socket error] timed out.)

[shweppsie]

Yup I stll get those a lot too. Often to lastfm and not musicbrainz surprisingly.

[craigbox]

I cared so much I learned how to work both python and git.

(Though I can't guarantee I did either well. :)

[shweppsie]

Tidy up retrying webservice lookup code. Closed by f934232.

• add the ability to do a specified number of retries
• added catch for URLError
• tidied up IOError catch code

[craigbox]

Still failing:

Tue Mar  1 14:40:02 2011: Traceback (most recent call last):
Tue Mar  1 14:40:02 2011: File "/usr/local/bin/renamealbum", line 361, in process_one_path
    renamealbum(srcpath, options)
Tue Mar  1 14:40:02 2011: File "/usr/local/bin/renamealbum", line 694, in renamealbum
    (release, method) = get_musicbrainz_release (disc, options)
Tue Mar  1 14:40:02 2011: File "/usr/local/bin/renamealbum", line 273, in get_musicbrainz_release
    return (get_release_by_fingerprints(disc), "fingerprint")
Tue Mar  1 14:40:02 2011: File "/usr/local/bin/renamealbum", line 96, in get_release_by_fingerprints
    dirinfo = albumidentify.get_dir_info(disc.dirname)
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/albumidentify.py", line 56, in get_dir_info
    if lastpuid is not None and trackinfo[fname].getPUID() == lastpuid:
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/musicfile.py", line 67, in getPUID
    self._fetchPUID()
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/musicfile.py", line 34, in _fetchPUID
    musicdns_key)
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/memocache.py", line 68, in memoify_replacement
    ret=func(*args,**kwargs)
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/lookups.py", line 59, in backoff_func
    return func(*args,**kwargs)
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/lookups.py", line 116, in delay
    ret=func(*args,**kwargs)
Tue Mar  1 14:40:02 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/musicdns.py", line 44, in lookup_fingerprint
    f = urllib.urlopen(url, data)
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/urllib.py", line 88, in urlopen
    return opener.open(url, data)
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/urllib.py", line 207, in open
    return getattr(self, name)(url, data)
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/urllib.py", line 344, in open_http
    h.endheaders()
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
    self._send_output()
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
    self.send(msg)
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/httplib.py", line 735, in send
    self.connect()
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/httplib.py", line 716, in connect
    self.timeout)
Tue Mar  1 14:40:02 2011: File "/usr/lib/python2.6/socket.py", line 514, in create_connection
    raise error, msg
Tue Mar  1 14:40:02 2011: IOError: [Errno socket error] timed out
Tue Mar  1 14:40:02 2011: fail!

[shweppsie]

Right so it's a socket error. It's probably safe to catch all socket exceptions and just retry on them. I wasn't so keen to do that on IO Errors.

[shweppsie]

urllib bug turns the socket error into a malformed IOError instead. http://bugs.python.org/issue6471

Hopefully this commit will finally end this: 27ef1de

[craigbox]

so close!

Tue Mar  1 22:07:03 2011: File "/usr/local/lib/python2.6/dist-packages/renamealbum/lookups.py", line 73, in backoff_func
    util.update_progress("Caught %s error. Retrying in 20s..." % e.reason)
Tue Mar  1 22:07:03 2011: AttributeError: 'HTTPError' object has no attribute 'reason'

[shweppsie]

Hmm line 73 is a comment here. Can you send the full stacktrace?

[craigbox]

Sorry, updated copy is line 77.

Traceback (most recent call last):
File "/usr/local/bin/renamealbum", line 361, in process_one_path
    renamealbum(srcpath, options)
File "/usr/local/bin/renamealbum", line 767, in renamealbum
    (srcfiles, destfiles, needs_replaygain) = name_album (disc, release, srcpath, options, imagemime, imagepath)
File "/usr/local/bin/renamealbum", line 991, in name_album
    md_lastfm.get_tags(tags, release, mbtrack, track_artist)
File "/usr/local/lib/python2.6/dist-packages/renamealbum/md_lastfm.py", line 12, in get_tags
    track_tags =  lastfm.get_track_toptags(artistname,mbtrack.title,mbtrack.id)
File "/usr/local/lib/python2.6/dist-packages/renamealbum/lastfm.py", line 81, in get_track_toptags
    trackid=mbtrackid)
File "/usr/local/lib/python2.6/dist-packages/renamealbum/lastfm.py", line 65, in _do_lastfm_query
    return _do_raw_lastfm_query(url)
File "/usr/local/lib/python2.6/dist-packages/renamealbum/memocache.py", line 68, in memoify_replacement
    ret=func(*args,**kwargs)
File "/usr/local/lib/python2.6/dist-packages/renamealbum/lookups.py", line 77, in backoff_func
    util.update_progress("Caught %s error. Retrying in 20s..." % e.reason)
AttributeError: 'HTTPError' object has no attribute 'reason'
fail!

urllib2 docs suggest HTTPError is a subclass of URLError, and the exception handler for URLError is catching it. However, there's no reason attribute? There is a code, and in my particular case, it's a 400. The error is trying to talk to last.fm - does it gets per-track data that might be corrupt at their end?

[shweppsie]

Wow totally loving the consistency of urllib exceptions. So this time it's HTTPError.msg not reason even though it inherits from URLError where it's reason.

I do have concerns about it being a 400 though. That usually implies the request could not be understood by the server and technically shouldn't be retried on. This probably means it won't fix the problem, you'll just get 3 400s and then it'll fail. I've changed it to print out the URL it's trying and then re-raise the exception. This way we can hopefully nail down why it's causing a 400.

[Pyrosfr]

first tries with fresh install on gentoo x64 and got timeout issues too

Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/albumidentify.py", line 56, in get_dir_info
if lastpuid is not None and trackinfo[fname].getPUID() == lastpuid:
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/musicfile.py", line 67, in getPUID
self._fetchPUID()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/musicfile.py", line 34, in _fetchPUID
musicdns_key)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/memocache.py", line 68, in memoify_replacement
ret=func(_args,__kwargs)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/lookups.py", line 61, in backoff_func
return func(_args,**kwargs)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/lookups.py", line 129, in delay
ret=func(*args,**kwargs)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/site-packages/renamealbum/musicdns.py", line 44, in lookup_fingerprint
f = urllib.urlopen(url, data)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/urllib.py", line 86, in urlopen
return opener.open(url, data)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/urllib.py", line 207, in open
return getattr(self, name)(url, data)
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/urllib.py", line 343, in open_http
errcode, errmsg, headers = h.getreply()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/httplib.py", line 1085, in getreply
response = self._conn.getresponse()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/httplib.py", line 1013, in getresponse
response.begin()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/httplib.py", line 402, in begin
version, status, reason = self._read_status()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/httplib.py", line 360, in _read_status
line = self.fp.readline()
Mon Apr 11 17:17:09 2011: File "/usr/lib64/python2.7/socket.py", line 430, in readline
data = recv(1)
Mon Apr 11 17:17:09 2011: IOError: [Errno socket error] timed out
Mon Apr 11 17:17:09 2011: fail!