Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP errors cause the Mopidy actor to die #11

Closed
jcass77 opened this issue Jun 26, 2015 · 7 comments
Closed

HTTP errors cause the Mopidy actor to die #11

jcass77 opened this issue Jun 26, 2015 · 7 comments

Comments

@jcass77
Copy link
Member

jcass77 commented Jun 26, 2015

HTTP connection and timeout errors raise exceptions which are not currently handled. This sometimes causes the Mopidy backend to die unexpectedly:

2015-06-26 03:39:56,619 - ERROR Unhandled exception in Core (urn:uuid:dc95f4be-887d-4bcf-a332-2d6cced5309f): Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 200, in _actor_loop response = self._handle_receive(message) File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 294, in _handle_receive return callee(*message['args'], **message['kwargs']) File "/var/lib/git/mopidy/mopidy/listener.py", line 49, in on_event getattr(self, event)(**kwargs) File "/var/lib/git/mopidy/mopidy/core/actor.py", line 87, in reached_end_of_stream self.playback._on_end_of_track() File "/var/lib/git/mopidy/mopidy/core/playback.py", line 217, in _on_end_of_track self._change_track(next_tl_track) File "/var/lib/git/mopidy/mopidy/core/playback.py", line 199, in _change_track self._play(on_error_step=on_error_step) File "/var/lib/git/mopidy/mopidy/core/playback.py", line 313, in _play backend.playback.change_track(tl_track.track).get() and File "/usr/local/lib/python2.7/dist-packages/pykka/future.py", line 299, in get exec('raise exc_info[0], exc_info[1], exc_info[2]') File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 200, in _actor_loop response = self._handle_receive(message) File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 294, in _handle_receive return callee(*message['args'], **message['kwargs']) File "/var/lib/git/mopidy-pandora/mopidy_pandora/actor.py", line 60, in change_track pandora_track = self._next_track(track_uri.station_token) File "/var/lib/git/mopidy-pandora/mopidy_pandora/actor.py", line 50, in _next_track self.tracks = self.backend.api.get_playlist(station_token) File "/var/lib/git/mopidy-pandora/mopidy_pandora/pydora.py", line 10, in with_authentication return f(self, *args, **kwargs) File "/var/lib/git/mopidy-pandora/mopidy_pandora/pydora.py", line 66, in get_playlist for station in self.api.get_playlist(station_token)['items']) File "/var/lib/git/pydora/pandora/__init__.py", line 308, in get_playlist includeTrackLength=True) File "/var/lib/git/pydora/pandora/__init__.py", line 172, in __call__ result = self._make_http_request(url, data) File "/var/lib/git/pydora/pandora/__init__.py", line 130, in _make_http_request return urlopen(req) File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 401, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 419, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1219, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open raise URLError(err) URLError: <urlopen error [Errno 110] Connection timed out> 2015-06-26 03:39:56,724 - WARNING Tried to communicate with dead actor. 2015-06-26 03:39:56,729 - WARNING Tried to communicate with dead actor. 2015-06-26 03:39:56,733 - WARNING Tried to communicate with dead actor. 2015-06-26 03:39:56,737 - WARNING Tried to communicate with dead actor. 2015-06-26 03:39:56,740 - WARNING Tried to communicate with dead actor.

@jcass77
Copy link
Member Author

jcass77 commented Jun 26, 2015

I've created PR #12 which contains a potential solution, but it won't work until mopidy/mopidy#17 has been fixed for repeat mode as well (also see mopidy/mopidy#1221).

...made worse by: https://github.com/mopidy/mopidy/blob/749c8baceb077f8a5155e1dab33b7154ab1ba481/mopidy/core/playback.py#L361-363 - just returning None or [] causes the infinite loop.

@jcass77
Copy link
Member Author

jcass77 commented Jul 25, 2015

Could be related to mopidy/mopidy#667 which was recently closed - so the actor should not die in the upcoming release of Mopidy 1.1.0 anymore if unhandled exceptions are encountered.

I believe that the expectation is still that Mopidy extensions should handle all of their own exceptions however, and not to rely on core to recover gracefully. At the moment the Mopidy core will just log the exception without any further error handling being done.

@jcass77
Copy link
Member Author

jcass77 commented Aug 2, 2015

Update: this is still an issue in Mopidy 1.0.8 as playback is not covered by the new backend exception handling logic (see mopidy/mopidy#1163 for details). Recent stack trace below:

2015-08-02 07:36:04,599 - WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /access/7768654303429234367.mp4?version=4&lid=1909531839&token=vpiKAZ1IuDx1yxflYc%2BFCN
cCfcDFiXQaf9xyWlUMh9k6amNk6eICaBEvYWjNoiFzFB7i87CwdZ3xPHEdOiIKiPkUnMaMAlWzDT8E1dUKaNIOLO25SJpB11MmJ3qkVH2G3FzmiVW9lKu4HIPq%2FtZHBC83bTbfi%2FSzzfalXBUczq3W2XjZmXaooLMtcQsgZqoQsTHHBSi6Zwa%2FO5rX7MjUxCB%2BF2K7%2F5DTdFPbbMEbDgCD58chcAGTyyJfdSbZZhYdlJLh12hUGTM1ERGMfw6o08h7VOz4KHgyM
Ee4PVNTYHy7yi5toFQP648h2y4mGoIyXTJbNRA%2BlPqg6NHyfBhOq1FF8uPrCGgnbXI5WZ2lCDRsJ6ig7vgZHbNUe0pRIe4v
2015-08-02 07:36:04,623 - ERROR Unhandled exception in Core (urn:uuid:35c70919-f958-4585-bc41-8ec4c8f2cfca):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 201, in _actor_loop
response = self._handle_receive(message)
File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 295, in _handle_receive
return callee(_message['args'], _message['kwargs'])
File "/var/lib/git/mopidy/mopidy/listener.py", line 49, in on_event
getattr(self, event)(
_kwargs)
File "/var/lib/git/mopidy/mopidy/core/actor.py", line 87, in reached_end_of_stream
self.playback._on_end_of_track()
File "/var/lib/git/mopidy/mopidy/core/playback.py", line 223, in _on_end_of_track
self._change_track(next_tl_track)
File "/var/lib/git/mopidy/mopidy/core/playback.py", line 199, in _change_track
self._play(on_error_step=on_error_step)
File "/var/lib/git/mopidy/mopidy/core/playback.py", line 319, in _play
backend.playback.change_track(tl_track.track).get() and
File "/usr/local/lib/python2.7/dist-packages/pykka/threading.py", line 52, in get
compat.reraise(_self._data['exc_info'])
File "/usr/local/lib/python2.7/dist-packages/pykka/compat.py", line 12, in reraise
exec('raise tp, value, tb')
File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 201, in _actor_loop
response = self._handle_receive(message)
File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 295, in _handle_receive
return callee(_message['args'], *_message['kwargs'])
File "/var/lib/git/mopidy-pandora/mopidy_pandora/actor.py", line 63, in change_track
return super(PandoraPlaybackProvider, self).change_track(self.get_next_track())
File "/var/lib/git/mopidy-pandora/mopidy_pandora/actor.py", line 68, in get_next_track
if track.audio_url and track.get_is_playable():
File "/var/lib/git/pydora/pandora/models/pandora.py", line 86, in get_is_playable
return self._api_client.transport.test_url(self.audio_url)
File "/var/lib/git/pydora/pandora/transport.py", line 127, in test_url
return self._http.head(url).status_code == requests.codes.OK
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 497, in head
return self.request('HEAD', url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, *_send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 424, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='audio-sv3-t1-1-v4v6.pandora.com', port=80): Max retries exceeded with url: /access/7768654303429234367.mp4?version=4&lid=1909531839&token=vpiKAZ1IuDx1yxflYc%2BFCNcCfcDFiXQaf9xyWlUMh9k6amNk6eICaBEvYWjNoiFzFB7i87CwdZ3xPHEdOiIKiPkUnMaMAlWzDT8E1dUKaNIOLO25SJpB11MmJ3qkVH2G3FzmiVW9lKu4HIPq%2FtZHBC83bTbfi%2FSzzfalXBUczq3W2XjZmXaooLMtcQsgZqoQsTHHBSi6Zwa%2FO5rX7MjUxCB%2BF2K7%2F5DTdFPbbMEbDgCD58chcAGTyyJfdSbZZhYdlJLh12hUGTM1ERGMfw6o08h7VOz4KHgyMEe4PVNTYHy7yi5toFQP648h2y4mGoIyXTJbNRA%2BlPqg6NHyfBhOq1FF8uPrCGgnbXI5WZ2lCDRsJ6ig7vgZHbNUe0pRIe4v (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",)))
2015-08-02 07:36:04,668 - WARNING Tried to communicate with dead actor.
2015-08-02 07:36:04,714 - WARNING Tried to communicate with dead actor.

@jcass77
Copy link
Member Author

jcass77 commented Aug 14, 2015

The code in the 'Develop' branch is a lot better at checking and preventing exceptions in the mopidy-pandora back-end. I am not aware of any unhandled exceptions that are not covered there at this stage.

@rectalogic, I'm not sure if the exception handling will be ported back to the 'Master' branch or if we will just merge the two when the time is right?

@rectalogic
Copy link
Contributor

@jcass77 if you've been running on develop with no issues, then I'll just merge it into master.

@jcass77
Copy link
Member Author

jcass77 commented Aug 14, 2015

I've been running https://github.com/jcass77/mopidy-pandora/tree/feature/thumbs_support (which is forked off 'Develop') for the last two weeks without issues.

So I think we are safe but would be good if someone else could also give the 'Develop' branch a try first.

@rectalogic
Copy link
Contributor

Closing now that develop is merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants