[ARD] Non-ASCII characters (umlauts) in URLs

[benFnord]

Hi!

Apparently, ardmediathek.de retains umlauts contained in video titles in their URLs, causing youtube-dl to fail.

For example, downloading with
$ youtube-dl --verbose "http://www.ardmediathek.de/tv/Sturm-der-Liebe/Folge-2036-Zu-Mann-und-Frau-erklärt/Das-Erste/Video?documentId=22673108&bcastId=5290"

fails, whereas after replacing the ä with %C3%A4

$ youtube-dl --verbose "http://www.ardmediathek.de/tv/Sturm-der-Liebe/Folge-2036-Zu-Mann-und-Frau-erkl%C3%A4rt/Das-Erste/Video?documentId=22673108&bcastId=5290",
downloading succeeds.

Full output:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://www.ardmediathek.de/tv/Sturm-der-Liebe/Folge-2036-Zu-Mann-und-Frau-erkl\xc3\xa4rt/Das-Erste/Video?documentId=22673108&bcastId=5290']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.07.30
[debug] Python version 2.7.8 - Darwin-13.3.0-x86_64-i386-64bit
[debug] Proxy map: {}
[ARD] 22673108: Downloading webpage
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 18, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 889, in main

  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 879, in _real_main

  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1052, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 516, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 170, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/ard.py", line 47, in _real_extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 283, in _download_webpage
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 223, in _download_webpage_handle
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 203, in _request_webpage
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1231, in urlopen
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 995, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1029, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 844, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 820, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 56: ordinal not in range(128)

Thanks everyone for making youtube-dl awesome, keep up the good work! :-)

[dstftw]

Thanks for the report, fixed. This will be available in the next release of youtube-dl.

[benFnord]

Wow, that was fast. Works like a charm, thank you very much!