Link preview explodes on some URLs that have a "utf-8" encoding

[hawkowl]

2018-11-05 15:59:17,452 - synapse.http.server - 112 - ERROR - GET-1929865 - Failed handle request via <function _async_render_GET at 0x7feaeeffae60>: <XForwardedForRequest at 0x7fea99f53a28 method=u'GET' uri=u'/_matrix/media/r0/preview_url?url=<redacted>: Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "synapse/http/server.py", line 81, in wrapped_request_handler
    
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/rest/media/v1/preview_url_resource.py", line 160, in _async_render_GET
    
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "synapse/rest/media/v1/preview_url_resource.py", line 226, in _do_preview
    
  File "synapse/rest/media/v1/preview_url_resource.py", line 473, in decode_and_calc_og
    
  File "src/lxml/parser.pxi", line 1689, in lxml.etree.HTMLParser.__init__
    
  File "src/lxml/parser.pxi", line 823, in lxml.etree._BaseParser.__init__
    
exceptions.LookupError: unknown encoding: '"utf-8"'

[hawkowl]

Related encoding problems: #2891

[hawkowl]

Fixed by #4157