UnicodeEncodeError encountered on a few cases

[clippered]

Im not sure how to describe but there were a few html body that has encoding errors. It throws UnicodeEncodeError on https://github.com/scrapy-plugins/scrapy-playwright/blob/master/scrapy_playwright/handler.py#L288
So it looks like these websites do not really follow their encodings.

for now, i've patched it with:

--- handler.py.orig	2022-03-22 17:28:32.000000000 +1100
+++ handler.py.mod	2022-03-22 17:16:08.000000000 +1100
@@ -279,7 +279,11 @@
         headers = Headers(response.headers)
         headers.pop("Content-Encoding", None)
         encoding = _get_response_encoding(headers, body_str) or "utf-8"
-        body = body_str.encode(encoding)
+        try:
+            body = body_str.encode(encoding)
+        except UnicodeEncodeError:
+            encoding = html_body_declared_encoding(body_str) or "utf-8"
+            body = body_str.encode(encoding)
         respcls = responsetypes.from_args(headers=headers, url=page.url, body=body)
         return respcls(
             url=page.url,

but maybe you have a better way of handling this case.

[elacuesta]

Sounds reasonable, can you share an example?

[clippered]

Thanks for the quick response. In one case I got a body_str with a \u2120 char in it and the encoding is cp1252. So doing this '\u2120'.encode('cp1252') is giving a lib/python3.9/encodings/cp1252.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_table) UnicodeEncodeError: 'charmap' codec can't encode character '\u2120' in position 0: character maps to <undefined>

I guess we have no control for incorrect contents and content types/encoding, so thats why this patch.

[clippered]

I just realised, I think it might be better to just do a utf-8 instead.

+        try:
+            body = body_str.encode(encoding)
+        except UnicodeEncodeError:
+            encoding = "utf-8"
+            body = body_str.encode(encoding)

[elacuesta]

Thanks, but I meant an example of a URL that shows this mismatch. It's not essential because it can be tested "artificially" but it'd be good to make sure it works on real sites. I will push a patch soon.

[clippered]

maybe this one https://loans.barclaysus.com/clapply/cyr/reward-disclosure.html