Encoding detection results in performance difference with other clients

[serathius]

Long story short

When using aiohttp for fetching pages I found strange performance problems. We started comparing timings with other clients like curl and requests and difference was significant. For other clients fetching page was 6 times faster compared to aiohttp. After some digging we found that problem was using method "text" that was used encoding detection from chardet.

Requests is also using chardet, but the difference is that it's skipping it if content-type contains word "text" by using 'ISO-8859-1'. https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L362

Expected behaviour

Matching behavior to other popular clients.

Actual behaviour

Huge performance hit for pages without explicit encoding. For example "Content-Type: text/html"
For 300kB pages time difference for using encoding and not is 8s to 2s respectively. (using method "text" instead of "read"), and for 1.3MB page difference is 33s to 4.5s.

Steps to reproduce

I'm sorry I cannot disclose the page that I used for testing.

Your environment

I tested it on two environments
Linux 4.8 Ubuntu 16.10 Python 3.6
aiohttp==1.0.5
chardet==2.3.0

Linux 4.8 Ubuntu 16.10 Python 3.5.2
aiohttp==2.0.6
chardet==3.0.1
^ for that environment problem was smaller by 30%

[asvetlov]

Not sure if Content-Type: text/html should be treated as ISO-8859-1.
Nowdays default encoding is most likely UTF-8.
AFAIK there is no standard for default encoding for HTTP body in general or text/html particularly.

If you need better performance and you know the page encoding you could use (await resp.read()).decode('ISO-8859-1')

[serathius]

I easily fixed the problem by using resp.text('ISO-8859-1').
I just wanted to provide some feedback that how requests is handling that.

[fafhrd91]

@serathius could you check if cchardet is installed and uses c-extension

[asvetlov]

The problem is: if charset was not provided by Content-Type header applying ISO-8859-1 might corrupt non-English text.
As I said now most part of non-English htmls are encoded by UTF-8 (at least should be). ISO-8859-1 decodes it without an error but resulting text is obviously broken.
That's why I think the current behavior should not be changed.

But documentation update with mentioning possible performance issue and resp.text('ISO-8859-1') would be fine.

@serathius would you make a PR for doc update?

[serathius]

@asvetlov Yes, after some more research and experiments with chardet

[fafhrd91]

do we have any actionable for this ticket?

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.