Proof of concept: Quart

[dalf]

asyncio demo / proof of concept:

• drop python 2 support (see https://github.com/dalf/searx/tree/python3 )
• use Quart ( https://pgjones.gitlab.io/quart/index.html ).
• requires Python 3.6, see https://pgjones.gitlab.io/quart/python_versions.html (aiohttp supports Python 3.5.2, but requires more works).
• use httpx ( https://github.com/encode/httpx ).
• image proxy seems faster (but can be done with Flask, see Improve searx image proxy ? #934).
• only one thread except to parse html document (lxml).

what is not working:

• tests
• queries are bytes, could be string
• some engines (minor adjustments to do)
• simple theme: bug with one jinja2 filter ( https://github.com/asciimoo/searx/blob/master/searx/templates/simple/results.html#L9 ).
• HTTP2: HTTP/2 connections are closed without responding to pings encode/httpx#381

related to #904 (comment)
whatever benchmark results are:

• P2P idea #792 (or a similar idea) requires most probably persistent connections. asyncio would avoid numerous threads.
• (compare to uwsgi) one process would help to have more accurate statistics (instead of getting the stats for one uwsgi process).
• (compare to uwsgi) one process makes embed searx-checker implementation easier (avoid file / whatever synchronization to avoid multiple checks, one per process).
• if the image proxy is fast enough, morty could be used only when the admin wants to provide a html proxy. Not sure if httpx (or aiohttp) could be as fast as valyala/fasthttp

[dalf]

See https://github.com/dalf/searx/tree/quart_aiohttp for a try of quart + aiohttp as a client.
It seems slower that quart + httpx.

Quart

Notes about quart:

• The version 0.6.15 is the last one to support Python 3.6
• https://github.com/pgjones/quart/blob/master/src/quart/flask_patch/_patch.py#L11 :

  it requires the python Task and Future implementations, which invokes some performance cost.

• https://jinja.palletsprojects.com/en/2.9.x/api/#async-support

  all template generation code internally runs async generators which means that you will pay a performance penalty even if the non sync methods are used!

Dirty workaround idea about the last point: monkey patch quart.templating._render to render the template in a ThreadPool using asyncio.loop.run_in_executor

httpx

httpx is based on h11 and h2. Both use the bring-your-own-I/O principle.
README.md says:

This project should be considered as an "alpha" release. It is substantially API complete, but there are still some areas that need more work.

HTTP/1.1 and HTTP/2

I agree, after doing some requests:

• HTTP/2 supports seems cause issue here and there (it is possible to disable it)
• HTTP/1 has also issues too to deal with broken servers (requests, pycurl works not httpx on some requests).
• But to send requests to wikipedia, duckduckgo, etc... it is okay (without HTTP/2).

Proxy and connection reuse

• https://github.com/encode/httpx/blob/1a32cf036a825f6eb35395af5388a3b23180a82e/httpx/dispatch/proxy_http.py#L108

After we receive the 2XX response from the proxy that our tunnel is open we switch the connection's origin to the original so the tunnel can be re-used.

aiohttp

aiohttp uses C to provide an server and a client.
See https://github.com/aio-libs/aiohttp/tree/master/aiohttp
The library declares itself as "5 - Production/Stable".

HTTP/2

The HTTP client doesn't support HTTP/2 : aio-libs/aiohttp#863 (comment)

Anyway it seems as fast as pycurl, see https://github.com/dalf/searx/blob/pycurl-2nd-edition/misc/httpclient_bench.py

Proxy and connection reuse

• https://github.com/aio-libs/aiohttp/blob/48bf8c356c7c3bac5d520484ede5c35bd1795afb/aiohttp/connector.py#L1008

Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response.

Which path ?

I think a full aiohttp searx version should be tried:

• aiohttp server
• aiohttp client
• aiohttp-jinja2 (referenced by the documentation)
• The CONNECT method seems supported: https://github.com/aio-libs/aiohttp/blob/master/tests/test_proxy_functional.py#L14

[nfk]

Hi @dalf, I had tested aiohttp few months ago. I had abandoned the idea because I did want to move apart from upstream.

You can see my work here after removing python2 support

Whatever moving to ASGI is a really good idea. Starting with quart is probably a reasonable choice to move the project toward asyncio.

[dalf]

See #2313