New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use GunicornWebWorker from aiohttp if available #1418

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
7 participants
@fafhrd91
Collaborator

fafhrd91 commented Dec 27, 2016

we are planing to deprecate aiohttp.wsgi module, so i'd like to deprecate AiohttpWorker as well.
GunicornWebWorker supports both aiohttp.web.Application and aiohttp.swgi applications

we are planing to remove aiohttp.wsgi in 9-12 months.

@fafhrd91 fafhrd91 referenced this pull request Dec 27, 2016

Closed

Deprecate aiohttp.wsgi #1108

@tilgovi

As long as everyone else is okay with bumping the requirement to version 3.4 for this worker, I think this is fine.

Show outdated Hide outdated gunicorn/workers/gaiohttp.py
Show outdated Hide outdated gunicorn/workers/gaiohttp.py
Show outdated Hide outdated gunicorn/workers/gaiohttp.py
@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc Feb 6, 2017

Owner

bump?

Owner

benoitc commented Feb 6, 2017

bump?

@fafhrd91

This comment has been minimized.

Show comment
Hide comment
@fafhrd91

fafhrd91 Feb 6, 2017

Collaborator

@benoitc do you want me to deprecate worker or replace it with new one?

Collaborator

fafhrd91 commented Feb 6, 2017

@benoitc do you want me to deprecate worker or replace it with new one?

@fafhrd91

This comment has been minimized.

Show comment
Hide comment
@fafhrd91

fafhrd91 Mar 16, 2017

Collaborator

Updated.

Collaborator

fafhrd91 commented Mar 16, 2017

Updated.

@fafhrd91

This comment has been minimized.

Show comment
Hide comment
@fafhrd91

fafhrd91 Mar 20, 2017

Collaborator

I can replace AiohttpWorker with GunicornWebWorker as well.

Collaborator

fafhrd91 commented Mar 20, 2017

I can replace AiohttpWorker with GunicornWebWorker as well.

@berkerpeksag

We also need to update the documentation.

Another question: Should we keep -k gaiohttp as an alias to -k aiohttp.worker.GunicornWebWorker when we get rid of gunicorn/workers/gaiohttp.py?

Thanks!

except ImportError:
from gunicorn.workers._gaiohttp import AiohttpWorker
utils.warn(

This comment has been minimized.

@berkerpeksag

berkerpeksag Apr 8, 2017

Collaborator
utils.warn(
    "AiohttpWorker is deprecated. Please install aiohttp 1.2+ "
    "and set aiohttp.worker.GunicornWebWorker as a custom worker."
)
@berkerpeksag

berkerpeksag Apr 8, 2017

Collaborator
utils.warn(
    "AiohttpWorker is deprecated. Please install aiohttp 1.2+ "
    "and set aiohttp.worker.GunicornWebWorker as a custom worker."
)
@fafhrd91

This comment has been minimized.

Show comment
Hide comment
@fafhrd91

fafhrd91 Apr 8, 2017

Collaborator

I asked question regarding deprecation month ago, you guys should decide. right now gunicorn does not work with aiohttp 2.x at all.

Collaborator

fafhrd91 commented Apr 8, 2017

I asked question regarding deprecation month ago, you guys should decide. right now gunicorn does not work with aiohttp 2.x at all.

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc Apr 21, 2017

Owner

@fafhrd91 is there any eplacement to support asgi/wsgi?

since you have the commit bit you could likely replace the current implementation by a new one btw ;)

Owner

benoitc commented Apr 21, 2017

@fafhrd91 is there any eplacement to support asgi/wsgi?

since you have the commit bit you could likely replace the current implementation by a new one btw ;)

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 15, 2017

Owner

I would like to understand what it means as it is not clear at all? is the AiohttpWorker inside can still work or not with the latest version of aiohttp? If not why the worker hasn't been updated there?

Owner

benoitc commented May 15, 2017

I would like to understand what it means as it is not clear at all? is the AiohttpWorker inside can still work or not with the latest version of aiohttp? If not why the worker hasn't been updated there?

@fafhrd91

This comment has been minimized.

Show comment
Hide comment
@fafhrd91

fafhrd91 May 15, 2017

Collaborator

AiohttpWorker does not work with aiohttp2.0. solution was siggested 5 months ago.

Collaborator

fafhrd91 commented May 15, 2017

AiohttpWorker does not work with aiohttp2.0. solution was siggested 5 months ago.

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 15, 2017

Owner

well my question was , why the worker hasn't been updated. So people could simply update later.

At least we need to mark the current one as deprecated and document it :)

Owner

benoitc commented May 15, 2017

well my question was , why the worker hasn't been updated. So people could simply update later.

At least we need to mark the current one as deprecated and document it :)

@bart3005

This comment has been minimized.

Show comment
Hide comment
@bart3005

bart3005 Jun 6, 2017

Is there a time frame for aiohttp2.0 support?

bart3005 commented Jun 6, 2017

Is there a time frame for aiohttp2.0 support?

@berkerpeksag berkerpeksag referenced this pull request Jun 30, 2017

Closed

Drop aiohttp support #1536

@tilgovi

This comment has been minimized.

Show comment
Hide comment
@tilgovi

tilgovi Jul 11, 2017

Collaborator

I think we just drop it for R20 and don't worry about deprecating, but we need to update the docs.

Collaborator

tilgovi commented Jul 11, 2017

I think we just drop it for R20 and don't worry about deprecating, but we need to update the docs.

@bart3005

This comment has been minimized.

Show comment
Hide comment
@bart3005

bart3005 Aug 3, 2017

So is there any plan to support aiohttp? In my real world use case with a Flask app, Gunicorn with the gaiohttp worker is 3rd faster than the other fastest worker.

Just wondering what I should do.

bart3005 commented Aug 3, 2017

So is there any plan to support aiohttp? In my real world use case with a Flask app, Gunicorn with the gaiohttp worker is 3rd faster than the other fastest worker.

Just wondering what I should do.

@tilgovi

This comment has been minimized.

Show comment
Hide comment
@tilgovi

tilgovi Aug 7, 2017

Collaborator

@bart3005 you can still set -k aiohttp.worker.GunicornWebWorker if you install aiohttp. My understanding is that this is just about removing the worker from being bundled with Gunicorn by default.

Collaborator

tilgovi commented Aug 7, 2017

@bart3005 you can still set -k aiohttp.worker.GunicornWebWorker if you install aiohttp. My understanding is that this is just about removing the worker from being bundled with Gunicorn by default.

@bart3005

This comment has been minimized.

Show comment
Hide comment
@bart3005

bart3005 Aug 7, 2017

@tilgovi Hmmm OK does not seem to work, with latest of both:

RuntimeError: aiohttp.wsgi is not supported anymore, consider to switch to aiohttp.web.Application

Works OK with aiohttp (1.3.5) but not with latest aiohttp.

bart3005 commented Aug 7, 2017

@tilgovi Hmmm OK does not seem to work, with latest of both:

RuntimeError: aiohttp.wsgi is not supported anymore, consider to switch to aiohttp.web.Application

Works OK with aiohttp (1.3.5) but not with latest aiohttp.

@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Aug 7, 2017

Collaborator

I think what they meant with that exception message is that you should use the web.Application API instead of a plain WSGI app. Here's a minimal working example with using aiohttp 2.2.5 and Gunicorn:

import asyncio

from aiohttp import web


# I have Python 3.4 on my current system so I didn't use the 'async' keyword.
@asyncio.coroutine
def handle(request):
    name = request.match_info.get("name", "Anonymous")
    text = "Hello, " + name
    return web.Response(text=text)

app = web.Application()
app.router.add_get('/', handle)
app.router.add_get('/{name}', handle)

And run it:

$ gunicorn foo:app  -b 127.0.0.1:8585 -k aiohttp.worker.GunicornWebWorker
[2017-08-07 15:28:13 +0300] [18979] [INFO] Starting gunicorn 19.7.1
[2017-08-07 15:28:13 +0300] [18979] [INFO] Listening at: http://127.0.0.1:8585 (18979)
[2017-08-07 15:28:13 +0300] [18979] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2017-08-07 15:28:13 +0300] [18984] [INFO] Booting worker with pid: 18984

And test it:

$ curl http://127.0.0.1:8585
Hello, Anonymous

And I got the same exception when I used examples/test.py (tweaked a bit) with aiohttp 2.2.5:

cd examples/
$ gunicorn test:app  -b 127.0.0.1:8585 -k aiohttp.worker.GunicornWebWorker
...
[2017-08-07 15:22:27 +0300] [18852] [INFO] Worker exiting (pid: 18852)
[2017-08-07 15:22:28 +0300] [18853] [INFO] Booting worker with pid: 18853
[2017-08-07 15:22:28 +0300] [18853] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/berker/projects/gunicorn/gunicorn/arbiter.py", line 579, in spawn_worker
    worker.init_process()
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 41, in init_process
    super().init_process()
  File "/home/berker/projects/gunicorn/gunicorn/workers/base.py", line 133, in init_process
    self.run()
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 49, in run
    self.loop.run_until_complete(self._runner)
  File "/usr/local/lib/python3.4/asyncio/base_events.py", line 341, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.4/asyncio/futures.py", line 276, in result
    raise self._exception
  File "/usr/local/lib/python3.4/asyncio/tasks.py", line 243, in _step
    result = coro.send(None)
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 105, in _run
    handler = self.make_handler(self.wsgi)
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 68, in make_handler
    "aiohttp.wsgi is not supported anymore, "
RuntimeError: aiohttp.wsgi is not supported anymore, consider to switch to aiohttp.web.Application
Collaborator

berkerpeksag commented Aug 7, 2017

I think what they meant with that exception message is that you should use the web.Application API instead of a plain WSGI app. Here's a minimal working example with using aiohttp 2.2.5 and Gunicorn:

import asyncio

from aiohttp import web


# I have Python 3.4 on my current system so I didn't use the 'async' keyword.
@asyncio.coroutine
def handle(request):
    name = request.match_info.get("name", "Anonymous")
    text = "Hello, " + name
    return web.Response(text=text)

app = web.Application()
app.router.add_get('/', handle)
app.router.add_get('/{name}', handle)

And run it:

$ gunicorn foo:app  -b 127.0.0.1:8585 -k aiohttp.worker.GunicornWebWorker
[2017-08-07 15:28:13 +0300] [18979] [INFO] Starting gunicorn 19.7.1
[2017-08-07 15:28:13 +0300] [18979] [INFO] Listening at: http://127.0.0.1:8585 (18979)
[2017-08-07 15:28:13 +0300] [18979] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2017-08-07 15:28:13 +0300] [18984] [INFO] Booting worker with pid: 18984

And test it:

$ curl http://127.0.0.1:8585
Hello, Anonymous

And I got the same exception when I used examples/test.py (tweaked a bit) with aiohttp 2.2.5:

cd examples/
$ gunicorn test:app  -b 127.0.0.1:8585 -k aiohttp.worker.GunicornWebWorker
...
[2017-08-07 15:22:27 +0300] [18852] [INFO] Worker exiting (pid: 18852)
[2017-08-07 15:22:28 +0300] [18853] [INFO] Booting worker with pid: 18853
[2017-08-07 15:22:28 +0300] [18853] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/berker/projects/gunicorn/gunicorn/arbiter.py", line 579, in spawn_worker
    worker.init_process()
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 41, in init_process
    super().init_process()
  File "/home/berker/projects/gunicorn/gunicorn/workers/base.py", line 133, in init_process
    self.run()
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 49, in run
    self.loop.run_until_complete(self._runner)
  File "/usr/local/lib/python3.4/asyncio/base_events.py", line 341, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.4/asyncio/futures.py", line 276, in result
    raise self._exception
  File "/usr/local/lib/python3.4/asyncio/tasks.py", line 243, in _step
    result = coro.send(None)
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 105, in _run
    handler = self.make_handler(self.wsgi)
  File "/home/berker/projects/gunicorn/venv/lib/python3.4/site-packages/aiohttp/worker.py", line 68, in make_handler
    "aiohttp.wsgi is not supported anymore, "
RuntimeError: aiohttp.wsgi is not supported anymore, consider to switch to aiohttp.web.Application
@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Aug 7, 2017

Collaborator

I think what they meant with that exception message is that you should use the web.Application API instead of a plain WSGI app.

To be clear, a Flask app is also a plain WSGI app so you'll either stay with aiohttp 1 or switch your application to use aiohttp's web.Application API.

Collaborator

berkerpeksag commented Aug 7, 2017

I think what they meant with that exception message is that you should use the web.Application API instead of a plain WSGI app.

To be clear, a Flask app is also a plain WSGI app so you'll either stay with aiohttp 1 or switch your application to use aiohttp's web.Application API.

@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Aug 7, 2017

Collaborator

Turns out, there is a third-party project called aiohttp_wsgi which lets you wrap your WSGI application with aiohttp_wsgi.WSGIHandler and pass it into aiohttp.web.Application: https://aiohttp-wsgi.readthedocs.io/en/stable/wsgi.html#run-a-simple-web-server

I'm going to open a new PR to document the current situation.

Collaborator

berkerpeksag commented Aug 7, 2017

Turns out, there is a third-party project called aiohttp_wsgi which lets you wrap your WSGI application with aiohttp_wsgi.WSGIHandler and pass it into aiohttp.web.Application: https://aiohttp-wsgi.readthedocs.io/en/stable/wsgi.html#run-a-simple-web-server

I'm going to open a new PR to document the current situation.

@bart3005

This comment has been minimized.

Show comment
Hide comment
@bart3005

bart3005 Aug 8, 2017

@berkerpeksag thanks so much for this. Works like a charm and performance is very good.

in the __init__.py of the flask app:

from aiohttp import web
from aiohttp_wsgi import WSGIHandler

app = Flask(__name__)
wsgi_handler = WSGIHandler(app)
aioapp = web.Application()
aioapp.router.add_route("*", "/{path_info:.*}", wsgi_handler)

And gunicorn:

gunicorn app:aioapp -b 127.0.0.1:9000 \
  --worker-class aiohttp.worker.GunicornWebWorker

bart3005 commented Aug 8, 2017

@berkerpeksag thanks so much for this. Works like a charm and performance is very good.

in the __init__.py of the flask app:

from aiohttp import web
from aiohttp_wsgi import WSGIHandler

app = Flask(__name__)
wsgi_handler = WSGIHandler(app)
aioapp = web.Application()
aioapp.router.add_route("*", "/{path_info:.*}", wsgi_handler)

And gunicorn:

gunicorn app:aioapp -b 127.0.0.1:9000 \
  --worker-class aiohttp.worker.GunicornWebWorker
@asvetlov

This comment has been minimized.

Show comment
Hide comment
@asvetlov

asvetlov Aug 8, 2017

Collaborator

@bart3005 aiohttp doesn't support aiohttp_wsgi but I hope next aiohttp releases will not break the library (or author will publish bugfix version)

Collaborator

asvetlov commented Aug 8, 2017

@bart3005 aiohttp doesn't support aiohttp_wsgi but I hope next aiohttp releases will not break the library (or author will publish bugfix version)

@bart3005

This comment has been minimized.

Show comment
Hide comment
@bart3005

bart3005 commented Aug 8, 2017

@asvetlov understood.

@Fover23

This comment has been minimized.

Show comment
Hide comment
@Fover23

Fover23 Aug 9, 2017

@bart3005
You can try aiohttp 1.3.5 with uvloop
is very fast and work well with my django project

pip install aiohttp==1.3.5
pip install uvloop

and gunicorn:

gunicorn -k aiohttp.worker.GunicornUVLoopWebWorker -w 1 yourproject.wsgi:application -b 127.0.0.1:8000

Fover23 commented Aug 9, 2017

@bart3005
You can try aiohttp 1.3.5 with uvloop
is very fast and work well with my django project

pip install aiohttp==1.3.5
pip install uvloop

and gunicorn:

gunicorn -k aiohttp.worker.GunicornUVLoopWebWorker -w 1 yourproject.wsgi:application -b 127.0.0.1:8000
@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Aug 9, 2017

Collaborator

Superseded by #1569. I've added an example app, document the current situation, and deprecated the gaiohttp worker there.

Collaborator

berkerpeksag commented Aug 9, 2017

Superseded by #1569. I've added an example app, document the current situation, and deprecated the gaiohttp worker there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment