Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ASGI Support #532

Merged
merged 101 commits into from Dec 4, 2019
Merged
Changes from 86 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
5eaf1a1
Trash
dmanchon Jun 13, 2019
853f476
requirements
dmanchon Jun 13, 2019
d023abb
Tests + changes to factory
masipcat Jun 14, 2019
3bd33e9
hyper
dmanchon Jun 14, 2019
ea8c6dd
Clean a bit
dmanchon Jun 14, 2019
4acf812
make hypercorn work
masipcat Jun 14, 2019
f59e459
Using custom Request instead of aiohttp.Request
masipcat Jun 14, 2019
b11dd0d
POST fixed. Molotov test
dmanchon Jun 14, 2019
032f68c
Fix testclient fixture
masipcat Jun 14, 2019
ab8a126
Small changes
masipcat Jun 15, 2019
898142b
websockets with asgi draft
dmanchon Jun 15, 2019
f010e3a
websocket close handling
dmanchon Jun 15, 2019
e2c38c6
Move requests implementations to a module
dmanchon Jun 15, 2019
1827366
fix test
vangheem Jun 15, 2019
55a66d8
Fixed GuillotinaRequest.query_string
masipcat Jun 15, 2019
62fff01
Checkpoint. Almost all tests are green
masipcat Jun 15, 2019
093e94f
Fix ws tests
dmanchon Jun 15, 2019
d8d6a13
Implement IRequest record method on asgi
dmanchon Jun 15, 2019
27e4b2a
Deleted aiohttp request
masipcat Jun 15, 2019
db9b798
Fix debug headers handler
dmanchon Jun 16, 2019
5ae7797
Merge branch 'master' into asgi-support
masipcat Jun 16, 2019
89fafb3
Merge branch 'asgi-support' of github.com:plone/guillotina into asgi-…
masipcat Jun 16, 2019
05d10ef
Fix deleted ws import
masipcat Jun 16, 2019
309dce8
Fixed AsgiStreamReader bug
masipcat Jun 16, 2019
2fc67b2
More fixes
masipcat Jun 16, 2019
9c944af
Small fix
dmanchon Jun 16, 2019
df15f81
Simulate incoming request in chunks for uploads
dmanchon Jun 16, 2019
826dd32
Remove starlette dep
dmanchon Jun 16, 2019
29dd329
flake8
dmanchon Jun 16, 2019
d72d373
Almost there
masipcat Jun 16, 2019
1c4d5c7
All tests green
masipcat Jun 16, 2019
69adf04
Now should be all green
masipcat Jun 16, 2019
7952fb0
Trying to fix failing tests in travis
masipcat Jun 16, 2019
7c10a56
I hope now is fixed...
masipcat Jun 16, 2019
76963a7
Small fix
masipcat Jun 16, 2019
5269254
Revert "Trying to fix failing tests in travis"
masipcat Jun 16, 2019
49ab931
Fix tests when runnign with DB_SCHEMA != public
masipcat Jun 16, 2019
c62bd3a
update to last asgi test client
dmanchon Jun 17, 2019
2a0146e
Updated async-asgi-testclient
masipcat Jun 17, 2019
223f083
Merge branch 'master' into asgi-support
masipcat Jun 17, 2019
9eec6ec
Small changes
masipcat Jun 17, 2019
eee7568
Fix merge
masipcat Jun 17, 2019
70a2a0f
Merge branch 'master' into asgi-support
masipcat Jun 18, 2019
ed3e7eb
Configured pg v10 in pytest-docker-fixtures + small change in fixtures
masipcat Jun 18, 2019
99e012b
Merge branch 'master' into asgi-support
masipcat Jun 18, 2019
f19412a
Merge branch 'master' into asgi-support
masipcat Jun 19, 2019
1c6dca7
Clean up unused methods
masipcat Jun 19, 2019
841333d
Merge branch 'master' into asgi-support
masipcat Jun 19, 2019
8ecb6b8
Rearrenged code and reduced code that depends on aiohttp
masipcat Jun 19, 2019
3c7eb26
Merge branch 'master' into asgi-support
masipcat Jun 19, 2019
efe666b
Updated changelog
masipcat Jun 19, 2019
4f0714e
Fix pg catalog tests
masipcat Jun 21, 2019
c8fe97c
Remove aiohttp dependecy for websockets
dmanchon Jun 22, 2019
dcba06c
Remove aiohttp dependecy for websockets
dmanchon Jun 22, 2019
7aafe6f
Remove aiohttp dependecy for request
dmanchon Jun 22, 2019
30b58fa
Flake8
dmanchon Jun 22, 2019
8fe4e10
Replaced 'loop' fixture from 'aiohttp' for 'event_loop' from 'pytest-…
masipcat Jun 22, 2019
ca8be11
Merge branch 'master' into asgi-support
masipcat Jun 22, 2019
2086715
Fix cockroach fixture
masipcat Jun 22, 2019
f6b49ba
Reduced aiohttp dependence. TODO: traversal/router and CORS
masipcat Aug 29, 2019
147c233
BOOM! Merge branch 'master' into asgi-support
masipcat Aug 29, 2019
fbb0b76
Lot of fixes
masipcat Aug 29, 2019
b375401
Fixed flake8 and mypy
masipcat Aug 29, 2019
133a30e
black
masipcat Aug 29, 2019
f7f6669
Added some tests and cleaned unused code
masipcat Aug 30, 2019
22c6a8a
Mypy
masipcat Aug 30, 2019
00eaacd
Asgi support: no aiohttp (#654)
vangheem Aug 31, 2019
699176e
Merge branch 'master' into asgi-support
masipcat Aug 31, 2019
168c560
Merge branch 'master' into asgi-support
masipcat Sep 3, 2019
80f25f1
isort
masipcat Sep 3, 2019
f73bef1
Merge branch 'master' into asgi-support
masipcat Sep 3, 2019
98f4c11
Merge branch 'master' into asgi-support
masipcat Sep 20, 2019
fbc08df
Merge branch 'master' into asgi-support
masipcat Sep 20, 2019
f1211c8
Documented how to use differents ASGI servers + small changes
masipcat Sep 22, 2019
272f331
Small fixes
masipcat Sep 22, 2019
307a31d
Merge branch 'master' into asgi-support
masipcat Sep 24, 2019
095125f
mypy-flake8
masipcat Sep 24, 2019
6726ce1
Merge branch 'asgi-support' of github.com:plone/guillotina into asgi-…
masipcat Sep 24, 2019
3203b0d
Changes and fixes
masipcat Sep 24, 2019
e5d6a90
Removed yarl
masipcat Sep 24, 2019
0265be1
Support for middlewares
masipcat Oct 2, 2019
e0955ef
Merge branch 'master' into asgi-support
masipcat Oct 2, 2019
0333e34
Black
masipcat Oct 2, 2019
61d7d20
Merge branch 'master' into asgi-support
masipcat Oct 18, 2019
49230f1
Updated Cython for python3.8 (required by uvloop)
masipcat Oct 18, 2019
515327a
fix uvloop
masipcat Oct 18, 2019
ed069ad
requested changes
masipcat Oct 18, 2019
12deec6
Merge branch 'master' into asgi-support
masipcat Oct 23, 2019
dc73ece
Removed python 3.8 in travis
masipcat Oct 23, 2019
f925322
Merge branch 'master' into asgi-support
masipcat Oct 25, 2019
b57cd01
Changed implementation of reify
masipcat Oct 25, 2019
88751c5
Merge branch 'master' into asgi-support
masipcat Nov 7, 2019
7ae3546
Merge branch 'master' into asgi-support
masipcat Nov 21, 2019
c94cd49
Fix some tests are skiped and mypy errors
masipcat Nov 21, 2019
6295819
Install extra 'testdata' in travis
masipcat Nov 21, 2019
a82df48
Merge branch 'master' into asgi-support
masipcat Nov 24, 2019
be427dd
Fixed tests
masipcat Nov 24, 2019
5547069
Merge branch 'master' into asgi-support
masipcat Nov 26, 2019
a7bedb5
Merge branch 'master' into asgi-support
masipcat Dec 4, 2019
827e228
Small fixes
masipcat Dec 4, 2019
52f4200
fix
masipcat Dec 4, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -3,7 +3,7 @@ codecov:

coverage:
precision: 1
round: up
round: up
range: "50...95"

status:
@@ -1,6 +1,13 @@
CHANGELOG
=========

6.0.0 (unreleased)
------------------

- Replaced aiohttp with ASGI (running with uvicorn by default)
[dmanchon,masipcat,vangheem]


5.0.25 (unreleased)
-------------------

@@ -1 +1 @@
5.0.25.dev0
6.0.0.dev0
@@ -0,0 +1,24 @@
# Advanced

## Running Guillotina on another ASGI server

Guillotina supoprt the following ASGI servers out-of-the-box:

- `uvicorn` (used by default)
- `hypercorn`

Use the argument `--asgi-server` to choose one of the previous servers:

```shell
guillotina serve -c config.yaml --asgi-server=hypercorn
```

You can use any other ASGI server by using `guillotina.entrypoint:app` as the app and the environment variable `G_CONFIG_FILE` to specificy the configuration file.

**Example:**

Running guillotina on `hypercorn` with QUIC support:

```shell
G_CONFIG_FILE=config.yaml hypercorn --quic-bind 127.0.0.1:4433 guillotina.entrypoint:app
```
@@ -29,7 +29,7 @@ You can provide your own CLI commands for guillotina through a simple interface.
- serve:
- `--host`: host to bind to
- `--port`: port to bind to
- `--reload`: auto reload on code changes. `requires aiohttp_autoreload`
- `--asgi-server`: choose which ASGI server to run (uvicorn by default)
- shell
- create
- `--template`: name of template to use
@@ -36,7 +36,7 @@ of functionality so it will never be as fast as say Pyramid.

## Asynchronous

`guillotina` is asynchronous from the ground up, built on top of `aiohttp`
`guillotina` is asynchronous from the ground up, built with `asgi`
using Python 3.7's asyncio features.

Practically speaking, being built completely on asyncio compatible technologies,
@@ -9,9 +9,9 @@ responses given depending on the exception type.
## Custom exception response

```python
from aiohttp.web_exceptions import HTTPPreconditionFailed
from guillotina import configure
from guillotina.interfaces import IErrorResponseException
from guillotina.response import HTTPPreconditionFailed
import json
@@ -29,3 +29,4 @@ Contents:
async_utils
component-architecture
debugging
advanced
@@ -47,9 +47,9 @@ library.
These response objects should have simple dict values for their content if provided.


### Bypassing reponses rendering
### Bypassing responses rendering

If you return any aiohttp based response objects, they will be ignored by the rendering
If you return a `guillotina.response.ASGISimpleResponse` response object, they will be ignored by the rendering
framework.

This is useful when streaming data for example and it should not be transformed.
@@ -1,6 +1,6 @@
# Router

Guillotina uses `aiohttp` for it's webserver. In order to route requests against
Guillotina uses `asgi` for it's webserver. In order to route requests against
Guillotina's traversal url structure, Guillotina provides it's own router
that does traversal: `guillotina.traversal.router`.

@@ -65,6 +65,7 @@ from guillotina import content
from guillotina import schema
from guillotina.factory import make_app
from zope import interface
import uvicorn
class IMyType(interface.Interface):
foobar = schema.TextLine()
@@ -96,7 +97,7 @@ if __name__ == '__main__':
},
"port": 8080
})
web.run_app(app, host='localhost', port=8080)
uvicorn.run(app, host='localhost', port=8080)
```

@@ -161,24 +161,14 @@ load_utilities:

## Middleware

`guillotina` is built on `aiohttp` which provides support for middleware.
`guillotina` is built on `asgi` which provides support for middleware.
You can provide an array of dotted names to use for your application.

```yaml
middlewares:
- guillotina_myaddon.Middleware
```

## aiohttp settings

You can pass `aiohttp_settings` to configure the aiohttp server.


```yaml
aiohttp_settings:
client_max_size: 20971520
```

## JWT Settings

If you want to enable JWT authentication, you'll need to configure the JWT
@@ -201,15 +191,14 @@ jwk:
## Miscellaneous settings

- `port` (number): Port to bind to. _defaults to `8080`_
- `access_log_format` (string): Customize access log format for aiohttp. _defaults to `None`_
- `store_json` (boolean): Serialize object into json field in database. _defaults to `false`_
- `host` (string): Where to host the server. _defaults to `"0.0.0.0"`_
- `port` (number): Port to bind to. _defaults to `8080`_
- `conflict_retry_attempts` (number): Number of times to retry database conflict errors. _defaults to `3`_
- `cloud_storage` (string): Dotted path to cloud storage field type. _defaults to `"guillotina.interfaces.IDBFileField"`_
- `loop_policy`: (string): Be able to customize the event loop policy used. For example, to use
uvloop, set this value to `uvloop.EventLoopPolicy`.
- `router`: be able to customize the main aiohttp Router class
- `router`: be able to customize the main Router class
- `oid_generator`: be able to customize the function used to generate oids on the system.
defaults to `guillotina.db.oid.generate_oid`
- `cors_renderer`: customize the cors renderer, defaults to `guillotina.cors.DefaultCorsRenderer`
@@ -65,7 +65,7 @@ To log errors for guillotina for example:
}
},
"loggers": {
"aiohttp.access": {
"guillotina": {
"level": "INFO",
"handlers": ["file"],
"propagate": 0
@@ -79,9 +79,3 @@ To log errors for guillotina for example:
## Available Loggers

- `guillotina`
- `aiohttp.access`
- `aiohttp.client`
- `aiohttp.internal`
- `aiohttp.server`
- `aiohttp.web`
- `aiohttp.websocket`
@@ -2,20 +2,18 @@

Websocket support is built-in to Guillotina.

It's as simple as using an `aiohttp` websocket in a service.
It's as simple as using an `asgi` websocket in a service.

Create a `ws.py` file and put the following code in:


```python
from aiohttp import web
from guillotina import configure
from guillotina.component import get_utility
from guillotina.interfaces import IContainer
from guillotina.transactions import get_tm
from guillotina_chat.utility import IMessageSender
import aiohttp
import logging
logger = logging.getLogger('guillotina_chat')
@@ -25,7 +23,9 @@ logger = logging.getLogger('guillotina_chat')
context=IContainer, method='GET',
permission='guillotina.AccessContent', name='@conversate')
async def ws_conversate(context, request):
ws = web.WebSocketResponse()
ws = request.get_ws()
await ws.prepare(request)
This conversation was marked as resolved by bloodbare

This comment has been minimized.

Copy link
@bloodbare

bloodbare Oct 18, 2019

Member

two ws.prepare?

This comment has been minimized.

Copy link
@masipcat

masipcat Oct 18, 2019

Author Contributor

Hmm... I'll take a look

This comment has been minimized.

Copy link
@masipcat

masipcat Oct 18, 2019

Author Contributor

done

utility = get_utility(IMessageSender)
utility.register_ws(ws, request)
@@ -34,12 +34,8 @@ async def ws_conversate(context, request):
await ws.prepare(request)
async for msg in ws:
if msg.tp == aiohttp.WSMsgType.text:
# ws does not receive any messages, just sends
pass
elif msg.tp == aiohttp.WSMsgType.error:
logger.debug('ws connection closed with exception {0:s}'
.format(ws.exception()))
# handle msg
pass
logger.debug('websocket connection closed')
utility.unregister_ws(ws)
@@ -58,7 +58,6 @@ Also, do a `GET` on `http://localhost:8080/db`.

## Useful run options

- `--reload`: auto reload on code changes. `requires aiohttp_autoreload`
- `--profile`: profile Guillotina while it's running
- `--profile-output`: where to save profiling output
- `--monitor`: run with aiomonitor. `requires aiomonitor`
@@ -8,7 +8,6 @@

app_settings: Dict[str, Any] = {
"debug": False,
"aiohttp_settings": {},
"databases": [],
"storages": {},
"cache": {"strategy": "dummy"},
@@ -1,4 +1,3 @@
from aiohttp.web import StreamResponse
from guillotina import configure
from guillotina._settings import app_settings
from guillotina.api.content import DefaultOPTIONS
@@ -10,6 +9,7 @@
from guillotina.interfaces import IResource
from guillotina.interfaces import IStaticDirectory
from guillotina.interfaces import IStaticFile
from guillotina.response import ASGIResponse
from guillotina.response import HTTPNotFound

import mimetypes
@@ -54,21 +54,19 @@ class FileGET(DownloadService):
filepath = str(fi.file_path.absolute())
filename = fi.file_path.name
with open(filepath, "rb") as f:
resp = StreamResponse()
resp = ASGIResponse(status=200)
resp.content_type, _ = mimetypes.guess_type(filename)

disposition = 'filename="{}"'.format(filename)
if "text" not in resp.content_type:
if "text" not in (resp.content_type or ""):
disposition = "attachment; " + disposition

resp.headers["CONTENT-DISPOSITION"] = disposition

data = f.read()
resp.content_length = len(data)
await resp.prepare(self.request)

await resp.write(data)
await resp.write_eof()
await resp.write(data, eof=True)
return resp

async def __call__(self):
@@ -98,7 +98,7 @@
},
)
async def search_get(context, request):
q = request.url.query.copy()
q = request.query.copy()
return await _search(context, request, q)


@@ -53,7 +53,7 @@ class Service(View):

def _validate_parameters(self):
if "parameters" in self.__config__:
data = self.request.url.query
data = self.request.query
for parameter in self.__config__["parameters"]:
if parameter["in"] == "query":
if "schema" in parameter and "name" in parameter:
@@ -1,4 +1,3 @@
from aiohttp import web
from guillotina import configure
from guillotina import logger
from guillotina import routes
@@ -8,10 +7,11 @@
from guillotina.auth.extractors import BasicAuthPolicy
from guillotina.component import get_utility
from guillotina.component import query_multi_adapter
from guillotina.interfaces import IAioHTTPResponse
from guillotina.interfaces import IApplication
from guillotina.interfaces import IASGIResponse
from guillotina.interfaces import IContainer
from guillotina.interfaces import IPermission
from guillotina.request import WebSocketJsonDecodeError
from guillotina.security.utils import get_view_permission
from guillotina.transactions import get_tm
from guillotina.utils import get_jwk_key
@@ -20,7 +20,6 @@
from jwcrypto.common import json_encode
from urllib import parse

import aiohttp
import time
import ujson

@@ -148,8 +147,8 @@ class WebsocketsView(Service):
view = (await view.prepare()) or view

view_result = await view()
if IAioHTTPResponse.providedBy(view_result):
raise Exception("Do not accept raw aiohttp exceptions in ws")
if IASGIResponse.providedBy(view_result):
raise Exception("Do not accept raw ASGI exceptions in ws")
else:
from guillotina.traversal import apply_rendering

@@ -165,31 +164,29 @@ class WebsocketsView(Service):
async def __call__(self):
tm = get_tm()
await tm.abort()
ws = web.WebSocketResponse()
ws = self.request.get_ws()
await ws.prepare(self.request)

async for msg in ws:
if msg.type == aiohttp.WSMsgType.text:
try:
message = msg.json
except WebSocketJsonDecodeError:
# We only care about json messages
logger.warning("Invalid websocket payload, ignored: {}".format(msg))
continue

if message["op"].lower() == "close":
await ws.close()
elif message["op"].lower() == "get":
txn = await tm.begin()
try:
message = ujson.loads(msg.data)
except ValueError:
logger.warning("Invalid websocket payload, ignored: {}".format(msg.data))
continue
if message["op"] == "close":
await ws.close()
elif message["op"].lower() == "get":
txn = await tm.begin()
try:
await self.handle_ws_request(ws, message)
except Exception:
logger.error("Exception on ws", exc_info=True)
finally:
# only currently support GET requests which are *never*
# supposed to be commits
await tm.abort(txn=txn)
elif msg.type == aiohttp.WSMsgType.error:
logger.debug("ws connection closed with exception {0:s}".format(ws.exception()))
await self.handle_ws_request(ws, message)
except Exception:
logger.error("Exception on ws", exc_info=True)
finally:
# only currently support GET requests which are *never*
# supposed to be commits
await tm.abort(txn=txn)

logger.debug("websocket connection closed")

return {}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.