/
todo
199 lines (115 loc) · 6.71 KB
/
todo
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
Handle static files - static root route and path.
Connection buffer size - after a file upload shrink the buffer back down. Behind cloudflare connections will remain open forever. Behind nginx its just one connection so who cares. Out front client connections will close so it doesn't matter?
support HEAD
disconnect
Protocol dealloc is never called - python bug?
set user sess - fix it!
Change to the new session ids everywhere. TODO user id in session necessary? No. Idea was to use MrWorkServers to cache user data .
Update dotests to setup the memcached/mrq/mrcache servers and set a session id so we find the user
def _get_address(self):
sock = self.transport.get_extra_info('socket')
if sock.family == socket.AF_INET:
self._socket = (self.transport.get_extra_info('peername') or
(None, None))
self._ip, self._port = self._socket
elif sock.family == socket.AF_INET6:
self._socket = (self.transport.get_extra_info('peername') or
(None, None, None, None))
self._ip, self._port, *_ = self._socket
else:
self._ip, self._port = (None, None)
@property
def remote_addr(self):
"""Attempt to return the original client ip based on X-Forwarded-For.
:return: original client ip.
"""
if not hasattr(self, '_remote_addr'):
forwarded_for = self.headers.get('X-Forwarded-For', '').split(',')
remote_addrs = [
addr for addr in [
addr.strip() for addr in forwarded_for
] if addr
]
if len(remote_addrs) > 0:
self._remote_addr = remote_addrs[0]
else:
self._remote_addr = ''
return self._remote_addr
subdomains
redirects?
bad.py - bad request
Review tests
mrpacker
memcached = clean it up
If MemcachedClient_set servers are down we return NULL and don't setup an exception
dotests disconnect
Request_reset - we doing in all necessary places
Make conn/req timeouts as arguments (20 / 5 right now)
Add tests for memc and mrq failover. ?
Add streaming upload
https://github.com/huge-success/sanic/tree/master/examples/request_stream
test for setUserSession with added cookies
/r/python - Why would you write your web server in anything but python
If we lose the mrq servers we have to tell client
!!
File "uvloop/handles/basetransport.pyx", line 92, in uvloop.loop.UVBaseTransport._maybe_pause_protocol
AttributeError: 'internals.MrqProtocol' object has no attribute 'pause_writing'
protocol.resume_writing() failed
protocol: <internals.MrqProtocol object at 0x7f1530f86cf0>
transport: <TCPTransport closed=False reading=True 0x2b62fc8>
Traceback (most recent call last):
File "uvloop/handles/basetransport.pyx", line 108, in uvloop.loop.UVBaseTransport._maybe_resume_protocol
AttributeError: 'internals.MrqProtocol' object has no attribute 'resume_writing'
Explained?
For the asyncio streams-based implementation, StreamReader automatically uses the {pause,resume}_reading methods to transmit backpressure upstream, and StreamWriter provides a friendly wrapper around {pause,resume}_writing to help us accept backpressure from downstream: the drain method – we just have to remember to use it. So in order to fix our proxy to transmit backpressure, all we need to do is to add one line of code to copy_all. Specifically, this line:
await drain()
await dest_writer.drain()
Flow Control
In the initial echo server example we had await writer.drain() as this paused the coroutine from writing more data to the socket till the client had caught up, it drained the socket. This is useful as until the client catches up the data will be stored in memory, hence a malicious client can make many requests for a lot of data, refuse to receive the data, and allow the server to exhaust its memory.
To combat this the coroutine sending data should await a drain function, that can be added to the protocol,
import asyncio
class FlowControlServer(asyncio.Protocol):
def __init__(self):
self._can_write = asyncio.Event()
self._can_write.set()
def pause_writing(self) -> None:
# Will be called whenever the transport crosses the
# high-water mark.
self._can_write.clear()
def resume_writing(self) -> None:
# Will be called whenever the transport drops back below the
# low-water mark.
self._can_write.set()
async def drain(self) -> None:
await self._can_write.wait()
WriteTransport.set_write_buffer_limits(high=None, low=None)
Set the high and low watermarks for write flow control.
These two values (measured in number of bytes) control when the protocol’s protocol.pause_writing() and protocol.resume_writing() methods are called. If specified, the low watermark must be less than or equal to the high watermark. Neither high nor low can be negative.
pause_writing() is called when the buffer size becomes greater than or equal to the high value. If writing has been paused, resume_writing() is called when the buffer size becomes less than or equal to the low value.
The defaults are implementation-specific. If only the high watermark is given, the low watermark defaults to an implementation-specific value less than or equal to the high watermark. Setting high to zero forces low to zero as well, and causes pause_writing() to be called whenever the buffer becomes non-empty. Setting low to zero causes resume_writing() to be called only once the buffer is empty. Use of zero for either limit is generally sub-optimal as it reduces opportunities for doing I/O and computation concurrently.
Wiki doc?
http://www.django-rest-framework.org/api-guide/requests/
vhosts -- how to do this? I don't like sanic's, maybe do cherrypy's redir
sanic:
@app.route('/', host=["example.com",
"somethingelse.com",
"therestofyourdomains.com"])
cherrypy:
conf = {
"/": {
"request.dispatch": cherrypy.dispatch.VirtualHost(
**{
'api.foo.com': '/api',
logging
Stream large file upload
https://github.com/channelcat/sanic/issues/546
Dev reload?
https://github.com/channelcat/sanic/issues/168
Write tutorials? github wiki pages
Flask:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world
Look at a mysql C client? We can do better than aiomysql. Either libmysqlclient or our own
https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html
https://dev.mysql.com/doc/internals/en/a-mysql-client-logs-in.html
If I want to try mrpacker again:
https://github.com/MarkReedZ/mrhttp/tree/cdaf7d8be3bcb0a88792875678aa2de515020566/src/mrhttp/utils