RESPONSE.write does not stream data in wsgi #662
Comments
|
Jérome Perrin wrote at 2019-6-24 19:07 -0700:
What seem to be a regression in the ZServer -> wsgi migration is that `RESPONSE.write` no longer send the response data to the client in a streaming way but just send everything when the published method returns.
For example this in a script python (with ID `stream_test` at the root of application):
```python
request = container.REQUEST
response = request.RESPONSE
for i in range(1000):
for x in range(1000):
y = x ** x # this introduce a delay
response.write(b'.')
```
if I use `curl --no-buffer http://127.0.0.1:8080/stream_test`, with ZPublisher we can see the `.` printed one after the other, but with wsgi we see all the `.....` after a delay.
`RESPONSE.write` is (still) documented in zope book as a way to stream data: https://github.com/zopefoundation/Zope/blob/e9f10101b8a4db64a2440e3a51561a7805ea44f5/docs/zdgbook/ObjectPublishing.rst#L1258-L1267
It's also used in `OFS.File` to stream file chunks:
https://github.com/zopefoundation/Zope/blob/b9a346903fe4a6874face0930876184ac321853e/src/OFS/Image.py#L279-L298
[`WSGIResponse.write`](
https://github.com/zopefoundation/Zope/blob/e9f10101b8a4db64a2440e3a51561a7805ea44f5/src/ZPublisher/HTTPResponse.py#L1065-L1077) docstring also states that "This allows the browser to display partial results while computation of a response to proceed", but it's currently not the case.
Your example requires a lot of support (which may not all be in place):
In order to use `Response.write`, usually the final `content-length`
must be known. The reason for this: even if the response is streamed,
the response header must be produced before the body starts.
Without a valid content length, the server must either use a
chunked response or close the communication channel
to indicate that the response is finished.
For efficiency reasons, the server (component) might cache some output
before it decides what to do.
WSGI itself provides strong control over the delivery of response
parts; thus, you can precisely control streaming in principle.
However, WSGI support for `Response.write` requires a special
(WSGI) interface method the use of which WSGI discourages.
The `ZPublisher` integration may support only the typical
use case for `Response.write` (avoid to construction of a huge
structure in memory) and not precise streaming control.
Note, that in general `Response.write` should not wait until
the chunk is delivered (what WSGI's streaming contol would enforce).
Note, that `ZServer` (unlike WSGI) does not provide precise streaming control.
`Response.write` simply gives a body part to `ZServer` which this
may deliver - at its discretion.
|
|
Thanks for the feedback, I did not know that of that "discouraged interface". I assume this is the "deprecated method" from the examples in https://stackoverflow.com/a/16775731 , ie: def application(environ, start_response):
write = start_response(status, headers)
write('content block 1')
write('content block 2')
write('content block 3')
return NoneIf I try a quick and dirty patch doing something like this: response.write = start_response(status, headers)
...then the response data seems properly sent chunk by chunk with waitress and other things look working ... but that's a discouraged approach. About Line 264 in b9a3469 Not sending a The use case I have in mind is the one you could guess, prevent the server from having to keep the full structure in memory, like I'm still wondering if there's a way for |
|
Hi @perrinjerome Once a proper solution is found, please update the documentation accordingly. @d-maurer Thanks for your explanation. In my Zope app, once there was a newsletter module, which streamed the Response and stopped working some time - even under Zope 2.13 - your hint may explain why. Meanwhile the newsletter module got deleted anyway. |
I don't see how that can be done. I know nothing in Python to turn a classic call stack (from mapply to RESPONSE.write) into a kind of coroutine.
Then if there's no discouraged approach, let's rather remove With the switch from Medusa to WSGI, we already broke all applications that stream data and I suppose many have already switched to stream iterators. Another problem with |
|
Julien Muchembled wrote at 2019-11-5 08:12 -0800:
...
Then if there's no discouraged approach, let's rather **remove** `RESPONSE.write`.
I still (current master) see a use of `Response.write` in `OFS.Image.File`
(line 388 and others). It is used there to decouple the ZODB use
from the delivery of the file content -- such that downloading a huge
file does not need a ZODB connection during the whole download time.
In my view, we should think about removing `Response.write`
only after at least this use case has another good solution.
|
|
We are talking about the fact that your case is already broken: with WSGI, it makes no difference between:
BTW, I am surprised by your example because one usually prefer the contrary: stream data directly from the DB to the user agent, without reading everything first into RAM or disk (someone pausing the download of a big file should cost nothing besides a few file descriptors). For this use case, there's nothing anymore:
Sure. It's premature to clean up code if there's anyway no other good solution. But my comment is rather to announce the direction I'd like we choose. |
|
Julien Muchembled wrote at 2019-11-5 11:32 -0800:
We are talking about the fact that your case is already broken: with WSGI, it makes no difference between:
- using `Response.write`
- make your `OFS.Image.File` code instanciate a `io.BytesIO()` object, fill it and return `.getvalue()`
Then, likely, the WSGI interface should improve!
BTW, I am surprised by your example because one usually prefer the contrary: stream data directly from the DB to the user agent, without reading everything first into RAM or disk (someone pausing the download of a big file should cost nothing besides a few file descriptors).
Of course, this requires a database connection -- during the whole
delivery of the file content. The required time depends on the bandwidth
which is not under the control of the server.
With Zope the database connection management is implicit:
as long as a database connection is open, a worker thread is blocked.
It is therefore vital to make the time the connection needs to be open
independent on things not under server control.
The `ZServer` interface was aware of these issues and supported them
(by transparently buffering, if necessary)
The `WSGI` interface should do so as well.
In my opinion, `Response.write` is an adequate interface to do this.
|
|
It looks like the future is ASGI and if one solution for this current issue brings us closer to ASGI, we should choose it. I looked at the WSGI compatibility of https://github.com/encode/uvicorn and their IOW, there are WSGI implementations that simply don't implement the
I didn't know about the transparent buffering for content bigger than 128k (each chunk bigger than 200 bytes goes via a temporary file). That's actually something we don't want, so it should not be hardcoded. By trying to avoid reaching the maximum number of ZODB connections, you get a lot of IO and you could fill the disk. That can also cause useless workload if the client decides after all to cancel its request. I don't see the problem with increasing the maximum number of ZODB connections and workers (currently threads but not necessarily in the future). |
|
I've just looked at the code of waitress (that's what we chose for WSGI) and it does the same kind of transparent buffering as ZServer. What's strange is that it also buffers to file for unbound streams. All this looks bad for me :( |
|
Julien Muchembled wrote at 2019-11-14 07:32 -0800:
...
> The `ZServer` interface was aware of these issues and supported them (by transparently buffering, if necessary)
I didn't know about the transparent buffering for content bigger than 128k (each chunk bigger than 200 bytes goes via a temporary file). That's actually something we don't want, so it should not be hardcoded.
The buffering is an elegant way to decouple request processing
and response delivery. Decoupling is a good thing as response delivery
can take arbitrarily long (depending on actions on the client
and the available bandwidth).
By trying to avoid reaching the maximum number of ZODB connections, you get a lot of IO and you could fill the disk.
Otherwise, you use an large amount of ZODB connections (limited by
the availability of file descriptors)
and those are much rarer than disk space.
If you do not decouple response delivery and request processing,
I can easily design a DOS attack: perform a large number of requests
and refuse to read the responses. Thus, you must have some form
of decoupling - disk buffering is one of the easiest ways.
|
I was wrong in a previous comment. A ZODB Connection (or a thread) does not use any file descriptor (what may use file descriptors is the ZODB Storage from which Connections get data, and the number of file descriptors does not depend on the number of Connections). You may argue that it's specific to ZODB and other DB like MariaDB would use 1 socket per connection. On the other side, current implementations of transparent buffering use 1 fd per request (for the temporary file). Sure, this could be improved with something more complex like a single sqlite3 DB buffering data for all requests, and even freeing data as soon as it's sent to the client.
Currently, it looks like the decoupling was not implemented for this reason: at least the fd argument is in favor of my position. Anyway, that's not where we can protect from a DoS attack. By not decoupling, it is indeed trivial to reach the maximum number of requests if one finds a request that returns too much data. With decoupling, 1 request finishes from time to time, giving some client a chance to get a slot, but frankly, the attacker can simply continue to open requests. I doubt decoupling helps significantly here (I mean the site is down for most users), whereas the cost is high: disk filled, wasted IO/CPU to fill the buffer. |
|
Julien Muchembled wrote at 2019-11-14 15:51 -0800:
> Otherwise, you use an large amount of ZODB connections (limited by the availability of file descriptors)
I was wrong in a previous comment. A ZODB Connection (or a thread) does not use any file descriptor
I am aware of this. My remark "limited by the availability of
file descriptors" refers to my DOS szenario -- create lots of
requests and refuse to read the responses -- each such request
uses a connection to the server and thus a file descriptor.
...
Currently, it looks like the decoupling was not implemented for this reason: at least the fd argument is in favor of my position.
`OFS.Image.File` uses `Response.write` in order to decouple the
request processing (bound to one of very few worker threads) from
the response delivery (with its timing completely outside the control
of the server).
When I looked at the `ZServer` code, I got the impression
that many of its design decisions have been influenced by
the uncontrollable delivery time; among them the disk buffering
for large responses. Think about it: if the server produces more
data than the client is ready to consume, something must be done
with the excess. Something like `ZServer` can either drop it (potentially
punishing well behaving clients in unfavorable situations (small bandwidth))
or keep it. Apparently, `ZServer` decided to keep it - and to use
disk space as a relatively cheap resource for this.
I am aware that you would like e.g. in the case of `OFS.Image.File`
to use a stream iterator in order to interleave reading from the ZODB
and delivering partial data (and thus avoiding large intermediate data).
This would require a new ZODB connection (not associated with
the current request) and non trivial transaction handling
(to ensure that the new ZODB connection sees potential modifications
by the current request). It can be done but it is considerably
more difficult than the use of `Response.write`.
Anyway, that's not where we can protect from a DoS attack. By not decoupling, it is indeed trivial to reach the maximum number of requests if one finds a request that returns too much data. With decoupling, 1 request finishes from time to time, giving some client a chance to get a slot, but frankly, the attacker can simply continue to open requests. I doubt decoupling helps significantly here (I mean the site is down for most users), whereas the cost is high: disk filled, wasted IO/CPU to fill the buffer.
The decoupling was likely not done to prevent DOS attacks but
to ensure satisfactory behaviour despite delivery odds (like low
bandwith, temporary network problems, etc.) at low costs
(use disk space rather than RAM).
|
That depends on the storage implementation. A storage that natively implements |
|
Oops, I forgot RelStorage.
Because of the GIL and the connection cache, one may indeed limit the number of threads within a process that serves heterogeneous requests. Sometimes, it's possible to specialize processes using a balancer, and then you can configure as many threads as you want, and small connection cache to avoid wasting RAM. (With everything that has been said previously, I assume FDs aren't an issue: about them, there's already one for the socket to the client so an extra for the DB is in the same order of magnitude, current buffering implementations are bad, and the limit of FDs can be increased.)
This does not require a new ZODB connection. This can be done with the same connection as the one used for the request, by postponing the closing of the connection after the iteration is over. This is trivial with the deprecated WSGI write API. With the recommended WSGI API, it is a bit tricky, as you can see at https://lab.nexedi.com/nexedi/erp5/commit/4ca4d7f65393d2caf08abfa774ef9a52b5497f63: the iterator is wrapped within a You also seem to mix up the use of I didn't know that ZServer and waitress already do buffering, which means that buffering is off-topic (if I want to get this feature removed, I'll discuss elsewhere) and WSGIPublisher does redundant buffering. So the way I'd like this issue to be resolved is, at least:
|
|
Julien Muchembled wrote at 2019-11-18 11:04 -0800:
...
> I am aware that you would like e.g. in the case of `OFS.Image.File` to use a stream iterator in order to interleave reading from the ZODB and delivering partial data (and thus avoiding large intermediate data). This would require a new ZODB connection (not associated with the current request) and non trivial transaction handling (to ensure that the new ZODB connection sees potential modifications by the current request). It can be done but it is considerably more difficult than the use of `Response.write`.
This does not require a new ZODB connection. This can be done with the same connection as the one used for the request, by postponing the closing of the connection after the iteration is over.
Currently, there is an automatism which couples the request, the ZODB
connection and the worker thread. If you drop "Response.write"
in preference to keeping the ZODB connection longer open
you must invent something which allows you to control this automatism
and I have my doubts that this will be more natural.
...
So the way I'd like this issue to be resolved is, at least:
- removing `Response.write`
- removing the use of BytesIO from WSGIPublisher.py
Apparently, someone needed a streaming facility and
found that `Response.write` does not do this under WSGI
(unlike `ZServer`). The most natural way would be
make `Response.write` stream under WSGI as well
and not to drop `Response.write` altogether.
Any solution you come up with must at least well accomodate
the use case in `OFS.Image.File`. Show us such a complete
solution and maybe it will be convincing...
|
|
BTW, @d-maurer which WSGI implementations do you use ? |
|
Julien Muchembled wrote at 2019-11-19 07:37 -0800:
BTW, @d-maurer which WSGI implementations do you use ?
The default one (i.e. `waitress`).
|
|
This discussion focuses on the use case of |
|
FYI: We added an item to our roadmap (at PerFact) that (possibly large) static files should no longer be served by Zope, relying on a lighttpd running in parallel for these instead (maybe with Zope handling the authentication). We just are under the impression that this use case is not one of Zope's strong points, in part due to the issue discussed here. |
What seem to be a regression in the ZServer -> wsgi migration is that
RESPONSE.writeno longer send the response data to the client in a streaming way but just send everything when the published method returns.For example this in a script python (with ID
stream_testat the root of application):if I use
curl --no-buffer http://127.0.0.1:8080/stream_test, with ZPublisher we can see the.printed one after the other, but with wsgi we see all the.....after a delay.RESPONSE.writeis (still) documented in zope book as a way to stream data:Zope/docs/zdgbook/ObjectPublishing.rst
Lines 1258 to 1267 in e9f1010
It's also used in
OFS.Fileto stream file chunks:Zope/src/OFS/Image.py
Lines 279 to 298 in b9a3469
WSGIResponse.writedocstring also states that "This allows the browser to display partial results while computation of a response to proceed", but it's currently not the case.The text was updated successfully, but these errors were encountered: