Ensure response headers are being sent before closing the response #75

josejulio · 2018-02-08T05:29:42Z

Fixes cherrypy/cherrypy#1404

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bugfix

What is the related issue number (starting with #)

cherrypy/cherrypy#1404

What is the current behavior? (You can also link to an open issue here)

on_end_request hook is being call before sending a response on cherrypy

What is the new behavior (if this is a feature change)?

on_end_request hook is being call after sending a response on cherrypy

Other information:

I found Lawouach/WebSocket-for-Python#158 which lead me to this "PR" https://bitbucket.org/cherrypy/cherrypy/pull-requests/121/fixing-issue-1404-on_end_request-hook-is/diff
I ported that to cheroot and addressed the last comment from "Sylvain Hellegouarch"

codecov · 2018-02-09T04:39:37Z

Codecov Report

Merging #75 into master will increase coverage by 0.33%.
The diff coverage is 96.96%.

@@           Coverage Diff            @@
##           master    #75      +/-   ##
========================================
+ Coverage   65.67%    66%   +0.33%     
========================================
  Files          15     15              
  Lines        2753   2780      +27     
========================================
+ Hits         1808   1835      +27     
  Misses        945    945

webknjaz · 2018-02-09T10:41:06Z

Original PR authors/reviewers seem to be: @moigagoo @ronin8600 @Lawouach

webknjaz · 2018-02-09T10:44:38Z

Thanks for the PR.
I cannot accept it w/o manually checking the flow you're changing, so I'll do this once I have some time. This should also trigger checks against CherryPy itself.
Covering this with tests might speed up the process.

josejulio · 2018-02-09T16:01:07Z

Thanks @webknjaz
I'll try to make time to write test.
Basically this writes the headers before closing the request.
See here and here

I suppose (haven't digged much on cherrypy code) cherrypy hooks on that close and triggers the on_end_request

josejulio · 2018-02-13T05:44:25Z

@webknjaz I added a test, I verified that it fails without this patch - and it passes with the patch :) -

Let me know how this can be improved.

webknjaz · 2018-02-13T08:27:45Z

@josejulio first of all, linters in CI fail: https://travis-ci.org/cherrypy/cheroot/jobs/340814348#L484
We use fail fast approach in our CI to save on computing resources and power, so tests are not even being run if fast checks crash.
Could you please correct this?

P.S. Also, it's possible to pip install pre-commit && pre-commit install locally to set up automatic linter running for each commit you do on your machine :) So that you won't wait for CI to be triggered and return the result.

webknjaz

Plus code needs to bypass linters

webknjaz · 2018-02-13T08:37:27Z

cheroot/test/test_core.py

+
+def test_send_header_before_closing(testing_server_close):
+    """Tests we are actually sending the headers before closing the response."""
+    testing_server_close.server_client.get('/')


How about checking http response headers in client?

resp_status_line, resp_headers, resp_body = testing_server_close.server_client.get('/')

Sorry, maybe my description is wrong here or i didn't understand well.

The headers are always sent, the problem is that the close [1] method is being call before the headers are being sent.

This is a problem in cherrypy, because the on_end_request is being called before the headers are actually sent, anything sent to the client from this hook will reach before the headers, thus the headers will be malformed. I don't think I can test this from the client side unless i can get a raw response, but i would still need to send something on that close [1] method.

It's done in a similar fashion in pep-0333 [2] (talking about the fix, not the tests)

try: for data in result: if data: # don't send headers until body appears write(data) if not headers_sent: write('') # send headers now if body was empty finally: if hasattr(result, 'close'): result.close()

[1] https://github.com/cherrypy/cheroot/pull/75/files#diff-08d927a6f63bcf494ed88cca2a6873d0R144
[2] https://www.python.org/dev/peps/pep-0333/

webknjaz · 2018-02-13T08:38:21Z

cheroot/test/test_core.py

+    def close(self):
+        """Hook for close, tests if headers already sent."""
+        self.sent_headers_before_closing = \
+            self.req is not None and self.req.sent_headers


This seems to be overengineered. Please see my comment below.

josejulio · 2018-02-16T05:58:01Z

@webknjaz I fixed linting issues and tried to simplify the test.

Here is the output of the test without the fix (commit: 742b1b9)

―――――――――――――――――――――――――――――――――――――― test_send_header_before_closing ――――――――――――――――――――――――――――――――――――――

testing_server_close = <cheroot.wsgi.Server object at 0x7fc57f98a350>

    def test_send_header_before_closing(testing_server_close):
        """Test we are actually sending the headers before calling 'close'."""
        _, _, resp_body = testing_server_close.server_client.get('/')
>       assert resp_body == 'hello'
E       AssertionError: assert 'helloHTTP/1....\r\n0\r\n\r\n' == 'hello'
E         + hello
E         - helloHTTP/1.1 200 OK\r
E         - Content-Type: text/html\r
E         - Transfer-Encoding: chunked\r
E         - Date: Fri, 16 Feb 2018 05:30:56 GMT\r
E         - Server: Cheroot/6.0.1.dev103+gc17bff6d\r
E         - \r...
E         
E         ...Full output truncated (3 lines hidden), use '-vv' to show

cheroot/test/test_core.py:402: AssertionError

josejulio · 2018-03-01T19:00:46Z

@webknjaz @jaraco @Lawouach
Is there anything else I can do here? Do you have additional comments on the tests?
Thanks.

webknjaz · 2018-03-21T12:01:13Z

cheroot/server.py


-        if (self.ready and not self.sent_headers):
-            self.sent_headers = True
-            self.send_headers()


Why completely remove this? How will it affect non-WSGI plain HTTP server?

I though that adding it to self.server.gateway(self).respond() would cover it

I suppose I can add it back to ensure we are always sending the headers in case the gateway is not sending the headers.

webknjaz · 2018-03-21T12:18:10Z

cheroot/test/test_core.py

+
+    def close(self):
+        """Hook for close, write hello."""
+        self.req.write('hello'.encode('utf-8'))


b'hello' would look nicer

webknjaz · 2018-03-21T12:19:46Z

cheroot/wsgi.py

                self.write(chunk)
        finally:
+            # Send headers if not already sent
+            if not self.req.sent_headers:


It's always paired with "if not sent" check. Try moving it inside the method and maybe rename it into smth like ensure_headers_sent or so

webknjaz · 2018-03-21T12:20:56Z

@josejulio sorry for late reply, I've added some thoughts on this

josejulio · 2018-03-21T18:18:07Z

@josejulio sorry for late reply, I've added some thoughts on this

Better late than never :).
I'll address your comments later today, thanks!

Moved the test if the headers are sent inside the ensure_headers_sent (renamed from write_headers) Puts back a "send headers" test.

josejulio · 2018-03-22T05:05:14Z

@webknjaz Addressed the comments, travis is green and this is ready for other review round, thanks!

webknjaz · 2018-03-26T22:38:56Z

cheroot/server.py

            self.rfile = KnownLengthRFile(self.conn.rfile, cl)

        self.server.gateway(self).respond()
+        self.ensure_headers_sent()


It looks like you've lost ready check here. Could be:

self.ready and self.ensure_headers_sent()

Is this intentional? What is the effect of a such change?

Not intentional, missed that, thanks for spotting it!

josejulio · 2018-04-04T04:19:03Z

@webknjaz Forgot to say that addressed your latest comments, thanks

webknjaz · 2018-04-04T08:21:26Z

LGTM! Thank you @josejulio!

webknjaz · 2018-04-04T22:20:00Z

@josejulio This code should reach PYPI soon, when this build is complete: https://travis-ci.org/cherrypy/cheroot/builds/362369631

It should become available at https://pypi.org/project/Cheroot/6.1.0/ in about 30 min from now.

josejulio · 2018-04-05T03:12:15Z

Awesome, thank you!

josejulio force-pushed the on_end_request_before_response branch 2 times, most recently from 959fd81 to ebedc24 Compare February 9, 2018 04:39

webknjaz requested review from a team, Lawouach, jaraco and webknjaz February 9, 2018 10:39

Ensure response headers are being sent before closing the response

742b1b9

josejulio force-pushed the on_end_request_before_response branch 2 times, most recently from 482f05b to 562913c Compare February 13, 2018 05:32

josejulio changed the title ~~Ensure response headers are being sent before calling on_end_request hook~~ Ensure response headers are being sent before closing the response Feb 13, 2018

Add tests to ensure headers are being sent before closing the response

c17bff6

josejulio force-pushed the on_end_request_before_response branch from 562913c to c17bff6 Compare February 13, 2018 05:44

webknjaz requested changes Feb 13, 2018

View reviewed changes

josejulio added 2 commits February 15, 2018 23:00

Fixes linting issues

a9932e1

Attempt to simplify the tests

bb2ddde

josejulio force-pushed the on_end_request_before_response branch from 376de01 to bb2ddde Compare February 16, 2018 05:44

webknjaz reviewed Mar 21, 2018

View reviewed changes

Addressed review comments

62ea2c7

Moved the test if the headers are sent inside the ensure_headers_sent (renamed from write_headers) Puts back a "send headers" test.

webknjaz reviewed Mar 26, 2018

View reviewed changes

Put back the "ready" check

af90e13

webknjaz approved these changes Apr 4, 2018

View reviewed changes

webknjaz merged commit 513f789 into cherrypy:master Apr 4, 2018

josejulio deleted the on_end_request_before_response branch July 27, 2023 17:42

Uh oh!

Ensure response headers are being sent before closing the response #75

Ensure response headers are being sent before closing the response #75

Conversation

josejulio commented Feb 8, 2018

Uh oh!

codecov bot commented Feb 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

webknjaz commented Feb 9, 2018

Uh oh!

webknjaz commented Feb 9, 2018

Uh oh!

josejulio commented Feb 9, 2018

Uh oh!

josejulio commented Feb 13, 2018

Uh oh!

webknjaz commented Feb 13, 2018

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josejulio Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josejulio commented Feb 16, 2018

Uh oh!

josejulio commented Mar 1, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz commented Mar 21, 2018

Uh oh!

josejulio commented Mar 21, 2018

Uh oh!

josejulio commented Mar 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josejulio commented Apr 4, 2018

Uh oh!

webknjaz commented Apr 4, 2018

Uh oh!

webknjaz commented Apr 4, 2018

Uh oh!

josejulio commented Apr 5, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Feb 9, 2018 •

edited

Loading

josejulio Feb 13, 2018 •

edited

Loading