New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
finagle-http: Remove body (and Content-Length) from 1xx, 204 and 304 responses #521
finagle-http: Remove body (and Content-Length) from 1xx, 204 and 304 responses #521
Conversation
Thanks for submitting the PR! We should make a decision about where we want to stick this behavior, and what we want finagle-http to enforce on behalf of its users. I think that you're right that finagle should make it easy to follow the spec. With that said, we need to make sure it's at least bugwards migration compatible. For example, if your service is depending upon being able to do something that doesn't follow the spec, it would be nice to provide a clear migration to spec-compatibility for them. This might be as simple as making validation optional. Netty does this all the time, see validateHeaders I think in the long-run we probably want to disallow constructing requests that are illegal, or configuring requests in an illegal way. This will improve the user experience, since you'll fail when you do the illegal thing, rather than when you send the bad object into the client. In the mean time, would it make sense to make request / response validation into a new filter? This way, if folks decide that they really can't live with this behavior, they can simply remove the filter themselves. It also separates concerns neatly, where the HttpServerDispatcher stays (mostly) just state machine stuff. |
Thanks a lot for your quick and detailed response!
Yes, I like your idea and strongly agree with it. :) With regard to the actual implementation, we can 1) create a new filter and 2)add it to |
@monkey-mas 👍 that makes a lot of sense to me. |
6cbea75
to
5a983fa
Compare
5a983fa
to
91919ad
Compare
91919ad
to
656d289
Compare
@@ -31,6 +31,11 @@ private[finagle] class HttpServerDispatcher( | |||
service.close() | |||
} | |||
|
|||
private def mayHaveContent(status: Status): Boolean = status match { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need this anymore, since we have the new Filter now. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIU, (Http)ServerDispatcher#handle is called after all the filters are applied? In handle()
, we add a Content-Length header field with a value of 0 in here. If I understand finagle's behaviour correctly, I think we can't avoid putting the code there?
(but I have to admit this is not nice though :(
I might be wrong, please give me your idea :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have the same question as @mosesn. What's the verdict here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @taylorleese cc: @mosesn ,
I might be wrong but it seems like handle(res: Reponse) is called multiple times and it's called again after all filters are applied (s.t., even after ComplianceFilter
is applied) as I mentioned before. I found this when I debug my commit with sample code like this.
I think this means a Content-Length is added to a 204 response for example. Suppose we're going to return a 204 response;
1)we remove a Content-Length and message-body of a 204 response by using ComplianceFilter
2)but then handle
is called and this line adds a Content-Length with a value of 0 to a 204 response as we've already removed a Content-Length in 1) and !rep.contentLength.isDefined
holds.
To achieve what we want to do, i.e., send a 204 without a Content-Length and message-body for example, we need to have this condition here. The condition checks whether we send a Content-Length or not.
This is why I think we still need mayHaveContent
. Anyway, I need your help to sort this out. Can you please help me? :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @monkey-mas, I'm new to this code so feel free to let me know if what follows is incorrect!
I don't think that handle(res: Response)
is called multiple times: it's responsible for writing the message to the wire so if it's called multiple times we're in big trouble. I do think you're right in that its going to be called after all the filters so in its current form, its a problem.
It sounds like the problem is accommodating people sending a 204 with a body. In the case of people sending a 204 without a body, the content-length header gets set to 0. Maybe what we can do is have handle
look for a 204 response with a content-length: 0
header, and strip those from the response? That will make 'legit' 204 responses legal by rfc 7230, and give your filter a way to signal that the header should be stripped.
To make this happen, the single if
statement on 96 (old) might turn into:
rep.contentLength match {
case Some(0) if rep.status == Status.NoContent =>
rep.headers.remove(Fields.ContentLength)
case Some(_) => // NOOP: don't perturb explicitly set content lengths other than 204 with length 0.
case None =>
rep.contentLength = rep.content.length
}
There might be a cleaner way to implement that logic.
The filter would then set the content-length
headers for Status(304), set it to 0 for Status(204), and strip away the bodies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @bryce-anderson,
Thanks for your advice! (and sorry for my late reply!) Well I think I don't fully follow your strategy, can you give me more details please?
I do think you're right in that its going to be called after all the filters so in its current form, its a problem.
Thanks for checking it, yeah that's what I expected :)
It sounds like the problem is accommodating people sending a 204 with a body
Do you mean we don't provide support for removing a body-message of 204 and 304 responses if a body is explicitly sent by users?
In the case of people sending a 204 without a body, the content-length header gets set to 0
Umm.. I think Content-Length
header field is not set to 0 for a 204 response when there's no message-body. (But I think it's correct that the header field is added when we explicitly send a message-body, which is our problem here.)
More exactly, in the case of a 204 response with no body, rep.contentLength
is None
and rep.content.length
is 0
I suppose.
Oh BTY, do you assume to use ComplianceFilter
(or some filters) to handle the problem? It seems like you tried to solve the problem by just changing the original code at 96 in HttpServerDispatcher
? (I thought so cos we don't have to check if a 204 response has contentLength=Some(0)
after applying ComplianceFitler
.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TLDR: I've had some more time to think about it, and I'm not certain of an easy solution.
Umm.. I think Content-Length header field is not set to 0 for a 204 response when there's no message-body. (But I think it's correct that the header field is added when we explicitly send a message-body, which is our problem here.)
More exactly, in the case of a 204 response with no body, rep.contentLength is None and rep.content.length is 0 I suppose.
Sorry, its not clear to me if we agree. I believe at a Response
with an empty body will get assigned a content length header of length 0. I haven't added tags to the handle
function to check, but a raw Response
should not be chunked and will not have a defined contentLength
, so line 97 (on master, maybe this is the source of our confusion) should set the content-length to 0 on a 204 response without a body.
I don't think this can be solved only in the compliance Filter
, there needs to be some coupling because, as is, the content-length is going to be set for non-chunked bodies regardless of status code. I personally would hate to make the HttpServerDispatcher
aware of a single Filter
, that just doesn't seem fair to the other filters. 😄 That suggests to me (and we should get @mosesn's opinion in light of what we've found) that this isn't actually a job for a filter. This feels more like the things that are getting delegated to the netty pipeline such as RespondToExpectContinue
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, its not clear to me if we agree. I believe at a Response with an empty body will get assigned a content length header of length 0. I haven't added tags to the handle function to check, but a raw Response should not be chunked and will not have a defined contentLength, so line 97 (on master, maybe this is the source of our confusion) should set the content-length to 0 on a 204 response without a body.
Yes, I understand the code in the same way. Thanks for your explanation! :) (I was thinking about Content-Length
of rep
where the code on line 97 is not applied.)
That suggests to me (and we should get @mosesn's opinion in light of what we've found) that this isn't actually a job for a filter. This feels more like the things that are getting delegated to the netty pipeline such as RespondToExpectContinue.
Okay, so you mean we create a ChannelHandler
, like RespondToExpectContinue
, instead of using a Filter
and give it a responsibility to handle 204 and 304 responses? Correct? Honestly, I'm not familiar with Netty's pipeline and not quite sure how it works in finagle. What sort of high-level solution (or actual implementation) do you have in your mind? (Can you please be more specific?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would definately wayt for @mosesn to chime in before going down the netty pipeline route. As a preview, I'll give my understanding of it. The code at https://github.com/twitter/finagle/blob/develop/finagle-http/src/main/scala/com/twitter/finagle/http/Codec.scala#L307 gives an example of how the RespondToExpectContinue
handler works, but it is an UpstreamHandler
(I believe that corresponds to InboundHandler
in netty4). The DownstreamHandler
works essentially the same but in the other direction. In my minds eye, the DownstreamHandler
intercepts the HttpResponse
and drops the body and requisite headers.
One thing that occurs to me is that streaming responses (chunked) don't appear to be addressed in this commit. The way streaming responses are rendered appears more complex and might make using the netty pipeline more tricky, if even possible. For example, In the case of a 304 or 204 I doubt that we want to use the Reader
, and instead we would want to discard it. I don't think that is a possibility, at least in the place I pointed you to above.
LGTM at a high level. Can you also add a note to finagle/CHANGES? I think this is worth calling out for our users. |
e2a4b9a
to
f0856db
Compare
Problem We currently have two problems. Firstly, as described in RFC2616[1] and RFC7231[2], 204 and 304 responses must not contain a message-body. This requirement was not enforced in finagle, i.e., a message-body can be added. Moreover, a Content-Length header field can be added to these responses, which is not allowed as mentioned in RFC7230#section-3.3.2[3]. Solution First of all, we have to clear a message-body, which we call content in finagle, if the response status code is either 204 or 304 to satisfy RFC7231. Secondly, we need to make sure not to add a Content-Length header field in a 204 response to follow RFC7230#section-3.3.2. With regard to a 304 response, a server may send a Content-Length header as mentioned in the same section. Thus it'd be better for us to allow users to set the field with a value of 0 if they like. Most importantly, we make sure that these things are achieved and the new system is bugwards-compatible by provding a new filter.[4] Result We follow the specification of RFC with regard to a message-header and message-body except that we have not added some header fields required in a response with status code 304 to satisfy RFC7232#section-4.1[5]; Any of the following header fields must be generated: Cache-Control, Content-Location, Date, ETag, Expires, and Vary. As we achieve this by creating a new filter and set it to Http.scala[6], users who are not willing to make use of the functionality can easily disable it by removing the filter. [1] https://tools.ietf.org/html/rfc2616 [2] https://tools.ietf.org/html/rfc7231 [3] https://tools.ietf.org/html/rfc7230#section-3.3.2 [4] twitter#521 (comment) [5] https://tools.ietf.org/html/rfc7232#section-4.1 [6] https://github.com/monkey-mas/finagle/blob/finagle-6.35.0/finagle-http/src/main/scala/com/twitter/finagle/Http.scala#L298-L304
Thanks for the review! Except this point, I think the fix commit should be fine. :) I'll squash additional commits into one after everything is ok. Looking forward to your feedback! 😀 |
f0856db
to
40f9b2e
Compare
logger.warning(s"Response with a status code of ${rep.statusCode} must not have a Content-Length header field thus the field has been removed.") | ||
} | ||
|
||
//Make sure a Content-Length header field is set with a value of 0 if it is explicitly sent for a 304 response |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, I think I was wrong. As RFC 7230 section3.3.2 says, if we do want to send a Content-Length header field, we should return the Content-Length value of the cached content which has been returned to a client in a 200 response. I thought a value of Content-Length header field was always 0 as the content of a 304 response is always empty...
The following is the citation from the section;
A server MAY send a Content-Length header field in a 304 (Not Modified) response to a conditional GET request (Section 4.1 of [RFC7232]); a server MUST NOT send Content-Length in such a response unless its field-value equals the decimal number of octets that would have been sent in the payload body of a 200 (OK) response to the same request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof, this doesn't feel like it's easy to know. I guess this is something that users must set? I guess it's better to just not modify the content length in this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess this is something that users must set?
I think so if users really want to send the header field in a 304 response... I'm not sure how to validate that they've chosen the correct Content-Length value. As the RFC says, if the field-value doesn't equal the decimal number of octets that would have been sent in the payload body of a 200 (OK) response to the same request then we MUST NOT send Content-Length.
I guess it's better to just not modify the content length in this case.
Yeah, not modifying is better I suppose so. Will fix it :)
I feel that things will be more complicated if we start to think about a Content-Length value of a 304 response as I've mentioned. Do you think we could stick to the original problems we had before and hopefully we can find a way to handle the new issue we've found now? What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed it :)
@monkey-mas sorry for the delay, haven't had a chance to take a look this week. but I haven't forgotten about you! |
@mosesn haha, sure sure. :) Please have a look when you have time and let me know then! |
Okay @monkey-mas, I have discussed this with @mosesn and there may be another way to get this done in a bugwards compatible way. A lot of the functionality in the HttpServerDispatcher.handle method deals with similar issues to what you're addressing, namely ensuring that the What do you think? |
@bryce-anderson, cc: @mosesn
Just to make sure, do you mean we leave If so, I think we need to make two things clear:
What do you think? |
@monkey-mas good catch with that comment. maybe the first step is to fix netty's content compressor? then we can make the rest of these changes more safely. |
@monkey-mas, I'm sorry we've been slow on this. We have started to move in the direction needed to incorporate your commit. See fa8cef1 |
- fix not to use ChannelBuffers.EMPTY_BUFFER - handle 1xx status code condition by checking 100 <= status < 200 - fix cols <= 100 - add payload size to error log
- remove netty dependency - move test(...) into a loop
- remove unnecessary end-to-end test cases - move testIfImplemented(...) in a loop - fix cols <= 100
@monkey-mas sorry, we dropped the ball on this! I'm going to see if I can get this over the goalpost and merge it in. Will keep you posted! |
this is failing in really strange ways. I think the messiness of finagle's HTTP compliance stuff is catching up to us. I'm going to keep on trying to make it work, but might have to give up if it takes too much time. If you have some time to look at this again, I'd be happy to send you what I've done so far. |
@mosesn
Honestly, I've got no idea why it fails. You have some clue, don't you? I think I can have some time to fix it. Can you please give me some information you've collected so far? |
@monkey-mas I ended up undoing most of my changes since they didn't seem to be helping. I'm not sure where the issue is, but I've merged against master and made a PR: monkey-mas#1 |
@mosesn Thanks so much! Let me have a look at it :) |
@mosesn @bryce-anderson ProblemThe current (wierd) problem we have is our response gives us
Based on the fact that the unit test What I've observed so farI've done some print-debugging (& stack tracing) and found our code works as follows:
Additionally, I roughly checked if our code adds the header somewhere, by using Based on this, I think (Naive) SolutionWith that said I'm still not 100% for sure how and when @@ -88,15 +90,21 @@ private[finagle] object Bijections {
}
def fullResponseToFinagle(rep: NettyHttp.FullHttpResponse): FinagleHttp.Response = {
val payload = ByteBufAsBuf(rep.content)
+ val status = rep.status
val resp = FinagleHttp.Response(
versionToFinagle(rep.protocolVersion),
- statusToFinagle(rep.status),
+ statusToFinagle(status),
BufReader(payload)
)
resp.setChunked(false)
+ if (status.codeClass() == HttpStatusClass.INFORMATIONAL || status == HttpResponseStatus.NO_CONTENT) {
+ rep.headers().remove("Content-Length")
+ }
writeNettyHeadersToFinagle(rep.headers, resp.headerMap)
resp.content = payload |
Good sleuthing! I would love to know why content length is being set to 0, but I'm OK with pulling this in before we know. @bryce-anderson what do you think? |
Finally I determined the cause of The following is the rough sketch of (reversed) execution paths to encounter
As I've mentioned before, the header's already set to 0 when
As
ref. readableBytes Therefore, |
I'll pull it in and take another look! Sounds like this is further down in the pipeline a bit, so this is more interesting than we'd hoped. :) Excellent sleuthing @monkey-mas. |
@monkey-mas, I've looked through your PR and wanted to get your take: it does look like Netty is responsible for the test failures as you've laid out above. How do you want to proceed? In my mind's eye, Netty should probably be fixed, so if you're interested in contributing to that project I think you've got a nice contribution on deck. In the meantime, we could disable the end-to-end tests that are failing due to the client issue. What do you think? |
@bryce-anderson Sorry for my late reply.
Do you mean we can just merge functionality of
Cool! I'll open a PR related to this issue though I'm not quite sure if Netty community likes the change to follow RFC7230. :( |
@monkey-mas, no worries! I'm in the process of merging this in with the end-to-end tests disabled: this works as expected when running a simple telnet session. I feel the Netty folks are generally quite receptive to patches that make sure they don't break things, which this seems to be an instance of. They already have similar logic laying around to deal with cases where a body shouldn't exist, for example here. If you ask me, that code would be better off renamed and in the HttpUtil object. |
@monkey-mas, after long last, this was merged as commit 1e4252e. Thanks a lot for both your patience and good work! Keep us updated on the status of any Netty PR you submit to address the chunk aggregation issue. |
@bryce-anderson @mosesn Okay, I'll get started with the aggregation issue. The easiest solution could be to modify the logic of finishAggregation(...) as this should be the last point before an actual response is returned to a client. |
@monkey-mas 👍 |
Hey @monkey-mas! We had to revert this temporary (see b5cbb3e) for some internal reasons. We'll put it back as soon as we can. Stay tuned and thanks again - this is a solid piece of work! |
Oops sorry to hear that :( I hope all is well at your end. |
Hi @monkey-mas! I'm trying to put your work back where it belongs. Will update this thread when it's merged in. |
It's merged (see f8208cd)! |
Problem
We currently have two problems. Firstly, as described in RFC2616 andRFC7231, 204 and 304 responses must not contain a message-body. This
requirement was not enforced in finagle, i.e., a message-body can be
added. Moreover, a Content-Length header field can be added to these responses,
which is not allowed as mentioned in RFC7230#section-3.3.2.
Solution
First of all, we have to clear a message-body, which we call content in finagle,if the response status code is either 204 or 304 to satisfy RFC7231. Secondly,
we need to make sure not to add a Content-Length header field in a 204 response
to follow RFC7230#section-3.3.2. With regard to a 304 response, a server may
send a Content-Length header as mentioned in the same section. Thus it'd be
better for us to allow users to set the field with a value of 0 if they like.
Most importantly, we make sure that these things are achieved and the new system
is bugwards-compatible by provding a new filter.
Result
We follow the specification of RFC with regard to a message-header and
message-body except that we have not added some header fields required in a
response with status code 304 to satisfy RFC7232#section-4.1; Any of the
following header fields must be generated: Cache-Control, Content-Location,
Date, ETag, Expires, and Vary.
As we achieve this by creating a new filter and set it to Http.scala, userswho are not willing to make use of the functionality can easily disable it
by removing the filter.