HttpPostMultipartRequestDecoder should decode header field parameters #7265

dminkovsky · 2017-09-29T22:50:23Z

Motivation:

I am receiving a mutlipart/form_data upload from a Mailgun webhook. This webhook used to send parts like this:

--74e78d11b0214bdcbc2f86491eeb4902
Content-Disposition: form-data; name="attachment-2"; filename="attached_Ñ�Ð°Ð¹Ð».txt"
Content-Type: text/plain
Content-Length: 32

This is the content of the file

--74e78d11b0214bdcbc2f86491eeb4902--

but now it posts parts like this:

--74e78d11b0214bdcbc2f86491eeb4902
Content-Disposition: form-data; name="attachment-2"; filename*=utf-8''attached_%D1%84%D0%B0%D0%B9%D0%BB.txt

This is the content of the file

--74e78d11b0214bdcbc2f86491eeb4902--

This new format uses field parameter encoding described in RFC 5987. More about this encoding can be found here.

Netty does not parse this format. The result is the filename is not decoded and the part is not parsed into a FileUpload.

Modification:

Added failing test in HttpPostRequestDecoderTest.java and updated HttpPostMultipartRequestDecoder.java
Refactored to please Netkins

Result:

Fixes #7265 (this):

HttpPostMultipartRequestDecoder identifies the RFC 5987 format and parses it.
Previous functionality is retained.

dminkovsky · 2017-09-30T01:27:50Z

First commit on PR is the actual patch.

Second commit was required to please netkins. I was already 5 indents in.

fenik17 · 2017-09-30T08:08:59Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

+            if (!shouldDecode) {
+                value = value.substring(1, value.length() - 1);
+            } else {
+                String[] split = value.split("''", 2);


Consider replacing String#split with use precompiled Pattern constant or a value.indexOf("''").

Thank you. I don't have much experience in performant Java string work, so I was following the lead of existing code a few lines up where you have value.split("=", 2). Does the compiler/runtime optimize this case because it's a unit-length string and therefore effectively a char?

Yep. For one-char string a String.split in OpenJDK doesn't use Pattern. But the string "''" has two chars.

Ah okay thanks. I opted for the precompiled pattern because it seemed simpler.

fenik17 · 2017-09-30T08:13:35Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

    }

+    private Attribute getContentDispositionAttribute(String... values) {
+        Attribute attribute;


Why is it necessary to declare the attribute here?

That is how it was before my patch on 4.1. For the actual test/patch, please see my first commit (e4a9853). My second commit, which this comes from, was done because Netkins complained I had gone above 5 indents and should refactor. So I extracted that whole block as its own private function. I would rather not include this second commit in this PR because it obfuscates the patch. But I wanted the green check mark in the list of PRs.

Ok. But now there is no need for this. Just use fast return without variable:

return factory.createAttribute(request, name, value);

Yes of course. Fixed this.

fenik17 · 2017-09-30T08:29:28Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

+        // See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html
+        if (HttpHeaderValues.FILENAME.contentEquals(name)) {
+            // filename value is quoted string so strip them
+            if (!shouldDecode) {


IIUC the RFC 5987 prescribe decoding not only filename header value? Otherwise we should not cut * from name for other headers..

I thought about that too. But I'm not sure that's the case (what other headers are there? they aren't being tested for flow control in this code). I decided to just cover this case and establish how it might be done if someone encountered this problem with other headers.

I mean, if we want apply new RFC only for filename header, we should not change the name attribute for other headers. But now you are doing this: name = name.substring(0, last);

Ah yes that makes sense. I think I avoid this now. Could add a test for it...

fenik17 · 2017-10-01T16:36:56Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

fenik17 · 2017-10-01T17:02:26Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

Would be better to reduce the possible allocations using filename* constant. Something like this:

private static final String FILENAME_ENCODED = HttpHeaderValues.FILENAME.toString() + '*'; private Attribute getContentDispositionAttribute(String... values) { String name = cleanString(values[0]); String value = values[1]; // See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html if (HttpHeaderValues.FILENAME.contentEquals(name)) { // filename value is quoted string so strip them value = value.substring(1, value.length() - 1); } else if (FILENAME_ENCODED.equals(name)) { // filename value is encoded. See https://tools.ietf.org/html/rfc5987 name = name.substring(0, name.length() - 1); String[] split = doubleSingleQuote.split(value, 2); value = QueryStringDecoder.decodeComponent(split[1], Charset.forName(split[0])); } else { // otherwise we need to clean the value value = cleanString(value); } return factory.createAttribute(request, name, value); }

Thanks. Pushed this version.

fenik17 · 2017-10-01T17:16:21Z

@dminkovsky RFC says that ext-value can optional contains language information:

foo: bar; title*=iso-8859-1'en'%A3%20rates

Maybe it makes sense to support for this too? At least, take this into account when parsing the value.

dminkovsky · 2017-10-01T17:48:46Z

Yes, I thought about that, but what would we do with it? Add a FileUpload attribute?

fenik17 · 2017-10-02T05:11:12Z

Yes, I thought about that, but what would we do with it?

Ignoring? Just split value through single quote "'".

String[] split = value.split("'", 3);
value = QueryStringDecoder.decodeComponent(split[2], Charset.forName(split[0]))

dminkovsky · 2017-10-02T11:12:59Z

I’m sorry that’s not what I meant. I meant: if we got the language code value—like you said by splitting a different way—then what would we do with that value? пн, 2 окт. 2017 г. в 1:12, Nikolay Fedorovskikh <notifications@github.com>:

…

Yes, I thought about that, but what would we do with it? Ignoring? Just split value through single quote "'". String[] split = value.split("'", 3); value = QueryStringDecoder.decodeComponent(split[2], Charset.forName(split[0])) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7265 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANWZe8g7ShFao2EoClhhRKBF-EGuxlNks5soHCvgaJpZM4PpUdL> .

fenik17 · 2017-10-02T16:06:11Z

I suggest not doing anything with the language field. Just skip this.
But the possibility of its presence compels us to split the header value through single quote into three parts.

normanmaurer

LGTM

carl-mastrangelo · 2017-10-02T18:21:47Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

+        // See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html
+        if (HttpHeaderValues.FILENAME.contentEquals(name)) {
+            // filename value is quoted string so strip them
+            value = value.substring(1, value.length() - 1);


consider checking that the first and last chars are actually quotes.

this was here before this PR. should i modify this behavior in this PR?

Adding this check too.

carl-mastrangelo · 2017-10-02T18:23:49Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

+            // filename value is encoded. See https://tools.ietf.org/html/rfc5987
+            name = name.substring(0, name.length() - 1);
+            String[] split = DOUBLE_SINGLE_QUOTE.split(value, 2);
+            value = QueryStringDecoder.decodeComponent(split[1], Charset.forName(split[0]));


Here too, you might consider check that the array size is 2 so that you don't throw an NPE.

Considered this, but if not an NPE, then what? There are other spots in the code around this PR that split and don't check.

carl-mastrangelo · 2017-10-02T18:24:13Z

codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java

+
+    // https://github.com/netty/netty/pull/7265
+    @Test
+    public void testDecodeContentDispositionFieldParameters() throws Exception {


add a negative case? perhaps a malformed header?

Yes, makes sense. Will do.

Added a few tests.

dminkovsky · 2017-10-05T22:50:12Z

@fenik17 just looked again at your previous comment and yes, of course, it's much better this way. thank you for your reviews!

carl-mastrangelo

One comment but otherwise LGTM

carl-mastrangelo · 2017-10-06T16:45:37Z

codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java

@normanmaurer do your tests require you to deref all buffers?

yep I think we should call req.release()

@dminkovsky please address this

@normanmaurer done. Please have a look. Thank you.

fenik17 · 2017-10-06T16:54:53Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

if (last > 0 && ...

Ah yes, thank you

value = token / quoted-string

https://tools.ietf.org/html/rfc5987#section-3.2.1

fenik17 · 2017-10-07T17:37:55Z

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java

We can avoid an extra allocation:

name = HttpHeaderValues.FILENAME.toString();

Yes, makes sense. Fixed. Thanks.

dminkovsky · 2017-10-13T01:02:12Z

By the way, could this make 4.1.17? I see only a 4.0 tag. I am wondering because I am on 4.1

normanmaurer · 2017-10-13T04:02:12Z

Everything that is merged into 4.0 will also be merged Into 4.1

dminkovsky · 2017-10-21T02:09:12Z

Good to hear. Thank you.

… parameters

normanmaurer · 2017-10-24T17:37:09Z

Cherry-picked into 4.1 (8aeba78) and 4.0 (82b7103).

@dminkovsky thanks!

fenik17 · 2017-10-30T12:29:48Z

@normanmaurer this is not pushed into 4.0?

normanmaurer · 2017-10-30T14:27:13Z

@fenik17 somehow I did not... Thanks for pinging. Cherry-picked as 82b7103

dminkovsky changed the title ~~add test~~ HttpPostMultipartRequestDecoder should decode header field parameters Sep 29, 2017

dminkovsky force-pushed the decode-field-parameters branch from 5153a3e to e4a9853 Compare September 29, 2017 23:39

dminkovsky force-pushed the decode-field-parameters branch from 69f1809 to 78977d5 Compare September 30, 2017 02:21

fenik17 reviewed Sep 30, 2017

View reviewed changes

fenik17 reviewed Oct 1, 2017

View reviewed changes

...ttp/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java Outdated

Copy link

Contributor

fenik17 Oct 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

final

fenik17 reviewed Oct 1, 2017

View reviewed changes

dminkovsky force-pushed the decode-field-parameters branch 2 times, most recently from 382ca6c to c24a4d4 Compare October 2, 2017 02:03

dminkovsky force-pushed the decode-field-parameters branch from c24a4d4 to 6555dd7 Compare October 2, 2017 16:11

normanmaurer approved these changes Oct 2, 2017

View reviewed changes

normanmaurer requested review from Scottmitch and carl-mastrangelo October 2, 2017 17:02

normanmaurer self-assigned this Oct 2, 2017

normanmaurer added the defect label Oct 2, 2017

normanmaurer added this to the 4.0.53.Final milestone Oct 2, 2017

carl-mastrangelo reviewed Oct 2, 2017

View reviewed changes

dminkovsky force-pushed the decode-field-parameters branch from 6555dd7 to 7742cbc Compare October 5, 2017 22:38

dminkovsky force-pushed the decode-field-parameters branch 2 times, most recently from 5bf0c4c to 5de5c86 Compare October 6, 2017 15:35

dminkovsky force-pushed the decode-field-parameters branch 2 times, most recently from 1354351 to 2b5dd05 Compare October 6, 2017 16:30

carl-mastrangelo approved these changes Oct 6, 2017

View reviewed changes

fenik17 reviewed Oct 6, 2017

View reviewed changes

dminkovsky force-pushed the decode-field-parameters branch from 2b5dd05 to 06dbf04 Compare October 7, 2017 16:20

fenik17 reviewed Oct 7, 2017

View reviewed changes

dminkovsky force-pushed the decode-field-parameters branch from 06dbf04 to 9f4b58a Compare October 21, 2017 02:08

fenik17 approved these changes Oct 23, 2017

View reviewed changes

netty#7265 HttpPostMultipartRequestDecoder should decode header field…

c485b39

… parameters

dminkovsky force-pushed the decode-field-parameters branch from 9f4b58a to c485b39 Compare October 24, 2017 14:09

normanmaurer closed this Oct 24, 2017

Uh oh!

HttpPostMultipartRequestDecoder should decode header field parameters #7265

HttpPostMultipartRequestDecoder should decode header field parameters #7265

Uh oh!

Conversation

dminkovsky commented Sep 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dminkovsky commented Sep 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dminkovsky Sep 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fenik17 Sep 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fenik17 commented Oct 1, 2017

Uh oh!

dminkovsky commented Oct 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fenik17 commented Oct 2, 2017

Uh oh!

dminkovsky commented Oct 2, 2017 via email

Uh oh!

fenik17 commented Oct 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

normanmaurer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dminkovsky commented Oct 5, 2017

Uh oh!

carl-mastrangelo left a comment

Choose a reason for hiding this comment

dminkovsky commented Sep 29, 2017 •

edited

Loading

dminkovsky commented Sep 30, 2017 •

edited

Loading

dminkovsky Sep 30, 2017 •

edited

Loading

fenik17 Sep 30, 2017 •

edited

Loading

dminkovsky commented Oct 1, 2017 •

edited

Loading

fenik17 commented Oct 2, 2017 •

edited

Loading

dminkovsky Oct 21, 2017 •

edited

Loading

normanmaurer commented Oct 24, 2017 •

edited

Loading