Add reactive multipart request support #1201

sdeleuze · 2016-10-12T09:24:06Z

This draft PR is a first Multipart support implementation here for high-level feedbacks and discussion.

It does not include RequestMapping level support, integration tests do not work yet with Reactor and RxNetty and NioMultipartResolver need to be rewritten using Flux.create() instead of ReplayProcessor.

sdeleuze · 2016-10-13T16:23:58Z

I have updated this PR witgh the following changes:

Tests now run fine with Reactor Netty and RxNetty
getAllParts() now caches the resulting mono to allow multiple invocations
There is a single Part object suitable usable for file and non file parts
We now use Flux.create() instead of ReplayProcessor
NioPart#transferTo() now uses zero copy when possible (when a temporary file has been previously created)

@rstoyanchev Could you please make a high-level review of this commit?

rstoyanchev · 2016-10-14T20:19:59Z

@sdeleuze, apologies for suggesting it but ServerHttpRequest doesn't seem like the right level for exposing multipart support. Having MultipartResolver take ServerHttpRequest as input is already an indication that it's a higher level concern. Then MultipartResolver has to be injected into every request implementation, which is rather repetitive and artificial since it doesn't benefit from being in every request implementation. It's also a cyclical dependency even at the package level since the http package is logically at a level below the web package and should not depend on anything from it.

As it looks currently it's a better fit exposing it in ServerWebExchange. To avoid a circular dependency there once again, MultipartResolver and Part should be in the server package with maybe a server.multipart sub-package for implementations. For initialization MultipartResolver could be a field of HttpWebHandlerAdapter and then passed into ServerWebExchange instances. Very similar to WebSessionManager and WebSession.

For the caching, the cache operator should really be called at construction time or otherwise concurrent calls to getParts can each create a separate pipeline. See how DefaultWebExchange does it for WebSession and also note that DefaultWebSession uses the defer operator to ensure it doesn't do anything at the time of declaring the pipeline.

When the request is not multipart could we be more lenient and return an empty stream from getParts and a Mono with empty map from getAllParts perhaps? It flips the model from having to call isMultipartRequest first to being able to call getParts and getAllParts without error or side effect and optionally checking isMultipartRequest first if you have to.

Do we need Part.getContentType if we have getHeaders? Just thinking there are plenty of methods on part already?

I'm still not entirely sure how a JSON multipart would work out? Would it be a call to getValue() or getContent()? The underlying library seems to split the callbacks that way, but I'm not sure what it would do if there is no filename but there is a non-text/plain content-type.

I haven't looked into the implementation library but I presume it is non-blocking indeed? It seems to be returning java.io.InputStream.

sdeleuze · 2016-10-18T13:49:07Z

No worry indeed, I agree that ServerHttpRequest is not the right level to implement that for the reasons you mentioned. I also agree with your remarks on caching and being more lenient if the request is not multipart.

We can remove Part.getContentType() since this is just a shortcut to Part.getHeaders().getContentType() (with maybe a Javadoc hint to help users for file parts).

Currently I think JSON parts will be natively treated by the underlying library like a part with a String based value since it has no filename header. Even if in NioPart I allowed to get the content both as a String or as a Publisher<DataBuffer>, that could be not optimal.

Not sure if that makes sens, but should we add some HttpMessageReader based capabilities with some <T> Flux<T> getContentAsStream(Class<T> type) and <T> Mono<T> getContentAs(Class<T> type) methods available on Part ? Or should we delegate that part to a composite MultipartHttpMessageReader that could be initialized with a List<HttpMessageReader> like we do on the writing side for SSE?

About the NIO Multipart library, it seems to me that the request parsing is done in a non blocking way but sadly we can only access to data as String or InputStream depending on the part type, so it may be blocking I suppose. I have implemented an optimisation to be able to perform a zero-copy transfer if the InputStream is a FileInputStream.

I can take care of the changes discussed here if you are ok.

I have created an issue on their GitHub to ask them more insight about the blocking behavior implied by InputStream and that fact that non-file part content is only accessible as a String.

rstoyanchev · 2016-10-18T20:20:24Z

Okay, I'm not too sure what you mean with the HttpMessageReader based capabilities. I would think that Part should only expose basic ways of getting to the data, but once you've updated the PR maybe it will be more clear.

sdeleuze · 2016-10-19T09:19:57Z

I have pushed an updated PR, and asked a new question to NIO Multipart project lead about how to perform non-blocking processing with their current InputStream / OutputStream based API.

sdeleuze · 2016-10-25T21:48:54Z

@rstoyanchev Could you please review the last version of this PR?

rstoyanchev

Having ServerWebExchange to expose multipart methods and delegate to MultipartResolver internally looks alright. In fact the MultipartResolver seems to belong in the http package. It's like an HttpMessageReader and could even be used with an HttpMessageReader to further parse the content of a part. I suggest moving web.multipart.reactive to http.reactive right next to the codec package.

rstoyanchev · 2016-10-27T15:43:45Z

spring-web/src/main/java/org/springframework/web/multipart/reactive/Part.java

+	/**
+	 * @return the content of the part as a String
+	 */
+	Mono<String> getValue();


The Javadoc should mention the algorithm to determine the charset and what if there is no indication, is there a natural default or does it error out?

Also as this is a shortcut method it would make sense to call it getContentAsString(). Technically the method could be even left out but It helps to support form field values.

rstoyanchev · 2016-10-27T15:59:15Z

...g-web/src/main/java/org/springframework/web/multipart/reactive/nio/NioMultipartResolver.java

+					MultiValueMap<String, Part> partsMap = new LinkedMultiValueMap<>();
+					parts.forEach(part -> partsMap.add(part.getName(), part));
+					return partsMap;
+				});


What about using collectMap instead of collectList + map?

Reactor collectMultimap() is using a Collection for the value type while LinkedMultiValueMap is taking a Map<K, List<V>> parameter, that's why I did not use it in my first implementation, but after more thoughts I was able to use it in my updated commit thanks to Collectors.toMap() to change the value type of the map.

rstoyanchev · 2016-10-27T16:11:02Z

...g-web/src/main/java/org/springframework/web/multipart/reactive/nio/NioMultipartResolver.java

+				}
+				catch (IOException ex) {
+					throw Exceptions.bubble(e);
+				}


Should this notify the parser about the error which calls error on the emitter, so that subscribers for the parts will get the error?

I made some changes to the error handling in order to notify the parser about errors and ensuring the only 1 error is emitted, but I am not 100% sure this is what you meant, so this is something we should discuss together.

rstoyanchev · 2016-10-27T16:12:49Z

...g-web/src/main/java/org/springframework/web/multipart/reactive/nio/NioMultipartResolver.java

+				public void onError(String message, Throwable cause) {
+					emitter.error(new MultipartException(message, cause));
+				}
+			});


An inner class perhaps given the number of methods?

rstoyanchev · 2016-10-27T16:43:02Z

spring-web/src/main/java/org/springframework/web/server/ServerWebExchange.java

+	 * or a {@link Mono} emitting an error the current request is not a multipart one.
+	 */
+	Mono<MultiValueMap<String, Part>> getAllParts();
+


These could be ordered right below getPrincipal() above with getAllParts ahead of getParts as the more common case. Same for decorator and default implementations.

I changed the order and also proposed new names in order to be more consistent with getFormData(): Mono<MultiValueMap<String, Part>> getFormParts() and Flux<Part> getFormPartsAsFlux(). Are you ok with these new names?

rstoyanchev · 2016-10-27T16:51:38Z

...ng-web/src/test/java/org/springframework/http/server/reactive/MultipartIntegrationTests.java

+						else {
+							fail();
+						}
+					})


Could exchange.getParts() with collectToMap be used to get all parts first and then invoke a method that asserts all parts, which can also be called from assertGetAllParts?

sdeleuze · 2016-10-30T16:31:58Z

@rstoyanchev I took in account your feedbacks and:

Changed the package to org.springframework.http.codec.multipart
Polished the Javadoc
Renamed getAllParts() to getFormParts()
Renamed getParts() to getFormPartsAsFlux()
Integrated the MultipartHttpMessageWriter commit

This commit introduces reactive multipart support by adding a new MultipartHttpMessageReader interface (with default methods) and a SynchronossMultipartHttpMessageReader implementation based on the Synchronoss NIO Multipart implementation (https://github.com/synchronoss/nio-multipart). Issue: SPR-14546

This commit adds a reactive HttpMessageWriter that allows to write multipart HTML forms with multipart/form-data media type. Issue: SPR-14546

Issue: SPR-14546

With this commit, ServerCodecConfigurer is now exposed as a bean in order to be provided to DefaultServerWebExchange via WebHttpHandlerBuilder and HttpWebHandlerAdapter. This allows DefaultServerWebExchange to get configured codecs for reading form or multipart requests. Issue: SPR-14546

Issue: SPR-14546

rstoyanchev · 2017-05-01T18:38:14Z

This is now in master.

sdeleuze force-pushed the SPR-14546 branch 3 times, most recently from 21e701a to a15cc85 Compare October 13, 2016 16:17

rstoyanchev self-assigned this Oct 14, 2016

sdeleuze mentioned this pull request Oct 18, 2016

Question about how to read part contents in a non-blocking way synchronoss/nio-multipart#4

Open

sdeleuze self-assigned this Oct 19, 2016

sdeleuze force-pushed the SPR-14546 branch from a15cc85 to 532a78f Compare October 19, 2016 09:15

sdeleuze force-pushed the SPR-14546 branch 4 times, most recently from e484c44 to c7fb77d Compare October 25, 2016 21:45

sdeleuze changed the title ~~Add reactive multipart request support (WIP)~~ Add reactive multipart request support Oct 25, 2016

sdeleuze force-pushed the SPR-14546 branch from c7fb77d to 6a38b53 Compare October 25, 2016 21:46

sdeleuze mentioned this pull request Oct 25, 2016

Add FormHttpMessageReader/Writer #1223

Closed

rstoyanchev requested changes Oct 27, 2016

View reviewed changes

sdeleuze force-pushed the SPR-14546 branch 9 times, most recently from 00dddba to d3a23ce Compare October 30, 2016 11:52

sdeleuze force-pushed the SPR-14546 branch 3 times, most recently from 9e37c87 to c4a8288 Compare January 17, 2017 16:51

sdeleuze force-pushed the SPR-14546 branch from c4a8288 to ac9ad50 Compare January 31, 2017 14:19

sdeleuze force-pushed the SPR-14546 branch from ac9ad50 to fa0abb4 Compare February 13, 2017 13:48

sdeleuze force-pushed the SPR-14546 branch 12 times, most recently from 1ee6336 to 7cc1db1 Compare April 28, 2017 14:42

sdeleuze added 5 commits April 28, 2017 16:42

Add MultipartHttpMessageWriter

852dc84

This commit adds a reactive HttpMessageWriter that allows to write multipart HTML forms with multipart/form-data media type. Issue: SPR-14546

Add multipart support to WebFlux functional API

a712c19

Issue: SPR-14546

Add multipart support to ServerWebExchange

4bfd04b

Issue: SPR-14546

sdeleuze force-pushed the SPR-14546 branch from 7cc1db1 to f140d29 Compare April 28, 2017 14:42

Add @request(Param/Part) support for multipart requests

b804ba1

Issue: SPR-14546

sdeleuze force-pushed the SPR-14546 branch from f140d29 to b804ba1 Compare April 28, 2017 14:53

rstoyanchev closed this May 1, 2017

spring-projects-issues mentioned this pull request Jan 11, 2019

Reactive multipart request support [SPR-14546] #19114

Closed

sdeleuze deleted the SPR-14546 branch April 28, 2021 09:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add reactive multipart request support #1201

Add reactive multipart request support #1201

sdeleuze commented Oct 12, 2016 •

edited

sdeleuze commented Oct 13, 2016 •

edited

rstoyanchev commented Oct 14, 2016 •

edited

sdeleuze commented Oct 18, 2016 •

edited

rstoyanchev commented Oct 18, 2016

sdeleuze commented Oct 19, 2016

sdeleuze commented Oct 25, 2016

rstoyanchev left a comment

rstoyanchev Oct 27, 2016

rstoyanchev Oct 27, 2016

sdeleuze Oct 30, 2016

rstoyanchev Oct 27, 2016

sdeleuze Oct 30, 2016

rstoyanchev Oct 27, 2016

rstoyanchev Oct 27, 2016

sdeleuze Oct 30, 2016

rstoyanchev Oct 27, 2016

sdeleuze commented Oct 30, 2016

rstoyanchev commented May 1, 2017

Add reactive multipart request support #1201

Add reactive multipart request support #1201

Conversation

sdeleuze commented Oct 12, 2016 • edited

sdeleuze commented Oct 13, 2016 • edited

rstoyanchev commented Oct 14, 2016 • edited

sdeleuze commented Oct 18, 2016 • edited

rstoyanchev commented Oct 18, 2016

sdeleuze commented Oct 19, 2016

sdeleuze commented Oct 25, 2016

rstoyanchev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdeleuze commented Oct 30, 2016

rstoyanchev commented May 1, 2017

sdeleuze commented Oct 12, 2016 •

edited

sdeleuze commented Oct 13, 2016 •

edited

rstoyanchev commented Oct 14, 2016 •

edited

sdeleuze commented Oct 18, 2016 •

edited