Support for appending arbitrary query string parameters to outgoing requests in HTTP outbound gateway/adapter [INT-4169]

[spring-operator]

Mauro Molinari opened INT-4169 and commented

I need to invoke a REST service which is mapped on the remote side with Spring Web MVC to a method with a given number of parameters (an Integer, a String and a String[]).

If this service is mapped against POST method, I can configure my <int-http:outbound-gateway> and send to it a message payload of type MultiValueMap so that it sends out my parameters with content type application/x-www-form-urlencoded or multipart/form-data (depending on the actual content). So far so good.

But if this service is mapped against GET method, it's much harder to invoke it with Spring Integration. I would need to add all the URI variables to the URI and then use, for instance, the uri-variables-expression to get a Map of those variables values. However, unless URI variables are few and known in advance, this approach is not flexible. In my specific case, I think it's even useless, because to send a multi-value parameter I would need to repeat the query-string parameter in the URL n times, being n not known in advance.

I mean, if I need to call:

http://www.example.com/getSomething?cid={cid}&foo={foo}&paramList={value1}&paramList={value2}&paramList={value3}

but I don't know how many values are in paramList, I can't write a proper URI template in my <int-http:outbound-gateway> definition and I hardly think that a Map containing an Iterable value for paramList parameter will be "expanded" to such an actual URL (honestly, I haven't tried).

I think an elegant solution to this problem could be an attribute like append-query-string-parameters-expression which must evaluate to a MultiValueMap (or just Map for simple scenarios) containing all the key-value pairs (or key-multiple value pairs) to be dynamically appended to the URL before invocation. something like:

<int-http:outbound-gateway
  id="myHttpGateway" request-channel="inputChannel"
  reply-channel="outputChannel" url="http://www.example.com/getSomething"
  http-method="GET" charset="UTF-8" append-query-string-parameters-expression="payload"
  expected-response-type="com.example.MyResponseBean" />

In this example, if payload is a MultiValueMap, it will be used to compute all the query string parameters to be appended to the URL to make the actual service invocation.

I know that passing parameters through arbitrary query string parameters may be risky (because you may have lose control con the final length of the URL), but if this is in some way controlled, I find it a reasonable alternative to a POST + application/x-www-form-urlencoded content type. After all, this is what a browser does when posting a FORM with GET method.

---

Affects: 4.2.11

Referenced from: pull request #2289

[spring-operator]

Artem Bilan commented

Mauro Molinari,

Thank you for the feedback and such a comprehensive research and description.

Have you noticed that there is something like:

<xsd:attribute name="url-expression" type="xsd:string" use="optional">
	<xsd:annotation>
		<xsd:documentation>
			<![CDATA[
	SpEL Expression resolving to a URL to which the requests should be sent. The resolved
	value may include {placeholders} for further evaluation against uri-variables.
			]]></xsd:documentation>
	</xsd:annotation>
</xsd:attribute>

As for me it looks like this one existing can cover your append-uri-variables-expression as well.

You can develop any custom logic based on the requestMessage and desired URL pattern.

Unfortunately I'm not able to find some out-of-the-box utility to expand URI template with your logic, but I think that won't be so difficult to iterate the map and concatenate strings.

To be honest I don't see reason to introduce such an option just for case when payload is the Map.
And that url-expression has been introduced especially for exotic cases like yours.

Does it make sense for you?

[spring-operator]

Mauro Molinari commented

Hi Artem, this is certainly the way to go in my case for now, but it's clearly some work to do (I have to add a bean that performs the necessary task, possibly by using UriComponentBuilder) to handle a case which I think should be quite common in practice (although I do not have many years of experience in REST). That is, some verbose "noise" to add to my configuration for something that I would have expected Spring Integration to handle elegantly as it usually does :)

In my example, payload is the map, but the map may come from a payload property, from a header value, or from a subset of the message headers (just like you map message headers to request headers). Also, if the map is simply held by the payload, switching from GET to POST (or viceversa) would be simply a matter of changing the http-method attribute value and removing/adding the suggested new attribute (which btw I have changed in my original report with a better name: append-query-string-parameters-expression).

Is my use case really so exotic? Are all REST requests in real-world scenarios actually done in POST whenever many/dynamic parameters must be passed to the remote host?

(P.S.: strange, I haven't received any mail notification about your added reply...)

[spring-operator]

Artem Bilan commented

Hello, Mauro!

Not sure what is the problem with notification, but that happens sometimes even for us.
Like an out-of-date SSL certificate on the mail server :).

Well, not sure how it is a common task, but looks like there is no direct simple solution to build a query string manually and people use different way to achieve the result: http://stackoverflow.com/questions/2809877/how-to-convert-map-to-url-query-string.

Yes, let's call things with their names!

What you want, in Servlet Specification is called like queryString:

/**
  * Returns the query string that is contained in the request
  * URL after the path. This method returns <code>null</code>
  * if the URL does not have a query string. Same as the value
  * of the CGI variable QUERY_STRING. 
  *
  * @return		a <code>String</code> containing the query
  *			string or <code>null</code> if the URL 
  *			contains no query string. The value is not
  *			decoded by the container.
  */
    public String getQueryString();

Or just query if we take a look into java.net.URI:

Now on the matter. So, looks like your case we still can achieve with the url-expression:

url-expression="T(org.springframework.web.util.UriComponentsBuilder).fromHttpUrl('http://HOST:PORT/PATH').queryParams(payload).build().toUri()"

Or you still can do that via uri-variable:

<outbound-gateway id="proxyGateway" request-channel="testChannel"
				  url-expression="'http://testServer/test?{queryString}'">
       <uri-variable name="queryString" expression="'a=A&amp;b=B'"/>
</outbound-gateway>

Or use mentioned above URLEncodedUtils.fromat() to convert some List<NameValuePair>
(I still try to figure out the simple way to convert MultiValueMap to that List<NameValuePair>... :))

I understand your wish to make it as simple as possible, but at the same time there is no guaranty which way would prefer end-user to build his/her query string in the end.

So, maybe we can just end up with something like Documentation how you can build query string on your own?

[spring-operator]

Artem Bilan commented

Mauro,

The Java Stream to build List<NameValuePair> from MultiValueMap<String, String>:

List<NameValuePair> nameValuePairs =
			params.entrySet()
					.stream()
					.flatMap(e -> e.getValue()
							.stream()
							.map(v -> new BasicNameValuePair(e.getKey(), v))
					)
					.collect(Collectors.toList());

That is for the case when URLEncodedUtils.fromat() is makes sense.

[spring-operator]

Mauro Molinari commented

Hi Artem, no mail notification again :-)

org.springframework.web.util.UriComponentsBuilder.queryParams(MultiValueMap<String, String>) is exactly what I had in mind (I missed it, it was not present in Spring 3.x API). Honestly, I don't know what URLEncodedUtils and NameValuePair are (it seems I don't have them in my classpath, apart from a NameValuePair in JDK for CORBA), but MultiValueMap is what RestTemplate uses by default to send application/x-www-form-urlencoded or multipart/form-data with the default registered message converters for POST method, so it is the natural choice for me.

Indeed the fact that you can perform the actual job with a single expression thanks to the above method of UriComponentsBuilder is nice and provides a good compromise. Thanks for pointing it out!

Indeed, mentioning it in the documentation could be a useful hint for whoever has to deal with non-trivial GET requests and parameters passing!

[spring-operator]

Mauro Molinari commented

By the way, I prefer url-expression to <uri-variable> for this use case, because when you write

http://testServer/test?{queryString}

in the URL you're already making some assumptions on the URL shape, especially if you want to also handle some fixed and known-in-advance parameters in addition to the dynamic ones, such as in case:

http://testServer/test?fixedParam={foo}&{queryString}

It also requires you to be careful about param value encoding. The UriComponentsBuilder in the url-expression attribute value, instead, is a fully modular and safe solution. But that's just my taste.

[spring-operator]

Artem Bilan commented

OK, Mauro!

Than it looks like a deal :).

You get a solution and we have a middle ground as a Documentation improvement.
No code change as I'd prefer. Just because <int-http:outbound-gateway> is complex enough already.
And we can achieve the solution with built-in option in face of url-expression.

I will have a discussion with our Spring Web team to consider some utility method to convert MultiValueMap to the queryString in easy way.

The URLEncodedUtils is in the Apache HTTP Client.