Provide a post() method on WS for POSTing multipart/form-data #902

Closed
studiodev opened this Issue Mar 25, 2013 · 32 comments

Projects

None yet
@studiodev
Contributor

There is actually no way to post a multipart/form-data, without encoding manually the body (and this is tricky!)

It's sad to not have a WS method which allow to call a controller like the "File upload example"

@ryanoglesby08

This functionality would be very helpful. Agreed.

A method as part of FakeRequest that is something along the lines of "fakeRequest().withMultipartFormData(formData);" would help immensely with testing controllers using multipart form data.

@jroper jroper added a commit to jroper/playframework that referenced this issue Apr 15, 2013
@jroper jroper [#902] Fixed bugs in enumerator streaming fbcd1af
@joost-de-vries

Another strong vote for this feature. I've closed the other issue #1302 in favour of this one.

@ewilson
ewilson commented Jul 9, 2014

Please consider implementing this.

@ornicar
Contributor
ornicar commented Jul 31, 2014

๐Ÿ‘

@AlexGalays

๐Ÿ‘

@cxvvs
cxvvs commented Jul 31, 2014

In the meantime, here is a workaround that was greatly inspired by some code in Play

import com.ning.http.client.{Response => AHCResponse, _}
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{Future, Promise}
import play.api.Play.current

  object WSUtil {

    def post(url: String, bodyParts: List[Part]): Future[Response] = {
      val client = WS.client(current).underlying.asInstanceOf[AsyncHttpClient]

      val builder = client.preparePost(url)

      builder.setHeader("Content-Type", "multipart/form-data")
      bodyParts.foreach(builder.addBodyPart)

      var result = Promise[NingWSResponse]()

      client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
        override def onCompleted(response: AHCResponse) = {
          result.success(NingWSResponse(response))
          response
        }

        override def onThrowable(t: Throwable) = {
          result.failure(t)
        }
      })

      result.future
    }
@toggm
toggm commented Oct 8, 2014

Based on WSClient and WSRequestHolder I implemented a multipart based request.

package helper

import com.ning.http.client.{ Response => AHCResponse, Part => AHCPart, StringPart => AHCStringPart, FilePart => AHCFilePart, _ }
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{ Future, Promise }
import play.api.Play.current
import play.api.libs.ws.WS
import com.ning.http.client.AsyncHttpClient._
import play.api.libs.iteratee.Enumerator
import collection.JavaConversions._
import java.io.File
import com.ning.http.multipart.{ FilePart => MPFilePart }

sealed trait Part {
  def toAHCPart: AHCPart
}
case class StringPart(key: String, value: String) extends Part {
  def toAHCPart = {
    new AHCStringPart(key, value)
  }
}
case class FilePart(key: String, file: File, contentType: String, charset: String) extends Part {
  def toAHCPart = {
    new MPFilePart(key, file, contentType, charset)
  }
}

case class MultiPartWSRequestHolder(client: AsyncHttpClient, builder: AsyncHttpClient#BoundRequestBuilder, url: String) extends WSRequestHolder {

  def withPart(part: Part): MultiPartWSRequestHolder = {
    builder.addBodyPart(part.toAHCPart)
    this
  }

  def withParts(parts: Part*): MultiPartWSRequestHolder = {
    parts.forall(p => builder.addBodyPart(p.toAHCPart) != null)
    this
  }

  /**
   * The method for this request
   */
  val method: String = ""

  /**
   * The body of this request
   */
  val body: WSBody = EmptyBody

  /**
   * 7
   * The headers for this request
   */
  val headers: Map[String, Seq[String]] = Map()

  /**
   * The query string for this request
   */
  val queryString: Map[String, Seq[String]] = Map()

  /**
   * A calculator of the signature for this request
   */
  val calc: Option[WSSignatureCalculator] = None

  /**
   * The authentication this request should use
   */
  val auth: Option[(String, String, WSAuthScheme)] = None

  /**
   * Whether this request should follow redirects
   */
  val followRedirects: Option[Boolean] = None

  /**
   * The timeout for the request
   */
  val requestTimeout: Option[Int] = None

  /**
   * The virtual host this request will use
   */
  val virtualHost: Option[String] = None

  /**
   * The proxy server this request will use
   */
  val proxyServer: Option[WSProxyServer] = None

  /**
   * sets the signature calculator for the request
   * @param calc
   */
  def sign(calc: WSSignatureCalculator): MultiPartWSRequestHolder = {
    this
  }

  /**
   * sets the authentication realm
   */
  def withAuth(username: String, password: String, scheme: WSAuthScheme): MultiPartWSRequestHolder = {
    //not yet supported
    this
  }

  /**
   * adds any number of HTTP headers
   * @param hdrs
   */
  def withHeaders(hdrs: (String, String)*): MultiPartWSRequestHolder = {
    hdrs.forall(h => builder.setHeader(h._1, h._2) != null)
    this
  }

  def withMethod(method: String): MultiPartWSRequestHolder = {
    builder.setMethod(method)
    this
  }

  /**
   * Sets the maximum time in milliseconds you expect the request to take.
   * Warning: a stream consumption will be interrupted when this time is reached.
   */
  def withRequestTimeout(timeout: Int): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the virtual host to use in this request
   */
  def withVirtualHost(vh: String): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the proxy server to use in this request
   */
  def withProxyServer(proxyServer: WSProxyServer): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the body for this request
   */
  def withBody(body: WSBody): MultiPartWSRequestHolder = {
    this
  }

  /**
   * adds any number of query string parameters to the
   */
  def withQueryString(parameters: (String, String)*): MultiPartWSRequestHolder = {
    val map = parameters.groupBy(_._1).map { case (k, v) => (k, v.map(_._2)) }
    val builderMap = new FluentStringsMap()
    map.forall { case (key, values) => builderMap.add(key, values) != null }

    builder.setQueryParameters(builderMap)

    this
  }

  /**
   * Sets whether redirects (301, 302) should be followed automatically
   */
  def withFollowRedirects(follow: Boolean): MultiPartWSRequestHolder = {
    builder.setFollowRedirects(follow)
    this
  }

  /**
   * Execute this request
   */
  def execute(): Future[WSResponse] = {
    var result = Promise[WSResponse]()

    client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
      override def onCompleted(response: AHCResponse) = {
        result.success(NingWSResponse(response))
        response
      }

      override def onThrowable(t: Throwable) = {
        result.failure(t)
        ()
      }
    })

    result.future
  }

  /**
   * Execute this request and stream the response body in an enumerator
   */
  def stream(): Future[(WSResponseHeaders, Enumerator[Array[Byte]])] = {
    Future.failed(new RuntimeException("Not supported"))
  }
}

case class MultPartWSClient(wsClient: WSClient) {
  val client = wsClient.underlying.asInstanceOf[AsyncHttpClient]

  def url(url: String): MultiPartWSRequestHolder = {
    val builder = client.preparePost(url)
    builder.setHeader("Content-Type", "multipart/form-data")
    MultiPartWSRequestHolder(client, builder, url)
  }
}

It can be used i..e:

val multiPartClient = MultPartWSClient(client)
    multiPartClient.url(url)
      .withParts(StringPart("title", title), 
          StringPart("someAttr1", attr1),
          StringPart("someAttr2", attr2),
          StringPart("someAttr3", attr3),
          FilePart("document", document, contentType, "UTF-8"))
      .withMethod("POST")
      .execute()

It is still not fully implemented, but can be of use to anyone. Feel free to integrate that into play or use it in your own project.

@nafg
Contributor
nafg commented Dec 5, 2014

Simpler solution, based on http://stackoverflow.com/a/18723326/333643:

(I'm calling the mailgun API)

// Use the Ning AsyncHttpClient multipart class to get the bytes
val parts = Array[Part](
      new StringPart("from", from),
      new StringPart("to", to),
      new StringPart("subject", subject),
      new StringPart("text", text),
      new FilePart("attachment", file)
    )
    val mpre = new MultipartRequestEntity(parts, new FluentCaseInsensitiveStringsMap)
    val baos = new ByteArrayOutputStream
    mpre.writeRequest(baos)
    val bytes = baos.toByteArray
    val contentType = mpre.getContentType

// Now just send the data to the WS API
    client.url(endpoint)
      .post(bytes)(Writeable.wBytes, ContentTypeOf(Some(contentType)))

I would have liked to be able to create implicit Writeable and ContentTypeOf instances for MultipartRequestEntity. Unfortunately however, their API is based on the assumption of one content type per typeclass instance (i.e. it would have to be the same for all MultipartRequestEntity instances).

However since it has to include the boundary value, this assumption is broken. If the typeclass could wrap an A => Option[String] rather than just an Option[String], we could just have an implicit Writeable[MultipartRequstEntity] etc.

@naderghanbari
Contributor

Is there any plan to fix this? this will simplify things to a great extent (without this supported out of the box, sending a multipart request is very tedious).

@jroper jroper added the community label Jun 3, 2015
@jroper jroper closed this Jun 3, 2015
@jroper jroper reopened this Jun 3, 2015
@cdmckay
cdmckay commented Jun 25, 2015

I wrote a Writeable to use the existing MultipartFormData object for POSTing multipart form data:

https://gist.github.com/cdmckay/4b269e9017a30111556a

And here's an example of usage:

https://gist.github.com/cdmckay/5f05cbcb5327259df1cf

So you just need to drop it in and it should work with the existing WS flow.

@brianwawok

@cdmckay @nafg @toggm do you have a version that works in play 2.4? I had my own home rolled attempt, but so far all my multipart form posts using play get back a "The request body is too large" from my API I am hitting..

@david-bouyssie

@cdmckay : is there a solution to observe the progression of the upload ?
Currently I'm using scalaj-http for this purpose (https://github.com/scalaj/scalaj-http) but I would like to switch to Play WS in order to avoid extra dependencies and use asynchronous requests.
Thanks !

@ergomesh

@brianwawok

I have a similar problem I am migrating from 2.3 to 2.4 and get this now

Compilation error[object multipart is not a member of package com.ning.http] using the@cdmckay example. Anyone have this working in play2.4.x ?

@brianwawok

@ergomesh yes I was able to make this work in 2.4. I had to forcibly override my async http client version to 1.9.29, and then I did:

https://gist.github.com/brianwawok/eb9ac542eb7b4795ac06

A few things like

"Content-Type" -> mpre.getContentType)

were key, because I needed to add back in the boundary (which was lost in the play upgrade)

@mkurz
Member
mkurz commented Jan 20, 2016

Could someone provide a pull request? This would be great!

@brianwawok

@mkurz i have it in a hack state with no tests, not sure I am going to have time to clean up into a proper PR

@mkurz
Member
mkurz commented Jan 20, 2016

@brianwawok If you could find some time to submit that PR that would be really awesome and would make some people really happy! ๐Ÿ˜„

@mkurz
Member
mkurz commented Jan 21, 2016

@brianwawok If you do some work on this and provide a PR I could maybe help out testing things.

@jakob85
jakob85 commented Feb 11, 2016

Not sure if you guys fixed the issue but I answered my own question at stackoverflow and got it working thanks to all posts here =)
http://stackoverflow.com/questions/35325942/timeout-when-sending-multipart-form-data-in-play2-2-4-6

@mkurz
Member
mkurz commented Feb 11, 2016

@jakob85 If you are keen to provide a pull request with implements this feature directly in Play you are welcome to do so.

@david-bouyssie

A solution including the monitoring of upload progression would be really awesome.
This feature is provided by this library: https://github.com/scalaj/scalaj-http

@schmitch
Member

Actually all the solutions here would be missing the ending, i.e:
Something like that:
--BRY7v0J3h3fIl91lUeQ6SxMJWqNHbRYi4Exp2C3s--

What would actually work is using the asyncHttpClient.preparePost().addBodyPart funtionallity i.e.:

val asyncHttpClient: AsyncHttpClient = ws.client.underlying

        val postBuilder = asyncHttpClient.preparePost(url).addBodyPart(filePart).addBodyPart(textPart)
          .addHeader(AUTHORIZATION, s"Basic $auth")
        val request = postBuilder.build()

        val result = Promise[NingWSResponse]()
        asyncHttpClient.executeRequest(request, new AsyncCompletionHandler[AHCResponse]() {
          override def onCompleted(response: AHCResponse) = {
            result.success(NingWSResponse(response))
            response
          }

          override def onThrowable(t: Throwable) = {
            result.failure(t)
          }
        })
        result.future

Not sure if a Writeable has access to that. I'm looking forward to maybe add a PR.

@schmitch
Member

Actually it would be great if somebody could help me with the PR on tests.

Actually I've left out the documentation since I'm not sure if it's good to either have something like .post(Seq(part1, part2)) and on the Java part a List. Actually maybe somebody from the core team could give me some hints.

Edit: Too bad this prolly won't go into 2.5. Actually I came too late across this.

@mkurz
Member
mkurz commented Feb 19, 2016

Maybe we can can still get this into 2.5?

@gmethvin
Member

@mkurz At this point there should be no problem getting it into 2.5. It would only be an issue if there are major API changes.

@mkurz
Member
mkurz commented Feb 19, 2016

@gmethvin Cool! ๐Ÿ˜„

@schmitch
Member

That would be great. I actually came across this since we are sending data to another provider which uses insecure SSL, so using the underlying AhcClient takes more work than just fixing this issue..

@david-bouyssie

Could it be possible to have access to onContentWriteProgress callback or is it too complicated to interface it with the Play API ?
https://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncCompletionHandler.html#onContentWriteProgress%28long,%20long,%20long%29

This would be a very useful feature !

@gmethvin gmethvin closed this in b9b4afe Mar 15, 2016
@david-bouyssie

Do you have any tip to monitor the upload progress ?
Do I have to use the underlying client to achieve that ?
I was thinking about using reactive streams to be informed periodically about the upload progress update.
Thanks !

@mkurz mkurz added a commit to mkurz/playframework that referenced this issue Mar 16, 2016
@schmitch @mkurz schmitch + mkurz Fixes #902 added multipart/form-data requests to play-ws 2c8d844
@AndreyIlin

Is there any workaround for multipart/form-data for play 2.5?

@schmitch
Member

it will when 2.5.1 hit @AndreyIlin

@AndreyIlin

Thanks @schmitch, waiting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment