Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a post() method on WS for POSTing multipart/form-data #902

Closed
julien-lafont opened this issue Mar 25, 2013 · 32 comments
Closed

Provide a post() method on WS for POSTing multipart/form-data #902

julien-lafont opened this issue Mar 25, 2013 · 32 comments

Comments

@julien-lafont
Copy link

@julien-lafont julien-lafont commented Mar 25, 2013

There is actually no way to post a multipart/form-data, without encoding manually the body (and this is tricky!)

It's sad to not have a WS method which allow to call a controller like the "File upload example"

@ryanoglesby08
Copy link

@ryanoglesby08 ryanoglesby08 commented Apr 3, 2013

This functionality would be very helpful. Agreed.

A method as part of FakeRequest that is something along the lines of "fakeRequest().withMultipartFormData(formData);" would help immensely with testing controllers using multipart form data.

@joost-de-vries
Copy link

@joost-de-vries joost-de-vries commented Jul 6, 2013

Another strong vote for this feature. I've closed the other issue #1302 in favour of this one.

@ewilson
Copy link

@ewilson ewilson commented Jul 9, 2014

Please consider implementing this.

@ornicar
Copy link
Contributor

@ornicar ornicar commented Jul 31, 2014

👍

1 similar comment
@AlexGalays
Copy link

@AlexGalays AlexGalays commented Jul 31, 2014

👍

@cxvvs
Copy link

@cxvvs cxvvs commented Jul 31, 2014

In the meantime, here is a workaround that was greatly inspired by some code in Play

import com.ning.http.client.{Response => AHCResponse, _}
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{Future, Promise}
import play.api.Play.current

  object WSUtil {

    def post(url: String, bodyParts: List[Part]): Future[Response] = {
      val client = WS.client(current).underlying.asInstanceOf[AsyncHttpClient]

      val builder = client.preparePost(url)

      builder.setHeader("Content-Type", "multipart/form-data")
      bodyParts.foreach(builder.addBodyPart)

      var result = Promise[NingWSResponse]()

      client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
        override def onCompleted(response: AHCResponse) = {
          result.success(NingWSResponse(response))
          response
        }

        override def onThrowable(t: Throwable) = {
          result.failure(t)
        }
      })

      result.future
    }
@toggm
Copy link

@toggm toggm commented Oct 8, 2014

Based on WSClient and WSRequestHolder I implemented a multipart based request.

package helper

import com.ning.http.client.{ Response => AHCResponse, Part => AHCPart, StringPart => AHCStringPart, FilePart => AHCFilePart, _ }
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{ Future, Promise }
import play.api.Play.current
import play.api.libs.ws.WS
import com.ning.http.client.AsyncHttpClient._
import play.api.libs.iteratee.Enumerator
import collection.JavaConversions._
import java.io.File
import com.ning.http.multipart.{ FilePart => MPFilePart }

sealed trait Part {
  def toAHCPart: AHCPart
}
case class StringPart(key: String, value: String) extends Part {
  def toAHCPart = {
    new AHCStringPart(key, value)
  }
}
case class FilePart(key: String, file: File, contentType: String, charset: String) extends Part {
  def toAHCPart = {
    new MPFilePart(key, file, contentType, charset)
  }
}

case class MultiPartWSRequestHolder(client: AsyncHttpClient, builder: AsyncHttpClient#BoundRequestBuilder, url: String) extends WSRequestHolder {

  def withPart(part: Part): MultiPartWSRequestHolder = {
    builder.addBodyPart(part.toAHCPart)
    this
  }

  def withParts(parts: Part*): MultiPartWSRequestHolder = {
    parts.forall(p => builder.addBodyPart(p.toAHCPart) != null)
    this
  }

  /**
   * The method for this request
   */
  val method: String = ""

  /**
   * The body of this request
   */
  val body: WSBody = EmptyBody

  /**
   * 7
   * The headers for this request
   */
  val headers: Map[String, Seq[String]] = Map()

  /**
   * The query string for this request
   */
  val queryString: Map[String, Seq[String]] = Map()

  /**
   * A calculator of the signature for this request
   */
  val calc: Option[WSSignatureCalculator] = None

  /**
   * The authentication this request should use
   */
  val auth: Option[(String, String, WSAuthScheme)] = None

  /**
   * Whether this request should follow redirects
   */
  val followRedirects: Option[Boolean] = None

  /**
   * The timeout for the request
   */
  val requestTimeout: Option[Int] = None

  /**
   * The virtual host this request will use
   */
  val virtualHost: Option[String] = None

  /**
   * The proxy server this request will use
   */
  val proxyServer: Option[WSProxyServer] = None

  /**
   * sets the signature calculator for the request
   * @param calc
   */
  def sign(calc: WSSignatureCalculator): MultiPartWSRequestHolder = {
    this
  }

  /**
   * sets the authentication realm
   */
  def withAuth(username: String, password: String, scheme: WSAuthScheme): MultiPartWSRequestHolder = {
    //not yet supported
    this
  }

  /**
   * adds any number of HTTP headers
   * @param hdrs
   */
  def withHeaders(hdrs: (String, String)*): MultiPartWSRequestHolder = {
    hdrs.forall(h => builder.setHeader(h._1, h._2) != null)
    this
  }

  def withMethod(method: String): MultiPartWSRequestHolder = {
    builder.setMethod(method)
    this
  }

  /**
   * Sets the maximum time in milliseconds you expect the request to take.
   * Warning: a stream consumption will be interrupted when this time is reached.
   */
  def withRequestTimeout(timeout: Int): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the virtual host to use in this request
   */
  def withVirtualHost(vh: String): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the proxy server to use in this request
   */
  def withProxyServer(proxyServer: WSProxyServer): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the body for this request
   */
  def withBody(body: WSBody): MultiPartWSRequestHolder = {
    this
  }

  /**
   * adds any number of query string parameters to the
   */
  def withQueryString(parameters: (String, String)*): MultiPartWSRequestHolder = {
    val map = parameters.groupBy(_._1).map { case (k, v) => (k, v.map(_._2)) }
    val builderMap = new FluentStringsMap()
    map.forall { case (key, values) => builderMap.add(key, values) != null }

    builder.setQueryParameters(builderMap)

    this
  }

  /**
   * Sets whether redirects (301, 302) should be followed automatically
   */
  def withFollowRedirects(follow: Boolean): MultiPartWSRequestHolder = {
    builder.setFollowRedirects(follow)
    this
  }

  /**
   * Execute this request
   */
  def execute(): Future[WSResponse] = {
    var result = Promise[WSResponse]()

    client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
      override def onCompleted(response: AHCResponse) = {
        result.success(NingWSResponse(response))
        response
      }

      override def onThrowable(t: Throwable) = {
        result.failure(t)
        ()
      }
    })

    result.future
  }

  /**
   * Execute this request and stream the response body in an enumerator
   */
  def stream(): Future[(WSResponseHeaders, Enumerator[Array[Byte]])] = {
    Future.failed(new RuntimeException("Not supported"))
  }
}

case class MultPartWSClient(wsClient: WSClient) {
  val client = wsClient.underlying.asInstanceOf[AsyncHttpClient]

  def url(url: String): MultiPartWSRequestHolder = {
    val builder = client.preparePost(url)
    builder.setHeader("Content-Type", "multipart/form-data")
    MultiPartWSRequestHolder(client, builder, url)
  }
}

It can be used i..e:

val multiPartClient = MultPartWSClient(client)
    multiPartClient.url(url)
      .withParts(StringPart("title", title), 
          StringPart("someAttr1", attr1),
          StringPart("someAttr2", attr2),
          StringPart("someAttr3", attr3),
          FilePart("document", document, contentType, "UTF-8"))
      .withMethod("POST")
      .execute()

It is still not fully implemented, but can be of use to anyone. Feel free to integrate that into play or use it in your own project.

@nafg
Copy link
Contributor

@nafg nafg commented Dec 5, 2014

Simpler solution, based on http://stackoverflow.com/a/18723326/333643:

(I'm calling the mailgun API)

// Use the Ning AsyncHttpClient multipart class to get the bytes
val parts = Array[Part](
      new StringPart("from", from),
      new StringPart("to", to),
      new StringPart("subject", subject),
      new StringPart("text", text),
      new FilePart("attachment", file)
    )
    val mpre = new MultipartRequestEntity(parts, new FluentCaseInsensitiveStringsMap)
    val baos = new ByteArrayOutputStream
    mpre.writeRequest(baos)
    val bytes = baos.toByteArray
    val contentType = mpre.getContentType

// Now just send the data to the WS API
    client.url(endpoint)
      .post(bytes)(Writeable.wBytes, ContentTypeOf(Some(contentType)))

I would have liked to be able to create implicit Writeable and ContentTypeOf instances for MultipartRequestEntity. Unfortunately however, their API is based on the assumption of one content type per typeclass instance (i.e. it would have to be the same for all MultipartRequestEntity instances).

However since it has to include the boundary value, this assumption is broken. If the typeclass could wrap an A => Option[String] rather than just an Option[String], we could just have an implicit Writeable[MultipartRequstEntity] etc.

@naderghanbari
Copy link
Contributor

@naderghanbari naderghanbari commented Jun 3, 2015

Is there any plan to fix this? this will simplify things to a great extent (without this supported out of the box, sending a multipart request is very tedious).

@jroper jroper added the help wanted label Jun 3, 2015
@jroper jroper closed this Jun 3, 2015
@jroper jroper reopened this Jun 3, 2015
@cdmckay
Copy link

@cdmckay cdmckay commented Jun 25, 2015

I wrote a Writeable to use the existing MultipartFormData object for POSTing multipart form data:

https://gist.github.com/cdmckay/4b269e9017a30111556a

And here's an example of usage:

https://gist.github.com/cdmckay/5f05cbcb5327259df1cf

So you just need to drop it in and it should work with the existing WS flow.

@brianwawok
Copy link

@brianwawok brianwawok commented Aug 6, 2015

@cdmckay @nafg @toggm do you have a version that works in play 2.4? I had my own home rolled attempt, but so far all my multipart form posts using play get back a "The request body is too large" from my API I am hitting..

@david-bouyssie
Copy link

@david-bouyssie david-bouyssie commented Aug 8, 2015

@cdmckay : is there a solution to observe the progression of the upload ?
Currently I'm using scalaj-http for this purpose (https://github.com/scalaj/scalaj-http) but I would like to switch to Play WS in order to avoid extra dependencies and use asynchronous requests.
Thanks !

@ergomesh
Copy link

@ergomesh ergomesh commented Aug 25, 2015

@brianwawok

I have a similar problem I am migrating from 2.3 to 2.4 and get this now

Compilation error[object multipart is not a member of package com.ning.http] using the@cdmckay example. Anyone have this working in play2.4.x ?

@brianwawok
Copy link

@brianwawok brianwawok commented Aug 28, 2015

@ergomesh yes I was able to make this work in 2.4. I had to forcibly override my async http client version to 1.9.29, and then I did:

https://gist.github.com/brianwawok/eb9ac542eb7b4795ac06

A few things like

"Content-Type" -> mpre.getContentType)

were key, because I needed to add back in the boundary (which was lost in the play upgrade)

@mkurz
Copy link
Member

@mkurz mkurz commented Jan 20, 2016

Could someone provide a pull request? This would be great!

@brianwawok
Copy link

@brianwawok brianwawok commented Jan 20, 2016

@mkurz i have it in a hack state with no tests, not sure I am going to have time to clean up into a proper PR

@mkurz
Copy link
Member

@mkurz mkurz commented Jan 20, 2016

@brianwawok If you could find some time to submit that PR that would be really awesome and would make some people really happy! 😄

@mkurz
Copy link
Member

@mkurz mkurz commented Jan 21, 2016

@brianwawok If you do some work on this and provide a PR I could maybe help out testing things.

@jakob85
Copy link

@jakob85 jakob85 commented Feb 11, 2016

Not sure if you guys fixed the issue but I answered my own question at stackoverflow and got it working thanks to all posts here =)
http://stackoverflow.com/questions/35325942/timeout-when-sending-multipart-form-data-in-play2-2-4-6

@mkurz
Copy link
Member

@mkurz mkurz commented Feb 11, 2016

@jakob85 If you are keen to provide a pull request with implements this feature directly in Play you are welcome to do so.

@david-bouyssie
Copy link

@david-bouyssie david-bouyssie commented Feb 11, 2016

A solution including the monitoring of upload progression would be really awesome.
This feature is provided by this library: https://github.com/scalaj/scalaj-http

@schmitch
Copy link
Member

@schmitch schmitch commented Feb 19, 2016

Actually all the solutions here would be missing the ending, i.e:
Something like that:
--BRY7v0J3h3fIl91lUeQ6SxMJWqNHbRYi4Exp2C3s--

What would actually work is using the asyncHttpClient.preparePost().addBodyPart funtionallity i.e.:

val asyncHttpClient: AsyncHttpClient = ws.client.underlying

        val postBuilder = asyncHttpClient.preparePost(url).addBodyPart(filePart).addBodyPart(textPart)
          .addHeader(AUTHORIZATION, s"Basic $auth")
        val request = postBuilder.build()

        val result = Promise[NingWSResponse]()
        asyncHttpClient.executeRequest(request, new AsyncCompletionHandler[AHCResponse]() {
          override def onCompleted(response: AHCResponse) = {
            result.success(NingWSResponse(response))
            response
          }

          override def onThrowable(t: Throwable) = {
            result.failure(t)
          }
        })
        result.future

Not sure if a Writeable has access to that. I'm looking forward to maybe add a PR.

@schmitch
Copy link
Member

@schmitch schmitch commented Feb 19, 2016

Actually it would be great if somebody could help me with the PR on tests.

Actually I've left out the documentation since I'm not sure if it's good to either have something like .post(Seq(part1, part2)) and on the Java part a List. Actually maybe somebody from the core team could give me some hints.

Edit: Too bad this prolly won't go into 2.5. Actually I came too late across this.

@mkurz
Copy link
Member

@mkurz mkurz commented Feb 19, 2016

Maybe we can can still get this into 2.5?

@gmethvin
Copy link
Member

@gmethvin gmethvin commented Feb 19, 2016

@mkurz At this point there should be no problem getting it into 2.5. It would only be an issue if there are major API changes.

@mkurz
Copy link
Member

@mkurz mkurz commented Feb 19, 2016

@gmethvin Cool! 😄

@schmitch
Copy link
Member

@schmitch schmitch commented Feb 19, 2016

That would be great. I actually came across this since we are sending data to another provider which uses insecure SSL, so using the underlying AhcClient takes more work than just fixing this issue..

@david-bouyssie
Copy link

@david-bouyssie david-bouyssie commented Feb 22, 2016

Could it be possible to have access to onContentWriteProgress callback or is it too complicated to interface it with the Play API ?
https://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncCompletionHandler.html#onContentWriteProgress%28long,%20long,%20long%29

This would be a very useful feature !

@gmethvin gmethvin closed this in b9b4afe Mar 15, 2016
gmethvin added a commit that referenced this issue Mar 15, 2016
(WIP) Fixes #902 added a way to sent multipart/form-data requests via play-ws
@david-bouyssie
Copy link

@david-bouyssie david-bouyssie commented Mar 16, 2016

Do you have any tip to monitor the upload progress ?
Do I have to use the underlying client to achieve that ?
I was thinking about using reactive streams to be informed periodically about the upload progress update.
Thanks !

mkurz added a commit to mkurz/playframework that referenced this issue Mar 16, 2016
gmethvin added a commit that referenced this issue Mar 16, 2016
[Backport] Fixes #902 added multipart/form-data requests to play-ws
@AndreyIlin
Copy link

@AndreyIlin AndreyIlin commented Mar 29, 2016

Is there any workaround for multipart/form-data for play 2.5?

@schmitch
Copy link
Member

@schmitch schmitch commented Mar 29, 2016

it will when 2.5.1 hit @AndreyIlin

@AndreyIlin
Copy link

@AndreyIlin AndreyIlin commented Mar 29, 2016

Thanks @schmitch, waiting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.