Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a post() method on WS for POSTing multipart/form-data

Closed
julien-lafont opened this issue Mar 25, 2013 · 32 comments
Closed

Provide a post() method on WS for POSTing multipart/form-data #902

julien-lafont opened this issue Mar 25, 2013 · 32 comments

Comments

@julien-lafont
Copy link

julien-lafont commented Mar 25, 2013

There is actually no way to post a multipart/form-data, without encoding manually the body (and this is tricky!)

It's sad to not have a WS method which allow to call a controller like the "File upload example"

@ryanoglesby08
Copy link

ryanoglesby08 commented Apr 3, 2013

This functionality would be very helpful. Agreed.

A method as part of FakeRequest that is something along the lines of "fakeRequest().withMultipartFormData(formData);" would help immensely with testing controllers using multipart form data.

@joost-de-vries
Copy link

joost-de-vries commented Jul 6, 2013

Another strong vote for this feature. I've closed the other issue #1302 in favour of this one.

@ewilson
Copy link

ewilson commented Jul 9, 2014

Please consider implementing this.

@ornicar
Copy link
Contributor

ornicar commented Jul 31, 2014

👍

1 similar comment
@AlexGalays
Copy link

AlexGalays commented Jul 31, 2014

👍

@cxvvs
Copy link

cxvvs commented Jul 31, 2014

In the meantime, here is a workaround that was greatly inspired by some code in Play

import com.ning.http.client.{Response => AHCResponse, _}
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{Future, Promise}
import play.api.Play.current

  object WSUtil {

    def post(url: String, bodyParts: List[Part]): Future[Response] = {
      val client = WS.client(current).underlying.asInstanceOf[AsyncHttpClient]

      val builder = client.preparePost(url)

      builder.setHeader("Content-Type", "multipart/form-data")
      bodyParts.foreach(builder.addBodyPart)

      var result = Promise[NingWSResponse]()

      client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
        override def onCompleted(response: AHCResponse) = {
          result.success(NingWSResponse(response))
          response
        }

        override def onThrowable(t: Throwable) = {
          result.failure(t)
        }
      })

      result.future
    }

@toggm
Copy link

toggm commented Oct 8, 2014

Based on WSClient and WSRequestHolder I implemented a multipart based request.

package helper

import com.ning.http.client.{ Response => AHCResponse, Part => AHCPart, StringPart => AHCStringPart, FilePart => AHCFilePart, _ }
import play.api.libs.ws._
import play.api.libs.ws.ning.NingWSResponse
import scala.concurrent.{ Future, Promise }
import play.api.Play.current
import play.api.libs.ws.WS
import com.ning.http.client.AsyncHttpClient._
import play.api.libs.iteratee.Enumerator
import collection.JavaConversions._
import java.io.File
import com.ning.http.multipart.{ FilePart => MPFilePart }

sealed trait Part {
  def toAHCPart: AHCPart
}
case class StringPart(key: String, value: String) extends Part {
  def toAHCPart = {
    new AHCStringPart(key, value)
  }
}
case class FilePart(key: String, file: File, contentType: String, charset: String) extends Part {
  def toAHCPart = {
    new MPFilePart(key, file, contentType, charset)
  }
}

case class MultiPartWSRequestHolder(client: AsyncHttpClient, builder: AsyncHttpClient#BoundRequestBuilder, url: String) extends WSRequestHolder {

  def withPart(part: Part): MultiPartWSRequestHolder = {
    builder.addBodyPart(part.toAHCPart)
    this
  }

  def withParts(parts: Part*): MultiPartWSRequestHolder = {
    parts.forall(p => builder.addBodyPart(p.toAHCPart) != null)
    this
  }

  /**
   * The method for this request
   */
  val method: String = ""

  /**
   * The body of this request
   */
  val body: WSBody = EmptyBody

  /**
   * 7
   * The headers for this request
   */
  val headers: Map[String, Seq[String]] = Map()

  /**
   * The query string for this request
   */
  val queryString: Map[String, Seq[String]] = Map()

  /**
   * A calculator of the signature for this request
   */
  val calc: Option[WSSignatureCalculator] = None

  /**
   * The authentication this request should use
   */
  val auth: Option[(String, String, WSAuthScheme)] = None

  /**
   * Whether this request should follow redirects
   */
  val followRedirects: Option[Boolean] = None

  /**
   * The timeout for the request
   */
  val requestTimeout: Option[Int] = None

  /**
   * The virtual host this request will use
   */
  val virtualHost: Option[String] = None

  /**
   * The proxy server this request will use
   */
  val proxyServer: Option[WSProxyServer] = None

  /**
   * sets the signature calculator for the request
   * @param calc
   */
  def sign(calc: WSSignatureCalculator): MultiPartWSRequestHolder = {
    this
  }

  /**
   * sets the authentication realm
   */
  def withAuth(username: String, password: String, scheme: WSAuthScheme): MultiPartWSRequestHolder = {
    //not yet supported
    this
  }

  /**
   * adds any number of HTTP headers
   * @param hdrs
   */
  def withHeaders(hdrs: (String, String)*): MultiPartWSRequestHolder = {
    hdrs.forall(h => builder.setHeader(h._1, h._2) != null)
    this
  }

  def withMethod(method: String): MultiPartWSRequestHolder = {
    builder.setMethod(method)
    this
  }

  /**
   * Sets the maximum time in milliseconds you expect the request to take.
   * Warning: a stream consumption will be interrupted when this time is reached.
   */
  def withRequestTimeout(timeout: Int): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the virtual host to use in this request
   */
  def withVirtualHost(vh: String): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the proxy server to use in this request
   */
  def withProxyServer(proxyServer: WSProxyServer): MultiPartWSRequestHolder = {
    this
  }

  /**
   * Sets the body for this request
   */
  def withBody(body: WSBody): MultiPartWSRequestHolder = {
    this
  }

  /**
   * adds any number of query string parameters to the
   */
  def withQueryString(parameters: (String, String)*): MultiPartWSRequestHolder = {
    val map = parameters.groupBy(_._1).map { case (k, v) => (k, v.map(_._2)) }
    val builderMap = new FluentStringsMap()
    map.forall { case (key, values) => builderMap.add(key, values) != null }

    builder.setQueryParameters(builderMap)

    this
  }

  /**
   * Sets whether redirects (301, 302) should be followed automatically
   */
  def withFollowRedirects(follow: Boolean): MultiPartWSRequestHolder = {
    builder.setFollowRedirects(follow)
    this
  }

  /**
   * Execute this request
   */
  def execute(): Future[WSResponse] = {
    var result = Promise[WSResponse]()

    client.executeRequest(builder.build(), new AsyncCompletionHandler[AHCResponse]() {
      override def onCompleted(response: AHCResponse) = {
        result.success(NingWSResponse(response))
        response
      }

      override def onThrowable(t: Throwable) = {
        result.failure(t)
        ()
      }
    })

    result.future
  }

  /**
   * Execute this request and stream the response body in an enumerator
   */
  def stream(): Future[(WSResponseHeaders, Enumerator[Array[Byte]])] = {
    Future.failed(new RuntimeException("Not supported"))
  }
}

case class MultPartWSClient(wsClient: WSClient) {
  val client = wsClient.underlying.asInstanceOf[AsyncHttpClient]

  def url(url: String): MultiPartWSRequestHolder = {
    val builder = client.preparePost(url)
    builder.setHeader("Content-Type", "multipart/form-data")
    MultiPartWSRequestHolder(client, builder, url)
  }
}

It can be used i..e:

val multiPartClient = MultPartWSClient(client)
    multiPartClient.url(url)
      .withParts(StringPart("title", title), 
          StringPart("someAttr1", attr1),
          StringPart("someAttr2", attr2),
          StringPart("someAttr3", attr3),
          FilePart("document", document, contentType, "UTF-8"))
      .withMethod("POST")
      .execute()

It is still not fully implemented, but can be of use to anyone. Feel free to integrate that into play or use it in your own project.

@nafg
Copy link
Contributor

nafg commented Dec 5, 2014

Simpler solution, based on http://stackoverflow.com/a/18723326/333643:

(I'm calling the mailgun API)

// Use the Ning AsyncHttpClient multipart class to get the bytes
val parts = Array[Part](
      new StringPart("from", from),
      new StringPart("to", to),
      new StringPart("subject", subject),
      new StringPart("text", text),
      new FilePart("attachment", file)
    )
    val mpre = new MultipartRequestEntity(parts, new FluentCaseInsensitiveStringsMap)
    val baos = new ByteArrayOutputStream
    mpre.writeRequest(baos)
    val bytes = baos.toByteArray
    val contentType = mpre.getContentType

// Now just send the data to the WS API
    client.url(endpoint)
      .post(bytes)(Writeable.wBytes, ContentTypeOf(Some(contentType)))

I would have liked to be able to create implicit Writeable and ContentTypeOf instances for MultipartRequestEntity. Unfortunately however, their API is based on the assumption of one content type per typeclass instance (i.e. it would have to be the same for all MultipartRequestEntity instances).

However since it has to include the boundary value, this assumption is broken. If the typeclass could wrap an A => Option[String] rather than just an Option[String], we could just have an implicit Writeable[MultipartRequstEntity] etc.

@naderghanbari
Copy link
Contributor

naderghanbari commented Jun 3, 2015

Is there any plan to fix this? this will simplify things to a great extent (without this supported out of the box, sending a multipart request is very tedious).

@jroper jroper closed this as completed Jun 3, 2015
@jroper jroper reopened this Jun 3, 2015
@cdmckay
Copy link

cdmckay commented Jun 25, 2015

I wrote a Writeable to use the existing MultipartFormData object for POSTing multipart form data:

https://gist.github.com/cdmckay/4b269e9017a30111556a

And here's an example of usage:

https://gist.github.com/cdmckay/5f05cbcb5327259df1cf

So you just need to drop it in and it should work with the existing WS flow.

@brianwawok
Copy link

brianwawok commented Aug 6, 2015

@cdmckay @nafg @toggm do you have a version that works in play 2.4? I had my own home rolled attempt, but so far all my multipart form posts using play get back a "The request body is too large" from my API I am hitting..

@david-bouyssie
Copy link

david-bouyssie commented Aug 8, 2015

@cdmckay : is there a solution to observe the progression of the upload ?
Currently I'm using scalaj-http for this purpose (https://github.com/scalaj/scalaj-http) but I would like to switch to Play WS in order to avoid extra dependencies and use asynchronous requests.
Thanks !

@ergomesh
Copy link

ergomesh commented Aug 25, 2015

@brianwawok

I have a similar problem I am migrating from 2.3 to 2.4 and get this now

Compilation error[object multipart is not a member of package com.ning.http] using the@cdmckay example. Anyone have this working in play2.4.x ?

@brianwawok
Copy link

brianwawok commented Aug 28, 2015

@ergomesh yes I was able to make this work in 2.4. I had to forcibly override my async http client version to 1.9.29, and then I did:

https://gist.github.com/brianwawok/eb9ac542eb7b4795ac06

A few things like

"Content-Type" -> mpre.getContentType)

were key, because I needed to add back in the boundary (which was lost in the play upgrade)

@mkurz
Copy link
Member

mkurz commented Jan 20, 2016

Could someone provide a pull request? This would be great!

@brianwawok
Copy link

brianwawok commented Jan 20, 2016

@mkurz i have it in a hack state with no tests, not sure I am going to have time to clean up into a proper PR

@mkurz
Copy link
Member

mkurz commented Jan 20, 2016

@brianwawok If you could find some time to submit that PR that would be really awesome and would make some people really happy! 😄

@mkurz
Copy link
Member

mkurz commented Jan 21, 2016

@brianwawok If you do some work on this and provide a PR I could maybe help out testing things.

@jakob85
Copy link

jakob85 commented Feb 11, 2016

Not sure if you guys fixed the issue but I answered my own question at stackoverflow and got it working thanks to all posts here =)
http://stackoverflow.com/questions/35325942/timeout-when-sending-multipart-form-data-in-play2-2-4-6

@mkurz
Copy link
Member

mkurz commented Feb 11, 2016

@jakob85 If you are keen to provide a pull request with implements this feature directly in Play you are welcome to do so.

@david-bouyssie
Copy link

david-bouyssie commented Feb 11, 2016

A solution including the monitoring of upload progression would be really awesome.
This feature is provided by this library: https://github.com/scalaj/scalaj-http

@schmitch
Copy link
Member

schmitch commented Feb 19, 2016

Actually all the solutions here would be missing the ending, i.e:
Something like that:
--BRY7v0J3h3fIl91lUeQ6SxMJWqNHbRYi4Exp2C3s--

What would actually work is using the asyncHttpClient.preparePost().addBodyPart funtionallity i.e.:

val asyncHttpClient: AsyncHttpClient = ws.client.underlying

        val postBuilder = asyncHttpClient.preparePost(url).addBodyPart(filePart).addBodyPart(textPart)
          .addHeader(AUTHORIZATION, s"Basic $auth")
        val request = postBuilder.build()

        val result = Promise[NingWSResponse]()
        asyncHttpClient.executeRequest(request, new AsyncCompletionHandler[AHCResponse]() {
          override def onCompleted(response: AHCResponse) = {
            result.success(NingWSResponse(response))
            response
          }

          override def onThrowable(t: Throwable) = {
            result.failure(t)
          }
        })
        result.future

Not sure if a Writeable has access to that. I'm looking forward to maybe add a PR.

@schmitch
Copy link
Member

schmitch commented Feb 19, 2016

Actually it would be great if somebody could help me with the PR on tests.

Actually I've left out the documentation since I'm not sure if it's good to either have something like .post(Seq(part1, part2)) and on the Java part a List. Actually maybe somebody from the core team could give me some hints.

Edit: Too bad this prolly won't go into 2.5. Actually I came too late across this.

@mkurz
Copy link
Member

mkurz commented Feb 19, 2016

Maybe we can can still get this into 2.5?

@gmethvin
Copy link
Member

gmethvin commented Feb 19, 2016

@mkurz At this point there should be no problem getting it into 2.5. It would only be an issue if there are major API changes.

@mkurz
Copy link
Member

mkurz commented Feb 19, 2016

@gmethvin Cool! 😄

@schmitch
Copy link
Member

schmitch commented Feb 19, 2016

That would be great. I actually came across this since we are sending data to another provider which uses insecure SSL, so using the underlying AhcClient takes more work than just fixing this issue..

@david-bouyssie
Copy link

david-bouyssie commented Feb 22, 2016

Could it be possible to have access to onContentWriteProgress callback or is it too complicated to interface it with the Play API ?
https://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncCompletionHandler.html#onContentWriteProgress%28long,%20long,%20long%29

This would be a very useful feature !

gmethvin added a commit that referenced this issue Mar 15, 2016
(WIP) Fixes #902 added a way to sent multipart/form-data requests via play-ws
@david-bouyssie
Copy link

david-bouyssie commented Mar 16, 2016

Do you have any tip to monitor the upload progress ?
Do I have to use the underlying client to achieve that ?
I was thinking about using reactive streams to be informed periodically about the upload progress update.
Thanks !

mkurz pushed a commit to mkurz/playframework that referenced this issue Mar 16, 2016
gmethvin added a commit that referenced this issue Mar 16, 2016
[Backport] Fixes #902 added multipart/form-data requests to play-ws
@ilinandrii
Copy link

ilinandrii commented Mar 29, 2016

Is there any workaround for multipart/form-data for play 2.5?

@schmitch
Copy link
Member

schmitch commented Mar 29, 2016

it will when 2.5.1 hit @AndreyIlin

@ilinandrii
Copy link

ilinandrii commented Mar 29, 2016

Thanks @schmitch, waiting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests