Accept per-field headers in multipart POST requests (and create multipart/mixed requests)

[JanJakes]

Currently there is no way of doing such requests:

POST /api/ HTTP/1.1
Host: 127.0.0.1:5000
Content-Type: multipart/form-data; boundary=495b38322a354377913185cb452518fd

--495b38322a354377913185cb452518fd
Content-Disposition: form-data; name="data"
Content-Type: application/json; charset=UTF-8

{"key2": "value2", "key1": "value1"}
--495b38322a354377913185cb452518fd
Content-Disposition: form-data; name="file"; filename="image.jpg"

...image data...
--495b38322a354377913185cb452518fd--

Sending a file along with a JSON encoded metadata is a very typical task for API implementations.

What is even is weirder that we can do this for files but not for other fields that are sent with the files.

Also, is there any way to compose a multipart POST request with multipart/mixed header? (With keeping the possibility to choose the fields and their headers.)

Thanks, Jan

[Lukasa]

Yes there is. Take a look at the changelog.

The syntax for doing this and including things that aren't files (like some JSON) is a bit of a pain, but here it is:

import json
from cStringIO import StringIO

metadata = {'key1': 'value1', 'key2': 'value2'}
json = StringIO(json.dumps(metadata))

files = {"data": ("", json, "application/json; charset=UTF-8", {}), "file": ("image.jpg", imagefile, "", {})}
r = requests.post('http://www.api.com/, files=files)

The reason this API is totally nasty is because the multipart/form-data content type is very flexible. This makes it difficult for us to provide a clean API that covers all use cases. I'm strongly averse to the current method of doing it. I think we should be providing a contrib library (something like requests-multipart) that is capable of doing all of this craziness, rather than leaving it in Requests.

Finally, yes, you can do a multipart/mixed header by breaking out the request preparation as you can see in the documentation. An example:

from requests import Request, Session

# Insert all that files nonsense here.

s = Session()
req = Request('POST',  url, files=files)
prepped = s.prepare_request(req)
prepped.headers['Content-Type'] = 'multipart/mixed'
resp = s.send(prepped)

[JanJakes]

Thanks a lot for such a great answer! Yes, the API for this isn't straightforward but at least it's possible without touching urllib3 and stuff.

I also don't like the current method because it kind of covers the most typical use cases but some more special cases (yet still valid ones) are maybe possible but only with those "hacks". The multipart request is so flexible that the best thing in my opinion would be to have something like a MultipartRequest that would be a composition of any Requests (or RequestFields or whatever).

[Lukasa]

Anything we do to improve our flexibility here should be done outside of requests core. It's just complexity we don't need. If anything, I'd like to yank most of this out of Requests core, allow people only to do the most basic multipart uploads, and then provide an external module that does the rest. Not sure Kenneth would agree with me though.

[sigmavirus24]

@Lukasa there was a reason I started https://github.com/sigmavirus24/requests-data-schemes. It was a bad name but the intent was to provide a better way of doing complicated multipart uploads (specifically when no files should be sent, which is an extremely common use case).

[piotr-dobrogost]

Related: urllib3/urllib3#215