Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate multipart posts without a file #1081

Closed
ghost opened this Issue Jan 3, 2013 · 36 comments

Comments

Projects
None yet
7 participants
@ghost
Copy link

ghost commented Jan 3, 2013

Currently, the only way to have a multipart form request is r = requests.post(url, data=payload, files=files)
which may have a component

--3eeaadbfda0441b8be821bbed2962e4d
Content-Disposition: form-data; name="file"; filename="filename.txt"
Content-Type: text/plain

content
--3eeaadbfda0441b8be821bbed2962e4d--

However, I run into instances where posts are required to be in a multipart format without an associated file, like:

--3eeaadbfda0441b8be821bbed2962e4d
Content-Disposition: form-data; name="key1"

value1
--3eeaadbfda0441b8be821bbed2962e4d

but the latter is impossible to generate without the former.

Perhaps we can add a flag like r = requests.post(url, data=payload, multipart=True) that forces a post to be multipart, even without a file.

I am happy to work on implementing this if it sounds like a good idea.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 3, 2013

This has been discussed before. It would represent a significant change to the API which I'm not sure @kennethreitz would like.

Personally, I would be more in favor of exposing a function to generate multipart data from dictioniaries (lists of tuples, etc) in the API so users can use that and just pass the generated data to requests. Ostensibly, if they're not using files, there shouldn't be a huge memory hit, but even so, they already have one huge string in memory, the second won't really kill them and it would be their fault, not ours.

Perhaps @kennethreitz would be more amenable to the second solution. I don't think it fits in with requests' design philosophy either and would be extraordinarily bizarre given the rest of the API, but shrug who knows.

@piotr-dobrogost

This comment has been minimized.

Copy link
Contributor

piotr-dobrogost commented Jan 3, 2013

Indeed, current API does not support sending multipart data other than files and this is a bad thing. See issue #935 and urllib3/urllib3/issues/120

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Jan 3, 2013

Maybe I'm missing something, but the changes seem rather minor to me. I forked the repository and have a proposed change in my version here: https://github.com/spacecase/requests/commit/45b0b3ce1e76b241b323570a5fc88ae2089c3c3d
(if there's a better way to do this let me know? I'm rather new to github).

It might need a couple unittests and would need a change in a docstring in api.py, but is something like this reasonable?

@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 5, 2013

The issue with the change isn't that it's complicated to implement, it's that it's a divergence in the API. I'm kind of on the fence in this ongoing discussion: I think it would probably be useful to have a good way to upload multipart form-data, but I also think the current files API is a very good one. I don't think that adding multipart to the Request API is the way to go though.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 5, 2013

Honestly, I would personally rather control it through the content type header but that would likely be 100x more error prone and confusing to new users than what we currently do. I'm on the fence about this as well, but I would still err on the side of not doing this.

Why? If we accept this feature request, then it becomes more likely that someone will complain about not having a json parameter. And I'm sure other people could come up with even more parameters they'd love to see added. As it is now, the API does exactly what it should and has little to no kruft. One way of looking at this is KISS. This makes the requests and returns a great object that makes using the response easy and natural. It does what is advertised and you do the rest. It does advertise multipart encoding and does it through its documented design. It may look awkward but it is documented and it works.

@piotr-dobrogost

This comment has been minimized.

Copy link
Contributor

piotr-dobrogost commented Jan 6, 2013

(...) someone will complain about not having a json parameter

There would be no ground for such a complain because json is a subtype of application media type (rfc4627) not multipart media type (httpbis draft 21)

It may look awkward (...)

It is awkward whereas it should not be.

After reading previous comments I'd like to reiterate my position: multipart/form-data is the most common multipart MIME type currently used (citation needed :)) and not supporting it is a gross omission.

@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 6, 2013

@piotr-dobrogost: Despite what I said above, I don't find the argument you just made even slightly compelling.

Requests does not support MIME types, it supports use cases. This viewpoint makes both your comments above seem weird. For instance, the complaint about not having a JSON parameter will be because uploading JSON-formatted data is very common - probably more common amongst users of Requests than uploading non-file multipart data. Arguing that we won't provide it because 'we only special-case subtypes of multipart' just seems like a bizarre thing to say.

Regardless, the core of the issue is this: the API is the whole point of this library. If you can't come up with a beautiful way to implement this functionality, it will not happen.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 6, 2013

There would be no ground for such a complain [sic]

@piotr-dobrogost you have more hope for the future of developers than I do. Beyond that, the requested parameter is inexact. It would need to be called something more like form_data than multipart. As you note, (although indirectly) multipart could refer to plenty of different media types.

It is awkward whereas it should not be.

Not all things can be elegant (even in python).

and not supporting it

But it is supported. But that aside, we have data for application/x-www-form-urlencoded, files (or files+data) for multipart/form-data, why do we need yet another parameter for just multipart/form-data when we have it?

You don't need to use a combination of data and files to get the intended result. You can do something akin to: requests.post('http://example.com/', files=[('key1', 'param1'), ('key2', 'param2')]) without having an actual file there.

And if we were to be exact, form_data might not be entirely obvious, so why not use the parameter multipart_form_data, but now that's clumsy as well. It's explicit, yes, and PEP 8 calls for explicity, but the existing behaviour is well documented. If @kennethreitz does decide to accept this feature request, all the parameter will need to do is act as an alias to the files parameter. But considering the behaviour is already supported I don't think it is necessary.

@kenneth-reitz

This comment has been minimized.

Copy link
Collaborator

kenneth-reitz commented Jan 6, 2013

@sigmavirus24 and @Lukasa summed this up perfectly.

@kenneth-reitz

This comment has been minimized.

Copy link
Collaborator

kenneth-reitz commented Jan 6, 2013

@piotr-dobrogost your contributions are appreciated, but not your tone. Please stop making firm assertions against our project and it's goals.

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Jan 6, 2013

From my perspective it is quite frustrating that requests already has support for multi-part posts, but does not give the user access to that without using a file. @sigmavirus24 's suggestion to just stick a file in there is not adequate. The web applications I work with will return error codes if something like this is tried.

I suspect this will continue to be a problem for users in the future, and I am a little confused why there doesn't seem to be an effort to address this.

@kenneth-reitz

This comment has been minimized.

Copy link
Collaborator

kenneth-reitz commented Jan 6, 2013

See my response to #935

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Jan 6, 2013

Oh, thanks. I look forward to it!

@piotr-dobrogost

This comment has been minimized.

Copy link
Contributor

piotr-dobrogost commented Jan 6, 2013

@Lukasa

Requests does not support MIME types, it supports use cases.

Sending multipart/form-data data is a common use case.

(...) uploading JSON-formatted data is very common (...)

We can't compare sending json with sending multipart/form-data. Sending json is easy; you set Content-type, encode data in one line using built-in module and that's it. Sending multipart/form-data is more complex because the request's body has to have specific structure. In other words sending json is kind of transparent as much as HTTP is concerned but sending multipart/form-data is not. As Requests is the HTTP library it should take care of creating such a structure.

If you can't come up with a beautiful way to implement this functionality, it will not happen.

Having current files param to send multipart/form-data which has NOTHING to do with files is not beautiful in any way (it's ugly), yet it somehow managed to get into codebase :). Coming up with something less ugly is really easy :)

@sigmavirus24

(...) but the existing behaviour is well documented.

No amount of documentation makes good API out of bad API. The better the API the less it needs documentation.

You can do something akin to (...)

Right, but this is very misleading and unintuitive. I described what's wrong with using files param for this in my previous comment.

Bottom line: to come up with something better we need to admit the current api is bad in respect to sending multipart/form-data data.

@spaceCase

From my perspective it is quite frustrating that requests already has support for multi-part posts, but does not give the user access to that without using a file

I agree and that's why I created issue #935.

@kenneth-reitz

This comment has been minimized.

Copy link
Collaborator

kenneth-reitz commented Jan 6, 2013

This issue is closed.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 7, 2013

Forgive me @kennethreitz but @spaceCase I did not use a file anywhere in that example. I used the files parameter to do exactly what you want. Had I used a file you would have seen open('filename').

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Jan 7, 2013

@sigmavirus24, Unfortunately that is not what I want. It's not a matter of whether the client is reading from a file or not. It's what is being told to the server. In your example, the post body is

--2f8732ee35564115a6c6e0c1032773e8
Content-Disposition: form-data; name="key1"; filename="key1"
Content-Type: application/octet-stream

param1
--2f8732ee35564115a6c6e0c1032773e8
Content-Disposition: form-data; name="key2"; filename="key2"
Content-Type: application/octet-stream

param2
--2f8732ee35564115a6c6e0c1032773e8--

Note the use of filename=. It's telling the server a file is being sent. In the web apps I work with this incorrect behavior results in an error.

I'm sorry, but for you to be so presumptuous to tell me that is exactly what I want offends me. I came to offer ideas and help, and it sounds like you don't want either in this case. That is fine, but please don't make me feel talked down to.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 7, 2013

Hm, seems I remembered the behavior incorrectly. Sorry about that. It wasn't my intention to offend you at all or make you feel talked down to. That's definitely enough to make me sway to needing a better way to handle multipart/form-data, but for now I still disagree that the ideas for the API are not at all elegant.

@kenneth-reitz

This comment has been minimized.

Copy link
Collaborator

kenneth-reitz commented Jan 7, 2013

@spaceCase please see my response to #935. Things will be better. Your feedback is important and much appreciated. :)

sigmavirus24 added a commit to sigmavirus24/requests-data-schemes that referenced this issue Jan 7, 2013

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 7, 2013

@spaceCase as a gesture to show that your opinion does matter, check out sigmavirus24/requests-data-schemes as a stop-gap measure. You'll have to set your own Content-Type header, but that may be a minor annoyance.

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Jan 7, 2013

Thanks @sigmavirus24 , I'll try it out. It looks like it will do the trick.
I appreciate that you didn't mean to offend me. I feel better about that and I don't hold anything against you or the project.

@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 7, 2013

Yeah, it seems I come off as brash to some people, so I guess I need to refine what I write on the internet. I don't understand it and others have agreed with me, but I'm working on it. I guess I just need a sample size large enough to realize what's offensive to people without my intending it to be (besides my making an ass of myself by remembering something working a way in which it doesn't).

@deerstalker

This comment has been minimized.

Copy link

deerstalker commented Jan 12, 2014

I have a file and some key-values to post. So what is the correct way to send such a mutilpart-form request? I hope you can help me.

-----------------------------7dee5302248e
Content-Disposition: form-data; name="up"; filename="aa.PNG"
Content-Type: image/png

file data
---------------------------7dee5302248e
Content-Disposition: form-data; name="exp"


-----------------------------7dee5302248e
Content-Disposition: form-data; name="ptext"

text
-----------------------------7dee5302248e
Content-Disposition: form-data; name="board"

DV_Studio
-----------------------------7dee5302248e--

I tried this way but only got a 504 error.
myfile=[('file',open('bb.jpg')),('exp','python'),('ptext',''),('board','DV_Studio')]
r = requests.post(url,files=myfile)
@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Jan 12, 2014

@deerstalker for questions please use StackOverflow. To answer your question though, there is a project underway to address this problem sigmavirus24/requests-toolbelt

@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 12, 2014

Also note that I've answered this question before on StackOverflow, as you can see here.

@jwoillez

This comment has been minimized.

Copy link

jwoillez commented Jan 21, 2014

I have the same issue of generating multipart posts without a file. At the moment, it makes no difference if you do requests.post(url, data=data_dict) or requests.post(url, data=data_dict, files={}). But, since the files keyword defaults to None, we should be able to have two distinct behaviours. A multipart/form-data when files={} is specified, and application/x-www-form-urlencoded when not.

Did I miss something?

@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 21, 2014

@jwoillez Did you follow the stack overflow link I posted?

@jwoillez

This comment has been minimized.

Copy link

jwoillez commented Jan 21, 2014

I think so, but your answer over there deals with Multipart POST with one file. I went back to the original issue: Multipart POST with no file.

@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 21, 2014

The file object is allowed to be a string. =) That should provide you with the information you wanted.

@jwoillez

This comment has been minimized.

Copy link

jwoillez commented Jan 21, 2014

But if I follow your suggestion, won't I end-up with something like this:

--3eeaadbfda0441b8be821bbed2962e4d
Content-Disposition: form-data; name="file"; filename="filename.txt"
Content-Type: text/plain

content
--3eeaadbfda0441b8be821bbed2962e4d--

where content is the string that you invite me to use instead of the file?

I'm really after this only:

--3eeaadbfda0441b8be821bbed2962e4d
Content-Disposition: form-data; name="key1"

value1
--3eeaadbfda0441b8be821bbed2962e4d
@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 21, 2014

The fields in the tuple you don't want can be left as the defaults:

files = {'name': ('', 'content')}

In the future, please direct your questions to StackOverflow. All the maintainers regularly keep track of it, and it's the more appropriate place to ask these questions.

@jwoillez

This comment has been minimized.

Copy link

jwoillez commented Jan 21, 2014

That's the answer I was looking for, thanks. Sorry for the noise.

@jwoillez

This comment has been minimized.

Copy link

jwoillez commented Jan 22, 2014

Maybe one last question, is the following possible (specified empty filename, empty content)?

--3eeaadbfda0441b8be821bbed2962e4d
Content-Disposition: form-data; name="file"; filename=""
Content-Type: text/plain


--3eeaadbfda0441b8be821bbed2962e4d--
@Lukasa

This comment has been minimized.

Copy link
Collaborator

Lukasa commented Jan 22, 2014

You can provide an empty content by using an empty string in the content section of the tuple. You can't provide a literal empty filename, but a non-present filename should be treated in exactly the same way.

@tomchristie

This comment has been minimized.

Copy link

tomchristie commented Oct 10, 2018

I've occasionally needed to do this for various odd reasons.
I'd suggest this approach for anyone wanting to strong-arm the API into this use case:

class ForceMultipartDict(dict):
    def __bool__(self):
        return True


FORCE_MULTIPART = ForceMultipartDict()  # An empty dict that boolean-evaluates as `True`.


client.post("/", data={"some": "data"}, files=FORCE_MULTIPART)
@sigmavirus24

This comment has been minimized.

Copy link
Collaborator

sigmavirus24 commented Oct 10, 2018

Or you can use the toolbelt and not resort to hacks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.