Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add file upload support? #42

Open
toastdriven opened this Issue · 27 comments
@toastdriven
Owner

See https://gist.github.com/709890 for an example.

Possible issues:

  • Settle on a final format.
  • Not wanting to download a large base64'd string with each API hit.
  • Validation?
@klipstein

There now is an alternative solution using multipart/related:

klipstein@e0f86dd

This solution relies on a standard (RFC 2387) that is mostly used within email. RFC 2387 defines that there is a root-document (xml/json/yaml/...) that contains links to other multiparts through "cid:the-linked-content-id" and where the linked documents are attached within the same multipart body.

@tiabas

I know that this feature has not yet been added to the code base however, I would like to use it. Does the code snippet provided work well?

@klipstein

I'm using the Base64FileField (https://gist.github.com/709890) for some resources of my API and it is working well with it. The only caveat is, that it is a custom solution and that you can't use such an API endpoint in every browser (in modern browsers you can access a file field on the client side and convert the data into base64, but for older ones you have to use multipart/form-data to submit file fields).

After a long time playing around with various solutions (klipstein@e0f86dd, https://github.com/klipstein/django-tastypie/commits/form-data) doing file-handling within tastypie I came to the conclusion that this can't be solved elegantly if you want to support file uploads from every browser.

Meanwhile I use a separate normal django view that allows uploading a file via multipart/form-data and which is returning a hash of that file. This hash then can be referenced in a custom TempfileField within tastypie like that: {"myfilefield": "tmpfile://123hash123"}.

If I know that a certain resource is just used from a server component I use the Base64FileField from the above gist.

@tiabas

Thanks. I just looked at your multipart/form-data implementation. It looks pretty good, a lot simpler than the base64 version. I only need this feature in order to upload data from any mobile device using tastypie as by API engine. I believe this should work fine. I'll hold off the base64 for now though.
I hope toastdriven will give this a look and consider adding the feature very soon.

@onyxfish

I'm looking to solve this problem for @pandaproject and I'm curious if anyone has done any further work on this issue? Also, @klipstein, I'm wondering about your statement, "this can't be solved elegantly if you want to support file uploads from every browser.". It seems that the multipart-form-data solution should be cross-browser, right? It there something about this solution that is otherwise problematic?

@claudiobi

I needed file upload (attachment) through POST and I solved the problem importing the patches by michaelmwu.
This is the commit that allows file uploads in my fork: ff0000@1fbc0a

@gourneau

Has this made it into 9.9.10?

@Uznick

Any updates on that?

@toastdriven
Owner

@Uznick @gourneau No, there are not updates yet. There a couple good solutions presented above. When I commit, I will update this ticket.

@rca

I ran into this problem and ended up using the ff0000 fork mentioned by claudiobi above.

@Uznick

using fork is not a very nice solution :(

@rca

@Uznick: I would rather be using the mainline, but at least my problem is fixed.

@doda

<3 @claudiobi works perfectly you can pip install django-tastypie-with-file-upload-and-model-form-validation

Here's requests example code to POST:
r = requests.post('http://localhost:8000/api/v1/img/',
files={'img_file':open('/new/01234.jpg','rb'), 'img_thumb':open('img.jpg','rb')}, auth=('user', 'pass))

for the class Img with fields img_file and img_thumb

@mkascel

@doda I'm confused about what exactly gets installed by pip with that command. The version reports as 1.0.0-beta but when I try going through the tutorial I get an error that tastypie.utils.timezone doesn't exist. I've verified that locally, the tastypie/utils/timezone.py file is not there, but it should be if it's merged with the upstream repo?

@doda

@mkascel i believe i got the same error when i went through the tutorial with the official tastypie branch, solution for me was to simple not use the timezone stuff (since i don't need it)

@amccloud

If deserialize was passed request this could be moved to a custom deserializer class like it ought to be. If you don't want to use a separate branch you can use this mixin.

class MultipartResource(object):
    def deserialize(self, request, data, format=None):
        if not format:
            format = request.META.get('CONTENT_TYPE', 'application/json')

        if format == 'application/x-www-form-urlencoded':
            return request.POST

        if format.startswith('multipart'):
            data = request.POST.copy()
            data.update(request.FILES)

            return data

        return super(MultipartResource, self).deserialize(request, data, format)

Something like this should work:

class Post(MultipartResource, ModelResource):
    class Meta(object):
        queryset = Post.objects.all()
        fields = ('title', 'body', 'image')

Then you can POST an image encoded as multipart form-data and a simple multipart form should also work.

curl -F "image=@hello-world.png" -F "title=Hello World" -F "body=Nothing to read here..." http://example.com/api/v1/post/
<form method="post" action="http://example.com/api/v1/post/" enctype="multipart/form-data">
    <input type="text" name="title">
    <textarea name="body"></textarea>
    <input type="file" name="image>
</form>
@paulbersch

The solution from @amccloud works great for POST, but on PUT or PATCH it fails with the error "You cannot access raw_post_data after reading from request's data stream".

This is because the convert_post_to_VERB function uses _load_post_and_files(), which sets the _read_started flag on the request object. A (probably bad, side-effect-causing) workaround is to set the _read_started flag back to false at the end of the convert_post_to_VERB function.

@amccloud

@paulbersch is right. I ran into this behavior the other day using 1.4.

I had to add to my mixin:

def put_detail(self, request, **kwargs):
    if not hasattr(request, '_body'):
        request._body = ''

    return super(MultipartResource, self).put_detail(request, **kwargs)

I have yet to use PATCH so i'm not sure yet.

@paulbersch

@amccloud That'll work if you're only sending multipart PUT requests, but a PUT in any other format will fail because the request will have an empty body.

For any multipart request, convert_post_to_VERB has a side-effect of preventing anything further down the chain from accessing request.body (formerly request.raw_post_data). However, request._load_post_and_files() will use request._body instead of reading from the file object if it's present, and request.body() sets request._body and then re-creates the file-like object, so we can safely set request._read_started back to false after accessing the request.body property.

This shouldn't have any side-effects.

def convert_post_to_VERB(request, verb):
    """
    Force Django to process the VERB.
    """
    if request.method == verb:
        if hasattr(request, '_post'):
            del(request._post)
            del(request._files)

        # sets request._body
        request.body

        # 'reset' the request object's stream
        # request.body() re-creates the file-like object anyway
        request._read_started = False

        try:
            request.method = "POST"
            request._load_post_and_files()
            request.method = verb
        except AttributeError:
            request.META['REQUEST_METHOD'] = 'POST'
            request._load_post_and_files()
            request.META['REQUEST_METHOD'] = verb
        setattr(request, verb, request.POST)

    return request

See the request.body property in Django's code:
https://github.com/django/django/blob/master/django/http/__init__.py#L283
And the _load_post_and_files function:
https://github.com/django/django/blob/master/django/http/__init__.py#L305

@x1a0

@paulbersch As you said:

That'll work if you're only sending multipart PUT requests, but a PUT in any other format will fail because the request will have an empty body.

Can I modify @amccloud 's codes to be like this:

def put_detail(self, request, **kwargs):
    if request.META.get('CONTENT_TYPE').startswith('multipart') and \
            not hasattr(request, '_body'):
        request._body = ''

    return super(MultipartResource, self).put_detail(request, **kwargs)
@philipn philipn referenced this issue
Closed

File uploads #606

@philipn

I did some work to pull this together into a branch / pull request here: #606

@issackelly
Collaborator

Copied from #606

We had a conversation about this in IRC and we've decided to do the following for file uploads:

  1. Implement a Base64FileField which accepts base64 encoded files (like the one in issue #42) for PUT/POST, and provides the URL for GET requests. This will be part of the main tastypie repo.
  2. We'd like to encourage other implementations to implement as independent projects. There's several ways to do this, and most of them are slightly finicky, and they all have different drawbacks, We'd like to have other options, and document the pros and cons of each
@alej0varas

I've built a base64 multiple form field that allow image upload without changing the content type, just add the field to your validation form. Check it out

https://gist.github.com/alej0varas/5051048

@cellofellow

I updated @klipstein's Base64Field gist to be a little more flexible and more cleanly coded. https://gist.github.com/cellofellow/5493290

@jstarcher

Thanks @cellofellow, I just had to add *args to the init() function to allow the attribute arg.

https://gist.github.com/jstarcher/ef8d91b8e8d058178f20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.