Merge pull request skoczen#1 from chromano/master

Support for backends from chromano

README.md

@@ -1,8 +1,8 @@
 `django-ajax-uploader` provides a useful class you can use to easily implement ajax uploads.
 
-It uses valum's great uploader: https://github.com/valums/file-uploader ,and draws heavy inspiration and some code from https://github.com/alexkuhl/file-uploader
+It uses valum's great uploader: https://github.com/valums/file-uploader , and draws heavy inspiration and some code from https://github.com/alexkuhl/file-uploader
 
-In short, it implements a callable class, `AjaxFileUploader` that you can subclass use to handle uploads.  By default, `AjaxFileUploader` assumes you want to upload to Amazon's S3 (and do so expediently!), but can be subclassed to change this behavior if desired.  Pull requests welcome! 
+In short, it implements a callable class, `AjaxFileUploader` that you can use to handle uploads.  By default, `AjaxFileUploader` assumes you want to upload to Amazon's S3, but you can select any other backend if desired or write your own (see backends section below).  Pull requests welcome! 
 
 Usage
 =====
@@ -74,29 +74,48 @@ This sample is included in the templates directory, but at the minimum, you need
 	</body>
 	</html>
 
-Step 4. Subclass and override if needed.
-----------------------------------------
-That's all you need to get rolling. However, it's likely you actually want to do something with those files the user just uploaded. For that, you can subclass AjaxFileUploader, and override functions and constants.  AjaxFileUploader has a fair bit of configurability.  The example below shows all of the most common functions and constants redefined.
 
-	class MyAjaxFileUploader(AjaxFileUploader):
-		NUM_PARALLEL_PROCESSES = 48   		# Your servers are way better than mine
-	    BUFFER_SIZE = 10485760  # 100MB		# In the future, 10 MB is nothing. 
+Backends
+========
+Backend Selection
+-----------------
 
-	    def _update_filename(self, request, filename):
-	    	# This example timestamps the filename, so we know they're always unique.
-	        import time
-	        return "import/%s.%s" % (int(time.time()), filename,)
+Backends are available in `ajaxuploader.backends`. To select the backend you want to use simply specify the `backend` parameter when instantiating `AjaxFileUploader`. For instance, if you want to the use `LocalUploadBackend` in order to store the uploaded files locally:
 
-	    def _upload_complete(self, request, filename):
-	        print "Save the fact that %s's upload was completed to the database, and do important things!"  % filename
-	        return {}
+views.py
+
+    from ajaxuploader.backends.local import LocalUploadBackend
+
+    ...
+    import_uploader = AjaxFileUploader(backend=LocalUploadBackend)
+
+Each backend has its own configuration. As an example, the `LocalUploadBackend` has the constant `UPLOAD_DIR` which specifies where the files should be stored, based on `MEDIA_ROOT`. By default, the `UPLOAD_DIR` is set to `uploads`, which means the files will be stored at `MEDIA_ROOT/UPLOAD_DIR`. If you want to use an alternative place for storing the files, you need to set a new value for this constant:
+
+    from ajaxuploader.backends.local import LocalUploadBackend
+
+	...
+	LocalUploadBackend.UPLOAD_DIR = "tmp"
+    import_uploader = AjaxFileUploader(backend=LocalUploadBackend)
+
+Similarly, the `ThumbnailUploadBackend` has the constant `DIMENSION`, which determines the dimension of the thumbnail image that will be created. The string format for this constant is the same as for `sorl-thumbnail`.
+
+Backends Available
+------------------
+
+The following backends are available:
+
+* `local.LocalUploadBackend`: Store the file locally. You can specify the directory where files will be saved through the `UPLOAD_DIR` constant. This backend will also include in the response sent to the client a `path` variable with the path in the server where the file can be accessed.
+* `s3.S3UploadBackend`: Store the file in Amazon S3.
+* `thumbnail.ThumbnailUploadBackend`: Depends on `sorl-thumbnail`. Used for images upload that needs re-dimensioning/cropping. Like `LocalUploadBackend`, it includes in the response a `path` variable pointing to the image in the server. The image dimension can be set through `ThumbnailUploadBackend.DIMENSION`, by default it is set to "100x100".
 
-	my_uploader = MyAjaxFileUploader()
+Customization
+-------------
 
+In order to write your custom backend, you need to inherit from `backends.base.AbstractUploadBackend` and implement the `upload_chunk` method, which will receive the string representing a chunk of data that was just read from the client. The following methods are optional and should be implement if you want to take advantage of their purpose:
 
-Advanced Usage / Not uploading to S3
-====================================
-At the moment, ajax-upload is built for s3.  However, you can easily redefine the `_save_upload` method, and save the stream/file wherever you'd like.  Pull requests are welcome for further abstraction.
+* `setup`: given the original filename, do any pre-processing needed before uploading the file (for example, for S3 backend, this method is used to establish a connection with S3 server).
+* `update_filename`: given the `request` object and the original name of the file being updated, returns a new filename which will be used to refer to the file being saved, also this filename will be returned to the client.
+* `upload_complete`: receives the `request` object and the updated filename (as described on `update_filename`) and do any processing needed after upload is complete (like croping the image or disconnecting from the server). If a dict is returned, it is used to update the response returned to the client.
 
 
 Caveats

ajaxuploader/backends/base.py

@@ -0,0 +1,34 @@
+class AbstractUploadBackend(object):
+    BUFFER_SIZE = 10485760  # 10MB
+
+    def setup(self, filename):
+        """Responsible for doing any pre-processing needed before the upload
+        starts."""
+
+    def update_filename(self, request, filename):
+        """Returns a new name for the file being uploaded."""
+
+    def upload_chunk(self, chunk):
+        """Called when a string was read from the client."""
+        raise NotImplementedError
+
+    def upload_complete(self, request, filename):
+        """Overriden to performs any actions needed post-upload, and returns
+        a dict to be added to the render / json context"""
+
+    def upload(self, uploaded, filename, raw_data):
+        try:
+            if raw_data:
+                # File was uploaded via ajax, and is streaming in.
+                chunk = uploaded.read(self.BUFFER_SIZE)
+                while len(chunk) > 0:
+                    self.upload_chunk(chunk)
+                    chunk = uploaded.read(self.BUFFER_SIZE)
+            else:
+                # File was uploaded via a POST, and is here.
+                for chunk in uploaded.chunks():
+                    self.upload_chunk(chunk)
+            return True
+        except:
+            # things went badly.
+            return False

ajaxuploader/backends/local.py

@@ -0,0 +1,26 @@
+from io import FileIO, BufferedWriter
+import os
+from StringIO import StringIO
+
+from django.conf import settings
+
+from ajaxuploader.backends.base import AbstractUploadBackend
+
+class LocalUploadBackend(AbstractUploadBackend):
+    UPLOAD_DIR = "uploads"
+
+    def setup(self, filename):
+        self._path = os.path.join(
+            settings.MEDIA_ROOT, self.UPLOAD_DIR, filename)
+        try:
+            os.makedirs(os.path.realpath(os.path.dirname(self._path)))
+        except:
+            pass
+        self._dest = BufferedWriter(FileIO(self._path, "wb"))
+
+    def upload_chunk(self, chunk):
+        self._dest.write(chunk)
+
+    def upload_complete(self, request, filename):
+        path = settings.MEDIA_URL + self.UPLOAD_DIR + "/" + filename
+        return {"path": path}

ajaxuploader/backends/s3.py

@@ -0,0 +1,32 @@
+from multiprocessing import Pool
+from StringIO import StringIO
+
+import boto
+from django.conf import settings
+
+from ajaxuploader.backends.base import AbstractUploadBackend
+
+class S3UploadBackend(AbstractUploadBackend):
+    NUM_PARALLEL_PROCESSES = 4
+
+    def _upload_chunk(self, chunk):
+        buffer = StringIO()
+        buffer.write(chunk)
+        self._pool.apply_async(
+            self._mp.upload_part_from_file(buffer, self._counter))
+        buffer.close()
+        self._counter += 1
+
+    def setup(self, filename):
+        self._bucket = boto.connect_s3(settings.AWS_ACCESS_KEY_ID,
+                                       settings.AWS_SECRET_ACCESS_KEY)\
+                            .lookup(settings.AWS_BUCKET_NAME)
+        self._mp = self._bucket.initiate_multipart_upload(filename)
+        self._pool = Pool(processes=self.NUM_PARALLEL_PROCESSES)
+        self._counter = 0
+
+    def upload_complete(self, request, filename):
+        # Tie up loose ends, and finish the upload
+        self._pool.close()
+        self._pool.join()
+        self._mp.complete_upload()

ajaxuploader/backends/thumbnail.py

@@ -0,0 +1,16 @@
+import os
+
+from django.conf import settings
+from sorl.thumbnail import get_thumbnail
+
+from ajaxuploader.backends.local import LocalUploadBackend
+
+class ThumbnailUploadBackend(LocalUploadBackend):
+    DIMENSION = "100x100"
+    KEEP_ORIGINAL = False
+
+    def upload_complete(self, request, filename):
+        thumbnail = get_thumbnail(self._path, self.DIMENSION)
+        if not self.KEEP_ORIGINAL:
+            os.unlink(self._path)
+        return {"path": settings.MEDIA_URL + thumbnail.name}

ajaxuploader/templates/django-ajax-uploader/sample.html

@@ -1,36 +1,39 @@
 <!doctype html> 
 <head>
-	<script src="{{STATIC_URL}}django-ajax-uploader/fileuploader.js" ></script>
-	<link href="{{STATIC_URL}}django-ajax-uploader/fileuploader.css" media="screen" rel="stylesheet" type="text/css" />
-	<script>
-		var uploader = new qq.FileUploader( {
-		    action: "/sample-upload/",
-		    element: $('#file-uploader')[0],
-		    multiple: true,
-		    onComplete: function( id, fileName, responseJSON ) {
-		      if( responseJSON.success )
-		        alert( "success!" ) ;
-		      else
-		        alert( "upload failed!" ) ;
-		    },
-		    onAllComplete: function( uploads ) {
-		      // uploads is an array of maps
-		      // the maps look like this: { file: FileObject, response: JSONServerResponse }
-		      alert( "All complete!" ) ;
-		    },
-		    params: {
-		      'csrf_token': '{{ csrf_token }}',
-		      'csrf_name': 'csrfmiddlewaretoken',
-		      'csrf_xname': 'X-CSRFToken',
-		    },
-		  }) ;
-	</script>
+    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" ></script>
+    <script src="{{STATIC_URL}}django-ajax-uploader/fileuploader.js" ></script>            
+    <link href="{{STATIC_URL}}django-ajax-uploader/fileuploader.css" media="screen" rel="stylesheet" type="text/css" />
 </head>
 <body>
-	<div id="file-uploader">       
-	    <noscript>          
-	        <p>Please enable JavaScript to use file uploader.</p>
-	    </noscript>         
-	</div>
+    <div id="file-uploader">       
+        <noscript>          
+            <p>Please enable JavaScript to use file uploader.</p>
+        </noscript>         
+    </div>
+
+    <script>
+        var uploader = new qq.FileUploader( {
+            action: "{% url my_ajax_upload %}",
+            element: $('#file-uploader')[0],
+            multiple: true,
+            onComplete: function( id, fileName, responseJSON ) {
+              if( responseJSON.success )
+                alert( "success!" ) ;
+              else
+                alert( "upload failed!" ) ;
+            },
+            onAllComplete: function( uploads ) {
+              // uploads is an array of maps
+              // the maps look like this: { file: FileObject, response: JSONServerResponse }
+              alert( "All complete!" ) ;
+            },
+            params: {
+              'csrf_token': '{{ csrf_token }}',
+              'csrf_name': 'csrfmiddlewaretoken',
+              'csrf_xname': 'X-CSRFToken',
+            },
+          }) ;
+    </script>    
+
 </body>
 </html>

ajaxuploader/views.py

@@ -1,97 +1,58 @@
-from StringIO import StringIO
-from multiprocessing import Pool
 from django.http import HttpResponse, HttpResponseBadRequest, Http404
-import boto
 import json
-from django.conf import settings
+
+from ajaxuploader.backends.s3 import S3UploadBackend
+
 class AjaxFileUploader(object):
-    NUM_PARALLEL_PROCESSES = 4
-    BUFFER_SIZE = 10485760  # 10MB
+    def __init__(self, backend=None):
+        if backend is None:
+            backend = S3UploadBackend
+        self._backend = backend()
 
     def __call__(self,request):
         return self._ajax_upload(request)
 
-    def _update_filename(self, request, filename):
-        return filename
-
-    def _upload_complete(self, request, filename):
-        """Overriden to performs any actions needed post-upload, and
-            returns a dict to be added to the render / json context"""
-        return {}
-
-    def _upload_chunk(self, pool, mp, chunk, counter):
-        buffer = StringIO()
-        buffer.write(chunk)
-        pool.apply_async(mp.upload_part_from_file(buffer, counter))
-        buffer.close()
-
-    def _save_upload(self, uploaded, filename, raw_data ):
-        try:
-            bucket = boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY).lookup(settings.AWS_BUCKET_NAME)
-            mp = bucket.initiate_multipart_upload(filename)
-            pool = Pool(processes=self.NUM_PARALLEL_PROCESSES)
-            counter = 0        
-
-            if raw_data:
-                # File was uploaded via ajax, and is streaming in.
-                chunk = uploaded.read(self.BUFFER_SIZE)
-                while len(chunk) > 0:
-                    counter += 1
-                    self._upload_chunk(pool, mp, chunk, counter)
-                    chunk = uploaded.read(self.BUFFER_SIZE)
-            else:
-                # File was uploaded via a POST, and is here.
-                for chunk in uploaded.chunks():
-                    counter += 1
-                    self._upload_chunk(pool, mp, chunk, counter)
-
-            # Tie up loose ends, and finish the upload
-            pool.close()
-            pool.join()
-            mp.complete_upload()
-            return True
-
-        except:
-            # things went badly.
-            return False
-
-
     def _ajax_upload(self, request):
         if request.method == "POST":        
             if request.is_ajax():
                 # the file is stored raw in the request
                 upload = request
                 is_raw = True
-                # AJAX Upload will pass the filename in the querystring if it is the "advanced" ajax upload
+                # AJAX Upload will pass the filename in the querystring if it
+                # is the "advanced" ajax upload
                 try:
-                    filename = request.GET[ 'qqfile' ]
+                    filename = request.GET['qqfile']
                 except KeyError: 
-                    return HttpResponseBadRequest( "AJAX request not valid" )
-            # not an ajax upload, so it was the "basic" iframe version with submission via form
+                    return HttpResponseBadRequest("AJAX request not valid")
+            # not an ajax upload, so it was the "basic" iframe version with
+            # submission via form
             else:
                 is_raw = False
-                if len( request.FILES ) == 1:
-                    # FILES is a dictionary in Django but Ajax Upload gives the uploaded file an
-                    # ID based on a random number, so it cannot be guessed here in the code.
-                    # Rather than editing Ajax Upload to pass the ID in the querystring,
-                    # observe that each upload is a separate request,
-                    # so FILES should only have one entry.
-                    # Thus, we can just grab the first (and only) value in the dict.
-                    upload = request.FILES.values()[ 0 ]
+                if len(request.FILES) == 1:
+                    # FILES is a dictionary in Django but Ajax Upload gives
+                    # the uploaded file an ID based on a random number, so it
+                    # cannot be guessed here in the code. Rather than editing
+                    # Ajax Upload to pass the ID in the querystring, observe
+                    # that each upload is a separate request, so FILES should
+                    # only have one entry. Thus, we can just grab the first
+                    # (and only) value in the dict.
+                    upload = request.FILES.values()[0]
                 else:
-                    raise Http404( "Bad Upload" )
+                    raise Http404("Bad Upload")
                 filename = upload.name
-
-            # custom filename handler
-            filename = self._update_filename(request, filename)
 
+            # custom filename handler
+            filename = (self._backend.update_filename(request, filename)
+                        or filename)
             # save the file
-            success = self._save_upload( upload, filename, is_raw )
-
+            self._backend.setup(filename)
+            success = self._backend.upload(upload, filename, is_raw)
             # callback
-            extra_context = self._upload_complete(request, filename)
+            extra_context = self._backend.upload_complete(request, filename)
 
             # let Ajax Upload know whether we saved it or not
-            ret_json = { 'success': success, }
-            ret_json.update(extra_context)
-            return HttpResponse( json.dumps( ret_json ) )
+            ret_json = {'success': success, 'filename': filename}
+            if extra_context is not None:
+                ret_json.update(extra_context)
+
+            return HttpResponse(json.dumps(ret_json))