What is Num-fu?

Num-fu is a plugin built for Woobius, which handles uploads coming via nginx’s upload module. More info on that upload module here and here.

It’s worth noting that this is not a straightforward adaptation of AttachmentFu to the Nginx Upload Module. I’ve picked out the functionality that we are actually using, and much of AttachmentFu has been left aside. This plugin has perhaps 5% of the functionality of Attachment-Fu, and no intention to build in things like image processing, S3 support, thumbnails, etc.

Can I use this?

Yes, go for it. You may fork it and/or use and abuse it in any way you like.

How can I use this?

Usage:

Your model should look something like this:

class FileReference < ActiveRecord::Base
	has_attachment	:max_size     => 2.gigabytes,
                  	:path_prefix  => STORAGE_FOLDER
end

In your controller, add an extract_upload call before processing the upload, to ensure that your code works whether or not nginx is active:

class MyController < ApplicationController
	include NumFu::UploadExtractor

	def upload
	    extract_upload  :file_reference
		FileReference.create(params[:file_reference])
	end
end

And add the following to your nginx config:

location /upload {
	
	client_max_body_size 2048m;
	
	# Pass altered request body to this location
	upload_pass /process_upload;
	
	# Store files to this directory
	upload_store /tmp/uploads;
	
	# Allow uploaded files to be read by everyone
	upload_store_access user:rw group:rw all:rw;
	
	# Set specified fields in request body
	# this puts the orginal filename, new path+filename and content type in the requests params
	upload_set_form_field $upload_field_name.name "$upload_file_name";
	upload_set_form_field $upload_field_name.content_type "$upload_content_type";
	upload_set_form_field $upload_field_name.path "$upload_tmp_path";

	upload_pass_form_field "^_session_id$";
	upload_pass_form_field "^folder\[id\]$";
	
	break;
}

# dummy location that needs to be defined. :-(
location /process_upload {
	client_max_body_size 2048m;
	proxy_pass http://upstream_server;
	break;
}

Worth noting:

• You have to explicitly tell nginx what fields to pass, if there are additional fields, on top of the file itself (and there likely will be things like session id)
• The location that you post to has to be of a direct route, off root – not /controller/action
• Make sure you include the client_max_body_size and the breaks, or you may hit problems
• To pass railsy parameters within parameters (e.g. folder[id]), don’t forget to escape the [] in the regular expression.

Schema setup

You will need the following fields set up on your FileReference schema:

t.string   "filename"
t.string   "content_type"
t.integer  "size"
t.datetime "created_at"
t.datetime "updated_at"
t.string   "original_filename"
t.datetime "deleted_at"

How are the files stored?

Num-Fu stores the files in whatever directory you pass as the path_prefix option. It organises them in subdirectories so that there are no more than 256 files per subdirectory. The folder structure is capable of supporting up to 10000 * 10000 * 256 = 25.6 billion files, via a structure like:

	0001/0001/c35e6f0358ce7390-IMG_A.JPG
	0001/0025/d0a706593750e972-IMG_B.JPG
	0123/0123/dcc3b7d5fc5e146f-IMG_C.JPG

The folder structure will start at 0001/0001 and fill out all the way until 9999/9999. I have not tested what happens when you exceed that limit… if you are storing 25 billion files, however, I would suggest you look into writing your own storage handling code (perhaps S3 storage might be more appropriate than filesystem storage).

The 16-alphanumeric-digit random string prepended to each filename is there to ensure, within reason, that there are no collisions between files with the same name. It is technically possible that two files uploaded with the same filename around the same time will end up in the same directory and overwrite each other. However, based on the usage patterns that we are seeing on Woobius, it’s probably more productive to worry about the Earth exploding spontaneously.