How to: Make a fast lookup able storage directory structure

kyledecot edited this page Apr 11, 2013 · 6 revisions

Partitioning

If you have many files dumped in the same directory disk lookups might be slow.
That is why it's a good idea to partition your files into directories like in the following example:

class FileUploader < CarrierWave::Uploader::Base
  storage :file
  
  def store_dir
    "documents/#{model.file_name[0, 2]}/#{model.id}"
  end  
end

In this example we use the first two letters of the filename to create directories that will partition our files.
This way we can have nice fast dictionary-like lookups.

See also: http://stackoverflow.com/questions/446358/storing-a-large-number-of-images

Cleanup

It is also a good idea to clean up your empty directories when the files in them are deleted, like in the following example:

class FileUploader < CarrierWave::Uploader::Base
  storage :file
  after :remove, :delete_empty_upstream_dirs

  def store_dir
    "#{base_store_dir}/#{model.id}"
  end
  
  def base_store_dir
    "documents/#{model.file_name[0, 2]}"
  end

  def delete_empty_upstream_dirs
    path = ::File.expand_path(store_dir, root)
    Dir.delete(path) # fails if path not empty dir
    
    path = ::File.expand_path(base_store_dir, root)
    Dir.delete(path) # fails if path not empty dir
  rescue SystemCallError
    true # nothing, the dir is not empty
  end
end
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.