Skip to content

Commit

Permalink
Store::File#cache! will retry after Errno::EMLINK
Browse files Browse the repository at this point in the history
== Problem

    Errno::EMLINK: Too many links @ dir_s_mkdir

== Full Backtrace (carrierwave 0.9)

    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `fu_mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:224 :in `block (2 levels) in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `reverse_each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `block in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `mkdir_p`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:290 :in `mkdir!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:209 :in `copy_to`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:131 :in `block in cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/callbacks.rb:17 :in `with_callbacks`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:122 :in `cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:327 :in `cache`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:179 :in `file=`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/orm/activerecord.rb:38 :in `file=`

== Why

Some file systems only allow a limited number of subdirectories.
Ext3 for example only allow ~32k subdirectories.

== Reproduce

Let's imagine a rails app with a User#avatar

    require 'tempfile'
    32001.times { User.new avatar: Tempfile.new("foo") }
    # This will create 32001 dirs within /public/uploads/tmp

== Fix

Clean cache after caching failed.

== Related Pull Requests / Issues / Links

* [How to: Delete cache garbage directories](https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Delete-cache-garbage-directories)
* [Remove the cached file after storing it #107](#107)
* [Remove the cached file after storing it #125](#125)
* [delete cache garbage dirs #338](#338)
* [delete cache garbage dirs issue #338 #346](#346)
* [Issue 338 #393](#393)
* [tmp files not being deleted #1489](#1489)
  • Loading branch information
Deradon committed Nov 26, 2015
1 parent 4644ade commit d74ee6b
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 0 deletions.
8 changes: 8 additions & 0 deletions lib/carrierwave/storage/file.rb
Expand Up @@ -64,6 +64,14 @@ def retrieve!(identifier)
#
def cache!(new_file)
new_file.move_to(::File.expand_path(uploader.cache_path, uploader.root), uploader.permissions, uploader.directory_permissions, true)
rescue Errno::EMLINK => e
raise(e) if @cache_called
@cache_called = true

# NOTE: Remove cached files older than 10 minutes
clean_cache!(600)

cache!(new_file)
end

##
Expand Down
17 changes: 17 additions & 0 deletions spec/storage/file_spec.rb
@@ -1,8 +1,16 @@
# encoding: utf-8

require 'spec_helper'
require 'support/file_utils_helper'
require 'tempfile'

describe CarrierWave::Storage::File do
include FileUtilsHelper

subject(:storage) { described_class.new(@uploader) }

let(:tempfile) { Tempfile.new("foo") }
let(:sanitized_temp_file) { CarrierWave::SanitizedFile.new(tempfile) }

before do
@uploader_class = Class.new(CarrierWave::Uploader::Base)
Expand Down Expand Up @@ -39,6 +47,15 @@
end
end

describe '#cache!' do
context "when FileUtils.mkdir_p raises Errno::EMLINK" do
before { fake_failed_mkdir_p }
after { storage.cache!(sanitized_temp_file) }

it { is_expected.to receive(:clean_cache!).with(600) }
end
end

describe '#clean_cache!' do
before do
five_days_ago_int = 1369894322
Expand Down
15 changes: 15 additions & 0 deletions spec/support/file_utils_helper.rb
@@ -0,0 +1,15 @@
module FileUtilsHelper
# NOTE: Make FileUtils.mkdir_p to raise `Errno::EMLINK` only once
def fake_failed_mkdir_p
original_mkdir_p = FileUtils.method(:mkdir_p)
mkdir_p_called = false
allow(FileUtils).to receive(:mkdir_p) do |args|
if mkdir_p_called
original_mkdir_p.call(*args)
else
mkdir_p_called = true
raise Errno::EMLINK
end
end
end
end

0 comments on commit d74ee6b

Please sign in to comment.