Close or reuse upload fixture for factories #616

Closed
jsmestad opened this Issue Feb 13, 2012 · 15 comments

Projects

None yet

9 participants

@jsmestad

I am having an issue with having too many open files when running carrierwave in our tests. When using something like Fabricator or FactoryGirl (the former in our case). We end up with thousands of entries of open files (which sometime overflow into a getcwd too many files open error).

Fabricator:

# NOTE this is necessary to prevent thousands of open files to the same image (lsof)
# @profile_image ||= Rack::Test::UploadedFile.new("#{Rails.root}/spec/fixtures/files/rails.png", 'image/png', true)

Fabricator :user do
  name { Faker::Name.name }
  email { Faker::Internet.email }
  picture { Rack::Test::UploadedFile.new("#{Rails.root}/spec/fixtures/files/rails.png", 'image/png', true) }
end

NOTE I have tried setting picture { @profile_image } to get it to reuse the same uploaded Tempfile, but that does not seem to work (it effectively sets the picture to nil with all sub-versions also set to nil) even when setting config.delete_tmp_file_after_storage = false.

lsof | grep ruby output:

...
ruby      19517 justinsmestad   55u     REG               14,2      87443 2921382 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-r22z3pruby      19517 justinsmestad   56u     REG               14,2      87443 2921392 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-sxde2truby      19517 justinsmestad   57u     REG               14,2      87443 2921397 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-wxqngi
ruby      19517 justinsmestad   58u     REG               14,2      87443 2921407 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1mp68wa
ruby      19517 justinsmestad   59u     REG               14,2      87443 2921412 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1xgzgv7
ruby      19517 justinsmestad   60u     REG               14,2      87443 2921422 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-gp1i6y
ruby      19517 justinsmestad   61u     REG               14,2      87443 2921427 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-pea10j
ruby      19517 justinsmestad   62u     REG               14,2      87443 2921437 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-l8knkpruby      19517 justinsmestad   63u     REG               14,2      87443 2921442 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1cetyyh
ruby      19517 justinsmestad   64u     REG               14,2      87443 2921452 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-ts4w0z
ruby      19517 justinsmestad   65u     REG               14,2      87443 2921457 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-460di8
ruby      19517 justinsmestad   66u     REG               14,2      87443 2921467 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-b1lk2y
ruby      19517 justinsmestad   67u     REG               14,2      87443 2921472 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-mb1xz1
ruby      19517 justinsmestad   68u     REG               14,2      87443 2921482 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1fggsgx
ruby      19517 justinsmestad   69u     REG               14,2      87443 2921487 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-61en33
ruby      19517 justinsmestad   70u     REG               14,2      87443 2921497 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1o2hfb9
ruby      19517 justinsmestad   71u     REG               14,2      87443 2921502 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1mjnlgn
ruby      19517 justinsmestad   72u     REG               14,2      87443 2921512 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-mo4fak
ruby      19517 justinsmestad   73u     REG               14,2      87443 2921517 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-oyzlj3
ruby      19517 justinsmestad   74u     REG               14,2      87443 2921527 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-16re0wt
ruby      19517 justinsmestad   75u     REG               14,2      87443 2921532 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1qluw2e
ruby      19517 justinsmestad   76u     REG               14,2      87443 2921542 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-138gerk
ruby      19517 justinsmestad   77u     REG               14,2      87443 2921547 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1yk7dcy
ruby      19517 justinsmestad   78u     REG               14,2      87443 2921557 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-caisea
ruby      19517 justinsmestad   79u     REG               14,2      87443 2921562 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-dgroqf
ruby      19517 justinsmestad   80u     REG               14,2      87443 2921572 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1372klb
ruby      19517 justinsmestad   81u     REG               14,2      87443 2921577 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-pikg0q
ruby      19517 justinsmestad   82u     REG               14,2      87443 2921587 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1fvyr9f
ruby      19517 justinsmestad   83u     REG               14,2      87443 2921592 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1khv7lt
ruby      19517 justinsmestad   84u     REG               14,2      87443 2921602 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-q3fbon
ruby      19517 justinsmestad   85u     REG               14,2      87443 2921607 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1ng3953
ruby      19517 justinsmestad   86u     REG               14,2      87443 2921617 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-kte79t
ruby      19517 justinsmestad   87u     REG               14,2      87443 2921622 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-a2zg02
ruby      19517 justinsmestad   88u     REG               14,2      87443 2921632 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1tsvrlx
ruby      19517 justinsmestad   89u     REG               14,2      87443 2921637 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-9vcejh
ruby      19517 justinsmestad   90u     REG               14,2      87443 2921647 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-og4jsc
ruby      19517 justinsmestad   91u     REG               14,2      87443 2921652 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1sl42fz
ruby      19517 justinsmestad   92u     REG               14,2      87443 2921662 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-krpdim
ruby      19517 justinsmestad   93u     REG               14,2      87443 2921667 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-tlvyn9
ruby      19517 justinsmestad   94u     REG               14,2      87443 2921677 /private/var/folders/6r/69zwcwrj3pl6hvqc1jf_zzy00000gn/T/rails.png20120212-19517-1oo7slw

I have also posted this issue over on the Fabrication gem GH issue tracker /cc @paulelliott

@bensie
Member
bensie commented Feb 15, 2012

Is this fixed by #612?

@jsmestad

@bensie it seems that you have git locked to activemodel >= 3.2.0, could you losen this dep so I can try it on Rails 3.1?


This probably will not change anything. I am using :file store in test mode. But seems like the same issue with the file not getting closed after reading.

@bensie
Member
bensie commented Feb 15, 2012

The next release (0.6.0) will support Rails 3.2 and later only. ActiveModel shouldn't be a dependency at all (I see it's in the latest gemspec but that's an error on my part), but ActiveSupport must be 3.2 or greater.

@jsmestad

@bensie this doesnt seem to fix the issue. Can you look at closing the file after store is called (in the file adapter) ? SanatizedFile does not have access to file.close it looks like.

@ghost
ghost commented Mar 30, 2012

I have the same issue when running a rake task to migrate a lot of media using carrierwave. Seems like all the files remain open until my mac pukes. I'm using the remote_attr_url feature to migrate the media from one server to a carrierwave-backed solution.

@avishai
avishai commented May 25, 2012

I've been experiencing the same issue in production when trying to process thousands of images, using Fog as the storage backend for S3. Eventually the resizing job just explodes and raises errors about too many open files.

@tonycoco
Contributor

Going to try the 0.5-stable git branch (which has the patch from #612) on my project and test... I'll confirm later tonight.

@trevorturk
Contributor

Any updates on this?

@tonycoco
Contributor

Moving to the stable branch fixed my issues. My guess is they need to do the same.

@trevorturk
Contributor

#word

@trevorturk trevorturk closed this Jun 14, 2012
@chrisnicola

I still am seeing this issue 0.6.2 using remote_{uploader}_url in a Resque task. After about 100 files it crashes with that message.

@cmer
cmer commented Oct 8, 2012

Having the same problem on 0.6.2

@chrisnicola

So I believe my problem was related to running the New Relic gem, confirmed by New Relic support. They have some sort of bug, generally I only saw the problem on my development environment.

@elthariel

I have this problem although i doesn't use NewRelic (rails 4)

@saneshark

+1

Having the same problem. It only became an issue after upgrading to 1.9.3. When the file is opened, possibly consider opening it within its own block, so that when that block is ended, garbage collector can destroy it appropriately?

File.open(image.path) { |f| f.read } vs File.open(image.path).read

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment