Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 338 #393

Merged
merged 5 commits into from Jul 11, 2011
Merged

Issue 338 #393

merged 5 commits into from Jul 11, 2011

Conversation

scottwb
Copy link
Contributor

@scottwb scottwb commented Jul 11, 2011

I believe this should re-close GH issue #338. To summarize:

  • The previous attempt at fixing this didn't work. It always silently failed on an incorrect relative path to the cache_id dir.
  • The test for that passed because the new delete_cache_id method was stubbed and never actually tested.
  • Fixing that broke anything having to do with a multi-version uploader, because it used FileUtils.rm_rf to delete the entire cache dir, deleting other versions before they were stored.
  • That was fixed by using Dir.rmdir which fails if the directory is not empty.
  • @jnicklas previously brought up the danger of deleting directories when the user could possibly override the cache_dir mechanism. I believe these concerns are addressed because:
    • The cache_dir is never deleted. Only the subdirs underneath it, that are created by CarrierWave, are.
    • Only empty directories are deleted now (with specs to confirm this)

If this gets pulled into master and released with the next gem version, I'd be happy to update the notes in this wiki to reflect the differences in the new version: https://github.com/jnicklas/carrierwave/wiki/How-to:-Delete-cache-garbage-directories

…c so that it covers the method in question instead of stubbing it out.

This is important because this actually makes this test case fail. There is a defect in the implementation of delete_cache_id that was never being covered by the specs because the call to it was stubbed out.
…s to parent" spec to not make assumptions about the order of Hash keys.

This test was breaking intermittently when unrelated tests changed, only by causing the Hash keys to be returned in a different order.
…#delete_cache_id

This fixes the broken spec "CarrierWave::Uploader#store! should delete the old cache_id" that was added in 7f9c35f

This now breaks numerous other specs and support for uploaders with multiple versions, because now that the cache dir is successfully being deleted after the first version is stored, other versions in that cache are being prematurely deleted.
…rage option set to false should not delete the old cache_id" spec more explicitly check for the presences of the directories we want to ensure are not deleted.
bensie added a commit that referenced this pull request Jul 11, 2011
@bensie bensie merged commit b7f0044 into carrierwaveuploader:master Jul 11, 2011
@clyfe
Copy link
Contributor

clyfe commented Jul 11, 2011

Thanks for the fixes and sorry for leaving warts behind me.

I'm new to testing, and my original issue was stubbing methods on globals like Dir, File.
As far as I can tell your test actually creates real files .
Is that considered ok? Where do they get created in test environment? Shouldn't these methods be stubbed?

@scottwb
Copy link
Contributor Author

scottwb commented Jul 11, 2011

@clyfe - The real files were being created during tests before I ever touched them - in spec/public/uploads/tmp. Note that the store_spec.rb cleans up after itself with this:

  after do
    FileUtils.rm_rf(public_path)
  end

I actually prefer it this way. The test code in question is spec'ing the behavior surrounding these files and directories, so in my opinion, it really should be exercising what you expect it to do in the real world as close as is practical.

In my experience very strict unit testing where you stub and mock everything outside your test concern can become a bit cumbersome. You would have had to unit test the delete_cache_id method separately and stub expectations on your calls to File and Dir as you mentioned. I find a happy middle ground between behavior/integration testing and strict unit testing is more practical with rspec, and tools like rcov are great for helping you visualize the coverage you are getting.

Anyway - thanks for carrying the ball most of the way on this. This was a good one to get nailed.

@trevorturk
Copy link
Contributor

Great work, guys. I'm hoping we get a couple more issues cleaned up in the next week or two. I think we should have a quick turnaround for another gem release -- so bug me if you haven't seen any movement after a few weeks.

Deradon added a commit to Deradon/carrierwave that referenced this pull request Aug 16, 2015
== Problem

    Errno::EMLINK: Too many links @ dir_s_mkdir

== Full Backtrace

    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `fu_mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:224 :in `block (2 levels) in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `reverse_each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `block in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `mkdir_p`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:290 :in `mkdir!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:209 :in `copy_to`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:131 :in `block in cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/callbacks.rb:17 :in `with_callbacks`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:122 :in `cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:327 :in `cache`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:179 :in `file=`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/orm/activerecord.rb:38 :in `file=`

== Why

<TODO>

== Reproduce

Let's imagine a rails app with a User#avatar

    require 'tempfile'
    32001.times { User.new avatar: Tempfile.new("foo") }
    # This will create 32001 dirs within /public/uploads/tmp

== Related Pull Requests / Issues / Links

* [How to: Delete cache garbage directories](https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Delete-cache-garbage-directories)
* [Remove the cached file after storing it carrierwaveuploader#107](carrierwaveuploader#107)
* [Remove the cached file after storing it carrierwaveuploader#125](carrierwaveuploader#125)
* [delete cache garbage dirs carrierwaveuploader#338](carrierwaveuploader#338)
* [delete cache garbage dirs issue carrierwaveuploader#338 carrierwaveuploader#346](carrierwaveuploader#346)
* [Issue 338 carrierwaveuploader#393](carrierwaveuploader#393)
* [tmp files not being deleted carrierwaveuploader#1489](carrierwaveuploader#1489)
Deradon added a commit to Deradon/carrierwave that referenced this pull request Aug 17, 2015
== Problem

    Errno::EMLINK: Too many links @ dir_s_mkdir

== Full Backtrace (carrierwave 0.9)

    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `fu_mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:224 :in `block (2 levels) in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `reverse_each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `block in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `mkdir_p`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:290 :in `mkdir!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:209 :in `copy_to`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:131 :in `block in cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/callbacks.rb:17 :in `with_callbacks`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:122 :in `cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:327 :in `cache`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:179 :in `file=`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/orm/activerecord.rb:38 :in `file=`

== Why

Some file systems only allow a limited number of subdirectories.
Ext3 for example only allow ~32k subdirectories.

== Reproduce

Let's imagine a rails app with a User#avatar

    require 'tempfile'
    32001.times { User.new avatar: Tempfile.new("foo") }
    # This will create 32001 dirs within /public/uploads/tmp

== Fix

Clean cache after caching failed.

== Related Pull Requests / Issues / Links

* [How to: Delete cache garbage directories](https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Delete-cache-garbage-directories)
* [Remove the cached file after storing it carrierwaveuploader#107](carrierwaveuploader#107)
* [Remove the cached file after storing it carrierwaveuploader#125](carrierwaveuploader#125)
* [delete cache garbage dirs carrierwaveuploader#338](carrierwaveuploader#338)
* [delete cache garbage dirs issue carrierwaveuploader#338 carrierwaveuploader#346](carrierwaveuploader#346)
* [Issue 338 carrierwaveuploader#393](carrierwaveuploader#393)
* [tmp files not being deleted carrierwaveuploader#1489](carrierwaveuploader#1489)
Deradon added a commit to Deradon/carrierwave that referenced this pull request Nov 26, 2015
== Problem

    Errno::EMLINK: Too many links @ dir_s_mkdir

== Full Backtrace (carrierwave 0.9)

    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `fu_mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:224 :in `block (2 levels) in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `reverse_each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `block in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `mkdir_p`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:290 :in `mkdir!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:209 :in `copy_to`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:131 :in `block in cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/callbacks.rb:17 :in `with_callbacks`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:122 :in `cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:327 :in `cache`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:179 :in `file=`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/orm/activerecord.rb:38 :in `file=`

== Why

Some file systems only allow a limited number of subdirectories.
Ext3 for example only allow ~32k subdirectories.

== Reproduce

Let's imagine a rails app with a User#avatar

    require 'tempfile'
    32001.times { User.new avatar: Tempfile.new("foo") }
    # This will create 32001 dirs within /public/uploads/tmp

== Fix

Clean cache after caching failed.

== Related Pull Requests / Issues / Links

* [How to: Delete cache garbage directories](https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Delete-cache-garbage-directories)
* [Remove the cached file after storing it carrierwaveuploader#107](carrierwaveuploader#107)
* [Remove the cached file after storing it carrierwaveuploader#125](carrierwaveuploader#125)
* [delete cache garbage dirs carrierwaveuploader#338](carrierwaveuploader#338)
* [delete cache garbage dirs issue carrierwaveuploader#338 carrierwaveuploader#346](carrierwaveuploader#346)
* [Issue 338 carrierwaveuploader#393](carrierwaveuploader#393)
* [tmp files not being deleted carrierwaveuploader#1489](carrierwaveuploader#1489)
mynock pushed a commit to mynock/carrierwave that referenced this pull request Aug 9, 2016
== Problem

    Errno::EMLINK: Too many links @ dir_s_mkdir

== Full Backtrace (carrierwave 0.9)

    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:247 :in `fu_mkdir`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:224 :in `block (2 levels) in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `reverse_each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:222 :in `block in mkdir_p`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `each`
    /opt/rubies/ruby-2.1.5/lib/ruby/2.1.0/fileutils.rb:208 :in `mkdir_p`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:290 :in `mkdir!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/sanitized_file.rb:209 :in `copy_to`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:131 :in `block in cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/callbacks.rb:17 :in `with_callbacks`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/uploader/cache.rb:122 :in `cache!`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:327 :in `cache`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/mount.rb:179 :in `file=`
    [GEM_ROOT]/gems/carrierwave-0.9.0/lib/carrierwave/orm/activerecord.rb:38 :in `file=`

== Why

Some file systems only allow a limited number of subdirectories.
Ext3 for example only allow ~32k subdirectories.

== Reproduce

Let's imagine a rails app with a User#avatar

    require 'tempfile'
    32001.times { User.new avatar: Tempfile.new("foo") }
    # This will create 32001 dirs within /public/uploads/tmp

== Fix

Clean cache after caching failed.

== Related Pull Requests / Issues / Links

* [How to: Delete cache garbage directories](https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Delete-cache-garbage-directories)
* [Remove the cached file after storing it carrierwaveuploader#107](carrierwaveuploader#107)
* [Remove the cached file after storing it carrierwaveuploader#125](carrierwaveuploader#125)
* [delete cache garbage dirs carrierwaveuploader#338](carrierwaveuploader#338)
* [delete cache garbage dirs issue carrierwaveuploader#338 carrierwaveuploader#346](carrierwaveuploader#346)
* [Issue 338 carrierwaveuploader#393](carrierwaveuploader#393)
* [tmp files not being deleted carrierwaveuploader#1489](carrierwaveuploader#1489)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants