Too many open files

[npadgett]

I am trying to process a list of 30,000 thumbnails. After 250 or so, I receive the error below.

Too many open files - identify -format %wx%h '/var/folders/s5/g2qlm5gs1nd2v10gkvqlsjww0000gn/T/d41d8cd98f00b204e9800998ecf8427e20120821-51888-uvsjks.jpg[0]'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/core_ext/kernel/agnostics.rb:7:in ' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/core_ext/kernel/agnostics.rb:7:in'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/cocaine-0.2.1/lib/cocaine/command_line.rb:30:in block in run' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/cocaine-0.2.1/lib/cocaine/command_line.rb:51:inwith_modified_path'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/cocaine-0.2.1/lib/cocaine/command_line.rb:28:in run' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/helpers.rb:29:inrun'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/geometry.rb:23:in block in from_file' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/core_ext/kernel/reporting.rb:43:insilence_stream'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/geometry.rb:22:in from_file' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/thumbnail.rb:35:ininitialize'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/processor.rb:33:in new' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/processor.rb:33:inmake'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:410:in block in post_process_style' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:409:ineach'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:409:in inject' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:409:inpost_process_style'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:402:in block in post_process_styles' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:401:ineach'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:401:in post_process_styles' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:394:inblock (2 levels) in post_process'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/callbacks.rb:414:in _run_thumbnail_post_process_callbacks' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/callbacks.rb:94:inrun_callbacks'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/callbacks.rb:26:in run_paperclip_callbacks' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:393:inblock in post_process'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/callbacks.rb:414:in _run_post_process_callbacks' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/activesupport-3.0.16/lib/active_support/callbacks.rb:94:inrun_callbacks'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/callbacks.rb:26:in run_paperclip_callbacks' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:392:inpost_process'
/Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:107:in assign' /Users/npadgett/.rvm/gems/ruby-1.9.2-p290@playon-video-portal/gems/paperclip-3.1.4/lib/paperclip/attachment.rb:296:inreprocess!'

[kaluznyo]

Same issue. With paperclip 3.1.4. I double check if I call file.close for each file.new .. So it's maybe on paperclip side ?

[calicoder]

I'm having the same issue.

[dmitry]

Have the same issue: Too many open files - identify -format %wx%h '/tmp/open-uri20121013-32567-f18ndb20121013-32567-rblakg[0]' (Errno::EMFILE)

[holetse]

I am also having this issue. I am reprocessing a few thousand thumbnails via:

ImageAttachment.find_each do |image|
image.file.reprocess!
end

The files are stored in S3.

[ryanstout]

Same issue here, anyone found a good fix for this? Is it in imagemagick or paperclip?

[holetse]

In my case it was an issue with the s3 interface, where a new connection pool was created for each attachment, instead of being shared across all of them. I worked around it with:

image.file.s3_interface.config.http_handler.pool.empty! if image.file.respond_to? :s3_interface

With that fix, the number of open sockets (files) was kept constant +-10.

The code in its entirety:

ImageAttachment.find_each do |image|
  image.file.reprocess!
  image.file.s3_interface.config.http_handler.pool.empty! if image.file.respond_to? :s3_interface
end

[ryanstout]

Thanks for the help. For some reason that didn't fix it for me. I'm trying to reprocess in threads, so maybe thats related.

[jyurek]

What version is everyone using? This was reported with 3.1.4, but does it still happen in 3.3.1?

[ryanstout]

I had updated to see if that fixed it, but its still happening on version 3.3.1. Thanks

[kaluznyo]

@holetse Your code didn't fix the prob for me... On paperclip 3.3.1...

[kaluznyo]

If I check the file open by ruby process while my script (lsof -p PID) I get lot of MYSERVER:35462->s3-3-w.amazonaws.com:https (CLOSE_WAIT)
and "can't identify protocol" (It's opening socket). So connection is like a file...

If it's too many connection open, the bug is for AWS gem ?

[holetse]

@kaluznyo That's the issue as I saw it, lots of open S3 connections (via lsof). In my case I traced it to a "shared" connection pool that was actually not shared at all (each Attachment instance got it's own). Since the pool was meant to be shared, the AWS gem doesn't close the connections in it unless explicitly told to. Paperclip sticks the pool in the configuration object to be shared across all the instances, but that configuration object is copied into each instance, resulting in the pool being duplicated.

And my code was for the issue as reported in 3.1.4. Sounds like things might be different in 3.3.1, I haven't had time to check.

[kaluznyo]

Ok... but I can't rollback to 3.1.4... Any chance to try to fixe it, @thoughtbot ?

[jyurek]

I'm looking at this now. So, @holetse, are you saying that we should simply not be calling AWS::S3.new each time, or is there something else we need to do to take advantage of the connection pool?

[holetse]

@jyurek Yes, I believe calling AWS::S3.new for each Paperclip::Attachment is the source of the problem. To use the connection pooling, a single instance of AWS::S3 needs to be used across all Paperclip::Attachment's that have the same S3 details. The fact that S3 credentials can be different for different attachments complicates things however. If Paperclip wants to use the connection pooling, it needs to reuse the same AWS::S3 instance for each attachment instance that uses the same @s3_options.

Another option would be to set the connection pool idle timeout in AWS:S3 to zero seconds, which would result in the connection always being discarded after use. The default is 60 seconds, which means that you only see open file issues if you make a lot of S3 attachment requests within 60 seconds. I think thats why people only see this when doing batch jobs.

[holetse]

It's worth mentioning that some people may hit the "open files" error even with these changes. If you are running your app close to your ulimit, there is nothing Paperclip can do. You need to either increase the ulimit or reduce your open file descriptor count in those cases.

[kaluznyo]

I set the connection pool idle timout in AWS:S3 to zero (https://gist.github.com/4202216) . But same error... I get 1000 connections in WAIT_CLOSE, The connection seems to never close

[holetse]

@kaluznyo Sorry, my mistake. It looks like AWS::S3 will only check that timeout if the pool gets used again, so I guess that isn't a valid solution. Getting the pool to be shared correctly (or manually emptying it) is the solution.

[jyurek]

I'm trying out a solution right now where I only make a new S3 object if I haven't already created one for this set of options in a given thread. I'll have this pushed to a branch shortly so you guys can try it out (as I don't have a failing scenario locally).

[kaluznyo]

Okay, just ready to test ;)

[jyurek]

OK it's up as the reuse-s3-connection branch. Please give that a try.

[kaluznyo]

That fix the bug ! My code have still somes monkey patch / workaround given in this thread. But I'll delete this part of my code, re-test. And I'll re-confirm to you

[jyurek]

That's great news! Thanks for testing for me.

[kaluznyo]

Work fine with no workaround ! 2 test with several images (5000 images). No more CLOSE_WAIT connexion, and no more "too many open files" !

Thanks @jyurek and @holetse !

[jyurek]

Ok, merged that into master and pushed it up. Thanks for being patient, all, and thanks for the help confirming it works, @kaluznyo.

If this comes back up, please start a new Issue, so we can keep discussion focused since it should hopefully be a different cause. :)