Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A lot of temp files which are not explicitly unlinked #1326

Closed
svetlins opened this issue Aug 29, 2013 · 37 comments

Comments

@svetlins
Copy link

@svetlins svetlins commented Aug 29, 2013

Hello,

On resizing uploaded images Paperclip creates a lot of Tempfiles (e.g. 6 styles = 13 Tempfiles) which are then not explicitly unlinked. On most Ruby implementations this is OK since the actual files are deleted when the Tempfiles instances are garbage collected. This is a problem though with JRuby since version 1.7.4 in which the behavior is changed so that files corresponding to Tempfiles are only deleted on process exit. Because of this our app started filling up /tmp since the upgrade to JRuby 1.7.4.

I would write an issue to JRuby about that but this piece of Ruby documentation discouraged me: http://www.ruby-doc.org/stdlib-1.9.3/libdoc/tempfile/rdoc/Tempfile.html#label-Explicit+close

It basically says that both deleting behaviors are allowed and that it's good practice to explicitly unlink created temp files anyway.

Do you think it's possible to add explicit unlinking of all created temp files to Paperclip? That would really help users of latest JRubies!

Best,
Svetlin

@nilbus

This comment has been minimized.

Copy link

@nilbus nilbus commented Aug 29, 2013

👍 Yes, same here.

@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Dec 4, 2013

+1 This also happens with normal ruby if the process runs long enough. Why GC isn't clearing them out I don't know, but I've hit this a couple of times just recently doing nothing but adding images using StringIO like:

StringIO.open(image_data) do |s|
   c.photo = s
   c.photo_file_name = "original#{extension}"
   c.photo_content_type = image.headers['content-type']
 end
@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Dec 4, 2013

From what I can tell, the problem seems to be with FileAdapter, #post_process_style, and thumbnail generation. post_process_style generates some temporary files and puts them in @queued_for_write, but then right after that that hash is updated with FileAdapters. Later one when #after_flush_writes tries to cleanup, there is no references to that initial temporary file, so it sticks around.

I'm not familiar enough with the rest of the code to know if it's as simple as removing that first temp file after updating @queued_for_write.

@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Dec 4, 2013

@jyurek

This comment has been minimized.

Copy link
Contributor

@jyurek jyurek commented Dec 4, 2013

The problem of open files was a major thorn in my side for a few weeks and I attempted to get the problem down a bit. I don't know if the fix I made in master would help general purpose, but it helped a lot in the tests. Would one of you be able to try to replicate the problem with that commit added?

@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Dec 4, 2013

@jyurek Will do. Initial tests seem good. I just kicked off a job that will chew through about 5,000 images in a single process. I'll let you know how it goes.

@svetlins

This comment has been minimized.

Copy link
Author

@svetlins svetlins commented Dec 5, 2013

This commit wouldn't solve the problem we had as it doesn't unlink the temporary files, only closes them. I added some monkey patches to Tempfile and File to track when and where tempfiles are created and unlinked and it seems they are not unlinked anywhere explicitly. I experimented with adding a similar line of code that also unlinks the unadapted files:

unadapted_file.unlink if unadapted_file.respond_to?(:unlink)

but this only unlinked some portion of the created files - the ones created from the Tempfile.new on lib/paperclip/thumbnail.rb:67. The rest are created from lib/paperclip/tempfile_factory.rb:8 and I don't know where any unlinking code for them might be put.

The code I used to track tempfiles creation and deletion is here: https://gist.github.com/svetlins/cedc5b9235e33c35fcd2

Meanwhile the JRuby team has fixed the Tempfile behaviour, so on newer versions(>= 1.7.6) Tempfiles get deleted on garbage collection, so that's not a big issue for us anymore. With that in mind I still believe it's a good idea to close and unlink all created tempfiles.

@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Dec 5, 2013

@jyurek the master branch has solved my issue. I put lsof in a loop while it ran over several hours and temp files did not accumulate.

@jyurek

This comment has been minimized.

Copy link
Contributor

@jyurek jyurek commented Dec 11, 2013

OK so open files are (by and large) fixed, but we still have a problem with them being unlinked. I mean, really, they weren't unlinked because that was part of the benefit of using Tempfiles -- they just go away on GC. Now that this is on the radar, it's something I'll have to watch out for. Thanks, @svetlins, for that monkeypatch. That will make finding where they're created and not unlinked easier.

@phallstrom

This comment has been minimized.

Copy link

@phallstrom phallstrom commented Jan 2, 2014

@jyurek Bad news... another run (weeks later) of the same script resulted in the too many files open error. So not totally fixed.

@maclover7

This comment has been minimized.

Copy link
Contributor

@maclover7 maclover7 commented Mar 9, 2015

Hi everybody! Is this still an issue for you in Paperclip; I know this issue is from approximately 2 years ago. If it is still an issue, can you please provide the code that's causing you the error? Thanks!

@scomma

This comment has been minimized.

Copy link

@scomma scomma commented Mar 25, 2015

This is still an issue for me. We use paperclip to store the attachments on S3, and the task that saves to S3 runs in a Resque job (which naturally executes in its own fork). Could this be a reason the garbage collector isn't kicking in?

@topherhunt

This comment has been minimized.

Copy link

@topherhunt topherhunt commented Apr 20, 2015

Looks like it's an issue for me too (Paperclip 4.2.0); batch Paperclip operations tend to crash at the following error:

An Errno::EMFILE occurred in background at 2015-04-19 22:25:02 -0700 :

 Too many open files - identify -format '%wx%h,%[exif:orientation]' '/tmp/6f53a5e7b68ac7105b51677c0bc3426320150420-23382-17zy612.jpg[0]' 2>/dev/null

...which traces back to Paperclip and is discussed more in this SO question.

@bluta

This comment has been minimized.

Copy link

@bluta bluta commented May 8, 2015

also for me, (Paperclip 4.2.0, S3, uploading in Resque)
shame on me, I've only realized when it made our EngineYard util instance unusable - filling the 16G /tmp on the root partition.

@jstorimer

This comment has been minimized.

Copy link

@jstorimer jstorimer commented Jul 3, 2015

I'm experiencing this issue on one of my apps as well. My situation is the exact same as @scomma above:

We use paperclip to store the attachments on S3, and the task that saves to S3 runs in a Resque job

We also process some paperclip attachments on our app servers and do not have a problem with the /tmp directory growing on those boxes.

@jyurek mentioned the usage of Tempfile:

that was part of the benefit of using Tempfiles -- they just go away on GC

but @scomma got the problem with that exactly right in his comment:

the task that saves to S3 runs in a Resque job (which naturally executes in its own fork). Could this be a reason the garbage collector isn't kicking in?

A resque worker will fork a new Ruby process to run each job and will, (by default), end that process with exit!. According to the docs on that method, using the bang version means that the process exits immediately. It won't run at_exit handlers and won't run GC! If the GC doesn't run then any instances of Tempfile that had yet to be GC'd will be orphaned and never unlinked from the system.

This can be verified with an irb session:

$ irb -rtempfile
> tf = Tempfile.new('test')
=> #<Tempfile:/tmp/test20150703-19388-jas6b5>
> File.exist?(tf)
=> true
> exit!
$ ls /tmp/test20150703-19388-jas6b5 && echo yikes
yikes

If you run the same session, but use exit instead of exit!, then the tempfile will be properly cleaned up with a GC run.

This is a known issue with resque/resque#1167. The suggested fix there is to set ENV['RUN_AT_EXIT_HOOKS'] so that exit is used over exit!.

My leftover tempfile issue was reproducible in my dev environment and this resque config change did fix the issue for me 👍

@Dagnan

This comment has been minimized.

Copy link

@Dagnan Dagnan commented Nov 20, 2015

Any way to prevent tempfile creation (in my case I'm uploading an already existing file on the system, without any resize because it's an audio file), in a long running task.

@jeremywadsack

This comment has been minimized.

Copy link
Contributor

@jeremywadsack jeremywadsack commented Mar 16, 2016

Thanks to @jstorimer for the analysis. We have the same problem as described (resque jobs) and changing the ~/.bashrc on our worker servers to include this line resolved it:

export RUN_AT_EXIT_HOOKS = true

(Yes, that's a pretty broad way to fix this.)

It would be nice if paperclip unlinked the tempfiles though, as this continues to be reported.

@erkki

This comment has been minimized.

Copy link
Contributor

@erkki erkki commented Mar 23, 2016

If anyone's up for testing #2143 to see if it helps with any of their issues, much appreciated. Warning: not 100% confident this didn't produce new issues so be careful in production :)

tute added a commit that referenced this issue Aug 19, 2016
#2120)

Paperclip duplicates the original files as part of its validation process. (#1642, #1326).

When uploading large files (several hundred megabytes to gigabyte range), this becomes a problem: The web server will be busy creating 3 - 4 duplicates on disk, while the app (and potentially the user) are waiting for the upload operation to complete.

This commit introduces hard links instead of `FileUtil.cp` where possible to keep the logic as-is but save time and disk space.
tute added a commit that referenced this issue Aug 28, 2016
#2290)

Rebased #2120 to master.
Paperclip duplicates the original files quite a lot as part of its validation process. (#1642, #1326).
When uploading large files (several hundred megabytes to gigabyte range), this becomes a problem: The web server will be busy creating 3 - 4 duplicates on disk, while the app (and potentially the user) are waiting for the upload operation to complete.
This pull request introduces hard links instead of ```FileUtil.cp``` where possible to keep the logic as-is but save time and disk space.
@NeoElit

This comment has been minimized.

Copy link

@NeoElit NeoElit commented Jan 13, 2017

Hi, I'm using paperclip (4.3.1). I'm experiencing similar a issues where around 3 tempfiles per attachment is left hanging in the /tmp till passenger kill the process. Is there any hack to avoid this issue?

@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented Jan 16, 2017

@NeoElit I think there is one - upgrade till the 5.x :)

@NeoElit

This comment has been minimized.

Copy link

@NeoElit NeoElit commented Jan 17, 2017

@dmitry Thanks.. will try that out.

@germanotm

This comment has been minimized.

Copy link

@germanotm germanotm commented Feb 10, 2017

@NeoElit
I'm having problem with temp files that are not unlinked for every paperclip conversion using paperclip 4.3.5.
I just upgrade to paperclip 5.1.0 and get the same behavior. Simple upgrade will not solve the problem.

@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented Feb 10, 2017

@germanotm how do you run paperclip conversion, in a rake task or resque/sidekiq/activejob?

@barbolo

This comment has been minimized.

Copy link

@barbolo barbolo commented Feb 14, 2017

I'm using Paperclip 5.1.0 and I'm also having issues with "too many open files". Not sure if Paperclip is the problem, but I'm really suspecting about it.

I use Sidekiq 4.2.9, but Paperclip doesn't run as a background job. It runs during a request to our web server (running Puma 3.7.0).

I created a controller action that shows me all files supposedly known by the GC. It does something like this:

def garbage_collect
  out = []
  ObjectSpace.each_object(File) do |f|
    out << [f.class, f.path, f.closed?].join(": ")
  end
  ObjectSpace.each_object(Tempfile) do |f|
    out << [f.class, f.path, f.closed?].join(": ")
  end
  if params[:perform].present?
    ObjectSpace.garbage_collect
  end
  render text: out.join("\n")
end

When I call this action I get a huge list of files created by Paperclip saved in /tmp/. Some of them are open, some are closed.

Then, when I pass the parameter perform, the Garbage Collect is performed and after that the list of files decreases to 2 ou 3 files.

I'm trying to investigate it more deeply. But for now it seems Paperclip may be the problem.

@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented Feb 14, 2017

@barbolo have you tried to use paperclip alone, in a ruby script and reproduct the same issue?

@barbolo

This comment has been minimized.

Copy link

@barbolo barbolo commented Feb 14, 2017

@dmitry No, I haven't. It wouldn't be so easy to do that.

This is an output from the action above. All files are generated by Paperclip.

File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1xtax9f.html: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-375ef5.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1sgu6ue.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1h1zeqp.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-7iva0e.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-ivdfwx.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-81a8pt.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1vaq545.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1dppyt5.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-15w6ay5.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-ttgpea.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-4u9b0.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-h0us0y.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-d1j11d.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1b55i5g.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-uwq141.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-g33p4w.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-g33p4w.html: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-115wqjl.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-115wqjl.html: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-mm91iz.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-m88a4o.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1botcaq.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1rckg1u.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-4olj6k.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-14j3qmw.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-pjvxk5.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1v52u3s.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-194lwc5.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-194lwc5.pdf: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-17hysha.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-8g4bcw.pdf: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-17hysha.pdf: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-wd49i.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-wd49i.html: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1xrupnr.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1xrupnr.html: false
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-16qm1a2.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1sscg6f.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-11jms9p.html: true
File: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-vjv1r4.html: true
Paperclip::Tempfile: : true
Paperclip::Tempfile: : true
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-g33p4w.html: false
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-115wqjl.html: false
Paperclip::Tempfile: : true
Paperclip::Tempfile: : true
Paperclip::Tempfile: : true
Tempfile: : true
Paperclip::Tempfile: : true
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-194lwc5.pdf: false
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-17hysha.pdf: false
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-wd49i.html: false
Paperclip::Tempfile: /tmp/9a0364b9e99bb480dd25e1f0284c855520170214-13804-1xrupnr.html: false
@barbolo

This comment has been minimized.

Copy link

@barbolo barbolo commented Feb 14, 2017

Analyzing logs like the one in the previous message, I can't find any problem. I'm running a server with this application for the last 3 days and about 300 files on /tmp were never removed after being used by Paperclip.

I've run several server (puma) hot restarts in these 3 days, but I don't think it should be a problem.

I'll investigate more. If you have any idea on what I could try, I'll appreciate.

@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented Feb 14, 2017

@barbolo are you using file adapter to store files, or s3 (aws) or similar?
Also would be good to see at least part of the code where you are creating a file (controller?) and the model source code as well.
PS. I tried https://github.com/janko-m/shrine for the same task (storing files), and it's easier to support as it's more modular than paperclip and featureful. :)

@barbolo

This comment has been minimized.

Copy link

@barbolo barbolo commented Feb 14, 2017

@dmitry I'm using S3 (AWS). The code is below.

I call it, for example this way:

ReceiptAttachment.generate(other_model, html: "<html>...</html>")

class ReceiptAttachment < ApplicationRecord
  belongs_to :other_model

  has_attached_file :document, {
    url: '/documents/:id_:hash.:extension',
    hash_secret: '...',
    storage: Rails.env.production? ? :s3 : :filesystem,
    path: Rails.env.production? ? ':url' : ':rails_root/public/attachments:url',
    s3_credentials: { ... },
    s3_protocol: 'https'
  }

  do_not_validate_attachment_file_type :document

  def self.generate(other_model, content = {})
    ra = ReceiptAttachment.new(other_model: other_model)
    file = ReceiptAttachment.file_from_content(content)
    ra.document = file
    ra.document_content_type = file.content_type
    ra.save!
  end

  def self.file_from_content(content)
    file = StringIO.new(content[:html] || content[:pdf])
    file.class.class_eval { attr_accessor :original_filename, :content_type}
    if content[:pdf]
      file.original_filename = 'content.pdf'
      file.content_type = 'application/pdf'
    else
      file.original_filename = 'content.html'
      file.content_type = 'text/html'
    end
    file
  end
end
@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented Feb 14, 2017

@barbolo nice, thanks!

Looks like an issue in the stringio_adapter.rb. There are no unlink method to get rid of created @tempfile after they were written to the filesystem or network.

Try out the following code, might solve your issue:

  def self.file_from_content(content)
    data = StringIO.new(content[:html] || content[:pdf])
    file = Paperclip.io_adapters.for(data)
    if content[:pdf]
      file.original_filename = 'content.pdf'
      file.content_type = 'application/pdf'
    else
      file.original_filename = 'content.html'
      file.content_type = 'text/html'
    end
    file
  end

PS. Not StringIOAdapter but StringIO.

@barbolo

This comment has been minimized.

Copy link

@barbolo barbolo commented Feb 15, 2017

@dmitry The problem seems to be gone! I'm still monitoring and will update here if anything changes.

I'll try to use the gem aws-sdk directly, because I really don't want to touch the disk when saving these attachments.

@kilink

This comment has been minimized.

Copy link

@kilink kilink commented Mar 7, 2017

I've seen Paperclip tempfiles accumulating from a long-running Sidekiq process. I'm using Paperclip 4.3.7 and Sidekiq 4.2.9 on Ruby 2.3.3.

What I have discovered while investigating the issue is that Paperclip will leave tempfiles lingering if a processor throws an exception. For example, we're using paperclip-av-transcoder (0.6.4), and sometimes ffmpeg doesn't like the uploaded file, or it's not a valid video, and an execption is thrown. Examining the Attachment object after such an exception, you will see that the queued_for_writes and file attributes both refer to tempfiles.

When an exception isn't thrown, these tempfiles get unlinked when after_flush_writes is called when your model is saved. In our case, when an exception occurs, we are not saving the model, so this callback is never run.

Under normal circumstances, these lingering tempfiles wouldn't be a problem, because the Ruby GC would eventually clean them up. Now, I am not a Ruby expert by any means, so this is all wild speculation, but I suspect that a long running process where the heap grows large enough may see significantly fewer major GCs, thus exacerbating this issue. There is also a circular reference between the Paperclip::Attachment instance and the Model instance, which I am guessing means it is less likely to die in "eden" space and thus stick around longer.

Anyway, here is my hacky solution: wrap the paperclipping code in a begin / rescue, and if an exception is thrown, do the following to remove any tempfiles:

upload.paperclip.send(:after_flush_writes)
upload.paperclip.queued_for_write.clear

Which is essentially just looping through all of the values in the queued_for_write map and calling close and unlink on them.

Now, how would this actually be fixed in Paperclip proper? I'm guessing by adding similar clean up code in either the assign method or one of the post_process methods.

kilink added a commit to kilink/paperclip-av-transcoder that referenced this issue Mar 10, 2017
Proactively clean up the tempfile if Av throws an exception and whiny is true.
This prevents tempfiles from lingering around unnecessarily until the next GC.

See related issue: thoughtbot/paperclip#1326
@mjankowski mjankowski mentioned this issue Apr 11, 2017
1 of 1 task complete
@NeoElit

This comment has been minimized.

Copy link

@NeoElit NeoElit commented Apr 20, 2017

@kilink Thanks..
That solution resolves issue to an extend for me.

I was using Paperclip::Geometry.from_file(attachment.queued_for_write[:original].path) to validate image width and height which will raise Paperclip::Errors::NotIdentifiedByImageMagickError in case imagemagick doesn't respond well. If not rescued and manually handled, all the temp files from paperclip seems to accumulate.

However RackMultipart* temp files from passenger seems to float around even after that. I'm not sure whose job is to clear that off. Is it the job of passenger or paperclip? I could see it cleaned up when passenger is stopped.

@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented May 30, 2017

Original file not closed when it's not selected for reprocessing. Could be closed with instance_variable_get(:@file).close. Issue somewhere in the post_process_style

@mike-burns

This comment has been minimized.

Copy link
Member

@mike-burns mike-burns commented Apr 27, 2018

Thank you for reporting this. Unfortunately, we will be deprecating Paperclip and therefore will not have the bandwidth to address this issue.

@mike-burns mike-burns closed this Apr 27, 2018
@filipeperdigaosousa

This comment has been minimized.

Copy link

@filipeperdigaosousa filipeperdigaosousa commented May 21, 2018

Any news about that?

I'm trying to initialize a thousand of models with attachments but after a dozen of instances it results in the too many files open error.

      file = Tempfile.new(["#{DateTime.now.to_i}", ".jpg"])
      file.binmode
      file.write(Avatarly.generate_avatar(org.name, size: 600))
      file.read
      begin
        model.picture.content = file
      ensure
        file.close
        file.unlink
      end
@dmitry

This comment has been minimized.

Copy link
Contributor

@dmitry dmitry commented May 22, 2018

What says lsof?

mike-burns pushed a commit that referenced this issue Jul 27, 2018
In an effort to avoid filling $TMPDIR with stray files, let's close all
Tempfiles after we are done with them. Additionally, add an around-filter to
each test in the integration suite to catch cases where we don't do this.

This exposes issues around re-processing a subset of our attached files: it
leaves Tempfiles around. Mark that test as skipped (with a detailed
explanation) because we cannot figure out how to make it work.

Related to #1326.
boatrite added a commit to mobiledoorman/paperclip that referenced this issue Jun 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.