Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

(#3581) Stop forking around: Add fork helper method #620

Merged
merged 1 commit into from about 2 years ago

4 participants

Kelsey Hightower Michael Stahnke Stefan Schulte Daniel Pittman
Kelsey Hightower

Without this patch we are forking all over the place. All this forking
around is problematic when it comes to things like ActiveRecord and
database connections. It turns out that forking has a nasty side effect
of aggressively closing database connections and spewing errors all over
the place.

This patch introduces a new Puppet::Util#safe_posix_fork method that
does forking the right way (closing file descriptors, and resetting
stdin, stdout, and stderr). Tagmail, Puppet kick, and
Puppet::Util#execute_posix have been updated to make use of this new
functionality.

In the future, Puppet::Util#safe_posix_fork should be used in-place of
direct calls to Kernel#fork.

This patch includes related spec tests.

(#3581) Stop forking around: Add fork helper method
Without this patch we are forking all over the place. All this forking
around is problematic when it comes to things like ActiveRecord and
database connections. It turns out that forking has a nasty side effect
of aggressively closing database connections and spewing errors all over
the place.

This patch introduces a new `Puppet::Util#safe_posix_fork` method that
does forking the right way (closing file descriptors, and resetting
stdin, stdout, and stderr). Tagmail, Puppet kick, and
`Puppet::Util#execute_posix` have been updated to make use of this new
functionality.

In the future, `Puppet::Util#safe_posix_fork` should be used in-place of
direct calls to `Kernel#fork`.

This patch includes related spec tests.
1b810b1
Stefan Schulte
Collaborator

Does it actually work and ActiveRecord runs correctly? I do not know if calling close is the right thing. And counting from 3 to 256 seems a bit arbitrary to me since a process can hold much more than 256 filehandles. But i stumbled over the following implementation I found here http://cmdrclueless.com/blog/2009/07/06/activerecord-fork-exec-boom/

require 'fcntl'

keepers = [STDIN,STDOUT,STDERR]
ObjectSpace.each_object(IO) do |io|
  if not io.closed? and not keepers.include?(io)
    flags = io.fcntl(Fcntl::F_GETFD, 0)
    io.fcntl(Fnctl::F_SETFD, Fcntl::FD_CLOEXEC | flags)
  end
end

But I dont know the real difference between the closeonexec flag and calling the "ruby close".

Kelsey Hightower

@stschulte Thanks for the link, I'm going to update this pull and try something like that.

Daniel Pittman
Owner

@stschulte - for the record, yes, that actually works, and is correct. (This is, in fact, what Ruby does when you use popen, or backticks, or any number of similar ways to fork and capture output.)

Using 3 through 256 is pretty much arbitrary, but it is more correct - we want to close all file handles, not just those exposed at the Ruby level as IO objects. The arbitrary part is that, indeed, you can have more than 256 handles, but - Puppet pretty much never should, and MRI has no portable mechanism to obtain the maximum file handle number.

Finally, CLOEXEC is a (recent) Linux-ism, and isn't available on many platforms.

Daniel Pittman
Owner

@stahnma - it should be completely irrelevant in terms of the possible file descriptor leak, since that is in a single process, not across fork. The other two reflect the same thing - something is causing AR to leak connections. This isn't that - it would shut down the connection incorrectly instead. Sorry.

Daniel Pittman daniel-pittman merged commit 0b13a37 into from
Daniel Pittman daniel-pittman closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Apr 03, 2012
(#3581) Stop forking around: Add fork helper method
Without this patch we are forking all over the place. All this forking
around is problematic when it comes to things like ActiveRecord and
database connections. It turns out that forking has a nasty side effect
of aggressively closing database connections and spewing errors all over
the place.

This patch introduces a new `Puppet::Util#safe_posix_fork` method that
does forking the right way (closing file descriptors, and resetting
stdin, stdout, and stderr). Tagmail, Puppet kick, and
`Puppet::Util#execute_posix` have been updated to make use of this new
functionality.

In the future, `Puppet::Util#safe_posix_fork` should be used in-place of
direct calls to `Kernel#fork`.

This patch includes related spec tests.
1b810b1
This page is out of date. Refresh to see the latest.
2  lib/puppet/application/kick.rb
@@ -201,7 +201,7 @@ def main
201 201
       # do, then do the next host.
202 202
       if @children.length < options[:parallel] and ! todo.empty?
203 203
         host = todo.shift
204  
-        pid = fork do
  204
+        pid = safe_posix_fork do
205 205
           run_for_host(host)
206 206
         end
207 207
         @children[pid] = host
2  lib/puppet/reports/tagmail.rb
@@ -123,7 +123,7 @@ def process
123 123
 
124 124
   # Send the email reports.
125 125
   def send(reports)
126  
-    pid = fork do
  126
+    pid = Puppet::Util.safe_posix_fork do
127 127
       if Puppet[:smtpserver] != "none"
128 128
         begin
129 129
           Net::SMTP.start(Puppet[:smtpserver]) do |smtp|
22  lib/puppet/util.rb
@@ -292,7 +292,7 @@ def execfail(command, exception)
292 292
   end
293 293
 
294 294
   def execute_posix(command, arguments, stdin, stdout, stderr)
295  
-    child_pid = Kernel.fork do
  295
+    child_pid = safe_posix_fork(stdin, stdout, stderr) do
296 296
       # We can't just call Array(command), and rely on it returning
297 297
       # things like ['foo'], when passed ['foo'], because
298 298
       # Array(command) will call command.to_a internally, which when
@@ -301,12 +301,6 @@ def execute_posix(command, arguments, stdin, stdout, stderr)
301 301
       command = [command].flatten
302 302
       Process.setsid
303 303
       begin
304  
-        $stdin.reopen(stdin)
305  
-        $stdout.reopen(stdout)
306  
-        $stderr.reopen(stderr)
307  
-
308  
-        3.upto(256){|fd| IO::new(fd).close rescue nil}
309  
-
310 304
         Puppet::Util::SUIDManager.change_privileges(arguments[:uid], arguments[:gid], true)
311 305
 
312 306
         ENV['LANG'] = ENV['LC_ALL'] = ENV['LC_MESSAGES'] = ENV['LANGUAGE'] = 'C'
@@ -320,6 +314,20 @@ def execute_posix(command, arguments, stdin, stdout, stderr)
320 314
   end
321 315
   module_function :execute_posix
322 316
 
  317
+  def safe_posix_fork(stdin=$stdin, stdout=$stdout, stderr=$stderr, &block)
  318
+    child_pid = Kernel.fork do
  319
+      $stdin.reopen(stdin)
  320
+      $stdout.reopen(stdout)
  321
+      $stderr.reopen(stderr)
  322
+
  323
+      3.upto(256){|fd| IO::new(fd).close rescue nil}
  324
+
  325
+      block.call if block
  326
+    end
  327
+    child_pid
  328
+  end
  329
+  module_function :safe_posix_fork
  330
+
323 331
   def execute_windows(command, arguments, stdin, stdout, stderr)
324 332
     command = command.map do |part|
325 333
       part.include?(' ') ? %Q["#{part.gsub(/"/, '\"')}"] : part
4  spec/unit/application/kick_spec.rb
@@ -239,14 +239,14 @@
239 239
         @kick.hosts = ['host1', 'host2', 'host3']
240 240
         Process.stubs(:wait).returns(1).then.returns(2).then.returns(3).then.raises(Errno::ECHILD)
241 241
 
242  
-        @kick.expects(:fork).times(3).returns(1).then.returns(2).then.returns(3)
  242
+        @kick.expects(:safe_posix_fork).times(3).returns(1).then.returns(2).then.returns(3)
243 243
 
244 244
         expect { @kick.main }.to raise_error SystemExit
245 245
       end
246 246
 
247 247
       it "should delegate to run_for_host per host" do
248 248
         @kick.hosts = ['host1', 'host2']
249  
-        @kick.stubs(:fork).returns(1).yields
  249
+        @kick.stubs(:safe_posix_fork).returns(1).yields
250 250
         Process.stubs(:wait).returns(1).then.raises(Errno::ECHILD)
251 251
 
252 252
         @kick.expects(:run_for_host).times(2)
41  spec/unit/util_spec.rb
@@ -223,15 +223,6 @@ def stub_process_wait(exitstatus)
223 223
         Puppet::Util.execute_posix('test command', {}, @stdin, @stdout, @stderr)
224 224
       end
225 225
 
226  
-      it "should close all open file descriptors except stdin/stdout/stderr" do
227  
-        # This is ugly, but I can't really think of a better way to do it without
228  
-        # letting it actually close fds, which seems risky
229  
-        (0..2).each {|n| IO.expects(:new).with(n).never}
230  
-        (3..256).each {|n| IO.expects(:new).with(n).returns mock('io', :close) }
231  
-
232  
-        Puppet::Util.execute_posix('test command', {}, @stdin, @stdout, @stderr)
233  
-      end
234  
-
235 226
       it "should permanently change to the correct user and group if specified" do
236 227
         Puppet::Util::SUIDManager.expects(:change_group).with(55, true)
237 228
         Puppet::Util::SUIDManager.expects(:change_user).with(50, true)
@@ -487,6 +478,38 @@ def stub_process_wait(exitstatus)
487 478
         }.not_to raise_error
488 479
       end
489 480
     end
  481
+
  482
+    describe "safe_posix_fork" do
  483
+      before :each do
  484
+        # Most of the things this method does are bad to do during specs. :/
  485
+        Kernel.stubs(:fork).returns(pid).yields
  486
+
  487
+        $stdin.stubs(:reopen)
  488
+        $stdout.stubs(:reopen)
  489
+        $stderr.stubs(:reopen)
  490
+      end
  491
+
  492
+      it "should close all open file descriptors except stdin/stdout/stderr" do
  493
+        # This is ugly, but I can't really think of a better way to do it without
  494
+        # letting it actually close fds, which seems risky
  495
+        (0..2).each {|n| IO.expects(:new).with(n).never}
  496
+        (3..256).each {|n| IO.expects(:new).with(n).returns mock('io', :close) }
  497
+
  498
+        Puppet::Util.safe_posix_fork
  499
+      end
  500
+
  501
+      it "should fork a child process to execute the block" do
  502
+        Kernel.expects(:fork).returns(pid).yields
  503
+
  504
+        Puppet::Util.safe_posix_fork do
  505
+          message = "Fork this!"
  506
+        end
  507
+      end
  508
+
  509
+      it "should return the pid of the child process" do
  510
+        Puppet::Util.safe_posix_fork.should == pid
  511
+      end
  512
+    end
490 513
   end
491 514
 
492 515
   describe "#execpipe" do
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.