Global destruction sometimes makes linux COW useless

[p5pRT]

Migrated from rt.perl.org#119937 (status was 'rejected')

Searchable as RT119937$

[p5pRT]

From victor@vsespb.ru

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e. with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks, memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data structures
without modifying them.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
  if (my $pid = fork) {
  push @​pids, $pid;
  } else {
  while() { sleep 1;}
  }
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

[p5pRT]

From @iabyn

On Sat, Sep 21, 2013 at 02​:42​:18PM -0700, Victor Efimov wrote​:

# New Ticket Created by Victor Efimov
# Please include the string​: [perl #119937]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=119937 >

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e. with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks, memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data structures
without modifying them.

I can't see how this could be done. Every SV is reference counted,
and in the case of your code below, the lexical $x goes out of scope,
which refcnt_dec()s the AV pointed to by it, which triggers
refcnt_dec()ing each of its members, which frees them, adding them back to
the pool of free SVs.

Note that this is happening earlier than global destruction. If you
made $x a package var, then (assuming it didn't contain any objects,
that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread),
then all the SVs are just abandoned rather than being freed.

Your other option would be to use POSIX​::_exit.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
if (my $pid = fork) {
push @​pids, $pid;
} else {
while() { sleep 1;}
}
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
  (with apologies to William Blake)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From victor@vsespb.ru

Indeed, if replace "my $x={};" with "our $x={};", problem seems to go
away (RAM usage spike is smaller). Sorry, didn't notice this earlier. I
think ticket can be closed then. Thanks.

On Mon Sep 30 13​:01​:38 2013, davem wrote​:

On Sat, Sep 21, 2013 at 02​:42​:18PM -0700, Victor Efimov wrote​:

# New Ticket Created by Victor Efimov
# Please include the string​: [perl #119937]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=119937 >

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e.
with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks,
memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory
is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data
structures
without modifying them.

I can't see how this could be done. Every SV is reference counted,
and in the case of your code below, the lexical $x goes out of scope,
which refcnt_dec()s the AV pointed to by it, which triggers
refcnt_dec()ing each of its members, which frees them, adding them
back to
the pool of free SVs.

Note that this is happening earlier than global destruction. If you
made $x a package var, then (assuming it didn't contain any objects,
that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread),
then all the SVs are just abandoned rather than being freed.

Your other option would be to use POSIX​::_exit.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
if (my $pid = fork) {
push @​pids, $pid;
} else {
while() { sleep 1;}
}
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

[p5pRT]

@iabyn - Status changed from 'open' to 'rejected'