Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global destruction sometimes makes linux COW useless #13297

Closed
p5pRT opened this issue Sep 21, 2013 · 5 comments
Closed

Global destruction sometimes makes linux COW useless #13297

p5pRT opened this issue Sep 21, 2013 · 5 comments

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Sep 21, 2013

Migrated from rt.perl.org#119937 (status was 'rejected')

Searchable as RT119937$

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 21, 2013

From victor@vsespb.ru

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e. with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks, memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data structures
without modifying them.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
  if (my $pid = fork) {
  push @​pids, $pid;
  } else {
  while() { sleep 1;}
  }
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 30, 2013

From @iabyn

On Sat, Sep 21, 2013 at 02​:42​:18PM -0700, Victor Efimov wrote​:

# New Ticket Created by Victor Efimov
# Please include the string​: [perl #119937]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=119937 >

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e. with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks, memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data structures
without modifying them.

I can't see how this could be done. Every SV is reference counted,
and in the case of your code below, the lexical $x goes out of scope,
which refcnt_dec()s the AV pointed to by it, which triggers
refcnt_dec()ing each of its members, which frees them, adding them back to
the pool of free SVs.

Note that this is happening earlier than global destruction. If you
made $x a package var, then (assuming it didn't contain any objects,
that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread),
then all the SVs are just abandoned rather than being freed.

Your other option would be to use POSIX​::_exit.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
if (my $pid = fork) {
push @​pids, $pid;
} else {
while() { sleep 1;}
}
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
  (with apologies to William Blake)

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 30, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 1, 2013

From victor@vsespb.ru

Indeed, if replace "my $x={};" with "our $x={};", problem seems to go
away (RAM usage spike is smaller). Sorry, didn't notice this earlier. I
think ticket can be closed then. Thanks.

On Mon Sep 30 13​:01​:38 2013, davem wrote​:

On Sat, Sep 21, 2013 at 02​:42​:18PM -0700, Victor Efimov wrote​:

# New Ticket Created by Victor Efimov
# Please include the string​: [perl #119937]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=119937 >

Below example (with $N high enough - 100 in my case) works fine if I
terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e.
with
USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks,
memory
is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory
is
modified in this particular example.

So I am wondering, maybe it's possible to free simple data
structures
without modifying them.

I can't see how this could be done. Every SV is reference counted,
and in the case of your code below, the lexical $x goes out of scope,
which refcnt_dec()s the AV pointed to by it, which triggers
refcnt_dec()ing each of its members, which frees them, adding them
back to
the pool of free SVs.

Note that this is happening earlier than global destruction. If you
made $x a package var, then (assuming it didn't contain any objects,
that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread),
then all the SVs are just abandoned rather than being freed.

Your other option would be to use POSIX​::_exit.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @​pids;
for (1..$N) {
if (my $pid = fork) {
push @​pids, $pid;
} else {
while() { sleep 1;}
}
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @​pids;

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 1, 2013

@iabyn - Status changed from 'open' to 'rejected'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.