Use OpenBSD's malloc by default

[p5pRT]

Migrated from rt.perl.org#122000 (status was 'resolved')

Searchable as RT122000$

[p5pRT]

From afresh1@openbsd.org

Created by afresh1@openbsd.org

OpenBSD would prefer that perl default to using the system malloc.

In perlbug #75742 it was successfully argued that the speed of perl's malloc
was reason enough to default to using it instead of the system malloc on
OpenBSD. While the system malloc on OpenBSD is slower than the perl provided
malloc, the OpenBSD malloc provides many security features that people using
OpenBSD expect to have.

OpenBSD users, by virtue of having chosen OpenBSD, have made the decision that
security is more important than speed, so I believe this patch will do what
most users of perl on OpenBSD would expect.

This has always been the default in the base system perl in OpenBSD.

Perl users who do need the additional speed can still build with mymalloc.

These speed tests are with perl-5.18.2 as that is what I had easily available.

The speed of the OpenBSD malloc is not significantly slower in most cases
except the array case.

OpenBSD malloc​:
Benchmark​: timing 1000 iterations of 1E5 chars + 1E4 x 1E2 chars, 1E6 chars + 1E4 x 1E2 chars, 1E7 (1E5 x 1E2 chars) array + 1E4 x 1E2 chars , 1E7 chars (pre-extend to 2E7) + 1E4 x 1E2 chars, 1E7 chars + 1E1 x 1E5 chars, 1E7 chars + 1E2 x 1E4 chars, 1E7 chars + 1E3 x 1E3 chars, 1E7 chars + 1E4 x 1E2 chars, 1E7 chars + 1E5 x 1E1 chars...
1E5 chars + 1E4 x 1E2 chars​: 1.0778 wallclock secs ( 1.08 usr + 0.00 sys = 1.08 CPU) @​ 925.93/s (n=1000)
1E6 chars + 1E4 x 1E2 chars​: 1.18271 wallclock secs ( 1.19 usr + 0.00 sys = 1.19 CPU) @​ 840.34/s (n=1000)
1E7 (1E5 x 1E2 chars) array + 1E4 x 1E2 chars : 63.6428 wallclock secs (36.31 usr + 27.30 sys = 63.61 CPU) @​ 15.72/s (n=1000)
1E7 chars (pre-extend to 2E7) + 1E4 x 1E2 chars​: 1.63427 wallclock secs ( 1.14 usr + 0.49 sys = 1.63 CPU) @​ 613.50/s (n=1000)
1E7 chars + 1E1 x 1E5 chars​: 2.01161 wallclock secs ( 2.01 usr + 0.00 sys = 2.01 CPU) @​ 497.51/s (n=1000)
1E7 chars + 1E2 x 1E4 chars​: 2.00405 wallclock secs ( 2.00 usr + 0.00 sys = 2.00 CPU) @​ 500.00/s (n=1000)
1E7 chars + 1E3 x 1E3 chars​: 2.08054 wallclock secs ( 2.08 usr + 0.00 sys = 2.08 CPU) @​ 480.77/s (n=1000)
1E7 chars + 1E4 x 1E2 chars​: 3.05625 wallclock secs ( 3.07 usr + 0.00 sys = 3.07 CPU) @​ 325.73/s (n=1000)
1E7 chars + 1E5 x 1E1 chars​: 10.0508 wallclock secs (10.01 usr + 0.00 sys = 10.01 CPU) @​ 99.90/s (n=1000)

perl malloc​:
Benchmark​: timing 1000 iterations of 1E5 chars + 1E4 x 1E2 chars, 1E6 chars + 1E4 x 1E2 chars, 1E7 (1E5 x 1E2 chars) array + 1E4 x 1E2 chars , 1E7 chars (pre-extend to 2E7) + 1E4 x 1E2 chars, 1E7 chars + 1E1 x 1E5 chars, 1E7 chars + 1E2 x 1E4 chars, 1E7 chars + 1E3 x 1E3 chars, 1E7 chars + 1E4 x 1E2 chars, 1E7 chars + 1E5 x 1E1 chars...
1E5 chars + 1E4 x 1E2 chars​: 1.04273 wallclock secs ( 1.04 usr + 0.00 sys = 1.04 CPU) @​ 961.54/s (n=1000)
1E6 chars + 1E4 x 1E2 chars​: 1.13512 wallclock secs ( 1.13 usr + 0.00 sys = 1.13 CPU) @​ 884.96/s (n=1000)
1E7 (1E5 x 1E2 chars) array + 1E4 x 1E2 chars : 13.2705 wallclock secs (13.24 usr + 0.00 sys = 13.24 CPU) @​ 75.53/s (n=1000)
1E7 chars (pre-extend to 2E7) + 1E4 x 1E2 chars​: 1.57271 wallclock secs ( 1.16 usr + 0.43 sys = 1.59 CPU) @​ 628.93/s (n=1000)
1E7 chars + 1E1 x 1E5 chars​: 2.00015 wallclock secs ( 2.00 usr + 0.00 sys = 2.00 CPU) @​ 500.00/s (n=1000)
1E7 chars + 1E2 x 1E4 chars​: 1.99175 wallclock secs ( 1.98 usr + 0.00 sys = 1.98 CPU) @​ 505.05/s (n=1000)
1E7 chars + 1E3 x 1E3 chars​: 2.0531 wallclock secs ( 2.04 usr + 0.00 sys = 2.04 CPU) @​ 490.20/s (n=1000)
1E7 chars + 1E4 x 1E2 chars​: 2.90756 wallclock secs ( 2.90 usr + 0.00 sys = 2.90 CPU) @​ 344.83/s (n=1000)
1E7 chars + 1E5 x 1E1 chars​: 8.85005 wallclock secs ( 8.85 usr + 0.01 sys = 8.86 CPU) @​ 112.87/s (n=1000)

This uses a modified version of a malloc speed test I found on the Internet.

https://groups.google.com/forum/#!msg/comp.lang.perl.misc/t8kTP_IACfI/4w02x-dWPB4J

use strict;
use warnings;

use Benchmark qw( timethese :hireswallclock );

my $c1E1 = '#' x 1E1;
my $c1E2 = '#' x 1E2;
my $c1E3 = '#' x 1E3;
my $c1E4 = '#' x 1E4;
my $c1E5 = '#' x 1E5;

my $str1 = '#' x 1E5;
my $str2 = '#' x 1E6;
my $str3 = '#' x 1E7;

my $str4 = '#' x 1E7;
my $str5 = '#' x 1E7;
my $str6 = '#' x 1E7;
my $str7 = '#' x 1E7;
my $str8 = '#' x 1E7;

my $str9 = '#' x 2E7;
$str9 = '#' x 1E7;

my @​ar1 = map{ $c1E2 } 1..1E5;

timethese( 1000, {
  '1E5 chars + 1E4 x 1E2 chars' => sub{ my $str = $str1; $str .= $c1E2 for 1..1E4 },
  '1E6 chars + 1E4 x 1E2 chars' => sub{ my $str = $str2; $str .= $c1E2 for 1..1E4 },
  '1E7 chars + 1E4 x 1E2 chars' => sub{ my $str = $str3; $str .= $c1E2 for 1..1E4 },
  '1E7 chars + 1E5 x 1E1 chars' => sub{ my $str = $str4; $str .= $c1E1 for 1..1E5 },
  '1E7 chars + 1E4 x 1E2 chars' => sub{ my $str = $str5; $str .= $c1E2 for 1..1E4 },
  '1E7 chars + 1E3 x 1E3 chars' => sub{ my $str = $str6; $str .= $c1E3 for 1..1E3 },
  '1E7 chars + 1E2 x 1E4 chars' => sub{ my $str = $str7; $str .= $c1E4 for 1..1E2 },
  '1E7 chars + 1E1 x 1E5 chars' => sub{ my $str = $str8; $str .= $c1E5 for 1..1E1 },
  '1E7 chars (pre-extend to 2E7) + 1E4 x 1E2 chars' => sub{ $str9 .= $c1E2 for 1..1E4 },
  '1E7 (1E5 x 1E2 chars) array + 1E4 x 1E2 chars ' => sub{ my @​ar = @​ar1; push @​ar, $c1E2 for 1..1E4 },
});

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.18.2:

Configured by root at Thu Jan  1  0:00:00 UTC 1970.

Summary of my perl5 (revision 5 version 18 subversion 2) configuration:
   
  Platform:
    osname=openbsd, osvers=5.5, archname=amd64-openbsd
    uname='openbsd'
    config_args='-dsE -Dopenbsd_distribution=defined -Dccflags=-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -Dmksymlinks'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include',
    optimize='-O2',
    cppflags='-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.2.1 20070719 ', gccosandvers='openbsd5.5'
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-Wl,-E  -fstack-protector'
    libpth=/usr/lib
    libs=-lm -lutil -lc
    perllibs=-lm -lutil -lc
    libc=/usr/lib/libc.so.74.2, so=so, useshrplib=true, libperl=libperl.so.15.0
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-R/usr/libdata/perl5/amd64-openbsd/5.18.2/CORE'
    cccdlflags='-DPIC -fPIC ', lddlflags='-shared -fPIC  -fstack-protector'

Locally applied patches:
    


@INC for perl 5.18.2:
    /usr/local/libdata/perl5/site_perl/amd64-openbsd
    /usr/libdata/perl5/site_perl/amd64-openbsd
    /usr/local/libdata/perl5/site_perl
    /usr/libdata/perl5/site_perl
    /usr/libdata/perl5/amd64-openbsd/5.18.2
    /usr/local/libdata/perl5/amd64-openbsd/5.18.2
    /usr/libdata/perl5
    /usr/local/libdata/perl5
    .


Environment for perl 5.18.2:
    HOME=/home/afresh1
    LANG (unset)
    LANGUAGE (unset)
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/afresh1/.plenv/bin:/home/afresh1/bin:/home/afresh1/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:/home/afresh1/.plenv/bin:.
    PERL_BADLANG (unset)
    SHELL=/bin/ksh

[p5pRT]

From afresh1@openbsd.org

0001-Default-to-using-OpenBSD-s-malloc.patch

From de5afa1c9bfe274e36e94aec99e866ba8d2c0a15 Mon Sep 17 00:00:00 2001
From: Andrew Fresh <andrew@afresh1.com>
Date: Sat, 31 May 2014 10:30:40 -0700
Subject: [PATCH] Default to using OpenBSD's malloc

While perl bug #75742 successfully convinced p5p that the speed of the
OpenBSD malloc was a good enough reason to instead use perl's malloc,
the system malloc has many security features that OpenBSD users expect.

The system malloc has always been used in the base system perl.

While the is slower than the perl malloc, the OpenBSD project believes
the tradeoff is worth it and would prefer that users who need the speed
specifically ask for it.
---
 hints/openbsd.sh | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/hints/openbsd.sh b/hints/openbsd.sh
index fd29c71..26a8fe6 100644
--- a/hints/openbsd.sh
+++ b/hints/openbsd.sh
@@ -8,12 +8,8 @@
 #	./Configure -des -Dopenbsd_distribution=defined
 #
 
-# In OpenBSD > 3.7, use perl's malloc [perl #75742]
-case "$osvers" in
-3.[89]*|[4-9]*)
-    test "$usemymalloc" || usemymalloc=y
-    ;;
-esac
+# OpenBSD has a better malloc than perl...
+test "$usemymalloc" || usemymalloc='n'
 
 # malloc wrap works
 case "$usemallocwrap" in
-- 
1.9.2

[p5pRT]

From eric@lalonde.com

OpenBSD would prefer that perl default to using the system malloc.

In perlbug #75742 it was successfully argued that the speed of perl's
malloc
was reason enough to default to using it instead of the system malloc
on
OpenBSD. While the system malloc on OpenBSD is slower than the perl
provided
malloc, the OpenBSD malloc provides many security features that people
using
OpenBSD expect to have.

OpenBSD users, by virtue of having chosen OpenBSD, have made the
decision that
security is more important than speed, so I believe this patch will do
what
most users of perl on OpenBSD would expect.

This has always been the default in the base system perl in OpenBSD.

Perl users who do need the additional speed can still build with
mymalloc.

These speed tests are with perl-5.18.2 as that is what I had easily
available.

The speed of the OpenBSD malloc is not significantly slower in most
cases
except the array case.

I would like to +1 this sentiment. Since the user has installed OpenBSD, the sane default assumption is that the user desires to experience the full breadth of security features built into the OS.

[p5pRT]

From [Unknown Contact. See original ticket]

OpenBSD would prefer that perl default to using the system malloc.

In perlbug #75742 it was successfully argued that the speed of perl's
malloc
was reason enough to default to using it instead of the system malloc
on
OpenBSD. While the system malloc on OpenBSD is slower than the perl
provided
malloc, the OpenBSD malloc provides many security features that people
using
OpenBSD expect to have.

OpenBSD users, by virtue of having chosen OpenBSD, have made the
decision that
security is more important than speed, so I believe this patch will do
what
most users of perl on OpenBSD would expect.

This has always been the default in the base system perl in OpenBSD.

Perl users who do need the additional speed can still build with
mymalloc.

These speed tests are with perl-5.18.2 as that is what I had easily
available.

The speed of the OpenBSD malloc is not significantly slower in most
cases
except the array case.

I would like to +1 this sentiment. Since the user has installed OpenBSD, the sane default assumption is that the user desires to experience the full breadth of security features built into the OS.

[p5pRT]

From @rurban

On Sat May 31 15​:03​:26 2014, eric@​lalonde.com wrote​:

OpenBSD would prefer that perl default to using the system malloc.

I would like to +1 this sentiment. Since the user has installed
OpenBSD, the sane default assumption is that the user desires to
experience the full breadth of security features built into the OS.

perl mymalloc is not thread-safe. I wouldn't trust it.

I found it odd that openbsd still has to use mymalloc.
cygwin switched to system malloc some time ago.

--
Reini Urban

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @b2gills

On Sun, Jun 1, 2014 at 1​:04 AM, Reini Urban via RT
<perlbug-followup@​perl.org> wrote​:

On Sat May 31 15​:03​:26 2014, eric@​lalonde.com wrote​:

OpenBSD would prefer that perl default to using the system malloc.

I would like to +1 this sentiment. Since the user has installed
OpenBSD, the sane default assumption is that the user desires to
experience the full breadth of security features built into the OS.

perl mymalloc is not thread-safe. I wouldn't trust it.

I found it odd that openbsd still has to use mymalloc.
cygwin switched to system malloc some time ago.

--
Reini Urban

I would also like to see this happen, as it would likely tease out
any use after free bugs.
( This was the main reason OpenSSL couldn't use the system
malloc on OpenBSD )

[p5pRT]

From @tonycoz

On Sat May 31 12​:02​:07 2014, afresh1@​openbsd.org wrote​:

OpenBSD would prefer that perl default to using the system malloc.

Thanks, applied as 9be9e8a with the discussed author change.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'resolved'