ABI incompatibility with -DDEBUGGING vs. without, caused by memory_debug_header

[p5pRT]

Migrated from rt.perl.org#127212 (status was 'resolved')

Searchable as RT127212$

[p5pRT]

From @ntyni

This is a bug report for perl from Niko Tyni <ntyni@​debian.org>,
generated with the help of perlbug 1.40 running under perl 5.22.1.

---

The strict XS binary interface checking in Perl_xs_handshake(), introduced
in the 5.21 cycle, has pinpointed a long standing ABI incompatibility in
-DDEBUGGING vs. normal builds. According to at least commit f2b8894
such builds are intended to be binary compatible, but we're getting

% debugperl -e 'use Cwd'
Cwd.c​: loadable library and perl binaries are mismatched (got handshake key 0xdb00080, needed 0xdc80080)

where 'debugperl' is a 5.22.1 -DDEBUGGING one, configured otherwise
identically to the 'normal' perl that the extension is built for.

The handshake check takes the interpreter size into account, but
the sizes differ by 24 bytes here on x86_64-linux-gnu, because of
the 'memory_debug_header' interpreter member that's conditional on
PERL_TRACK_MEMPOOL, enabled by -DDEBUGGING. While 'memory_debug_header'
is inserted almost at the end, it does move 'sv_consts' and 'random_state'
forward, making this a "real" ABI incompatibily, albeit apparently minor
enough that it hasn't been noticed earlier.

I suppose either PERL_TRACK_MEMPOOL needs to not get enabled
by -DDEBUGGING, or the interpreter variables need a similarly
sized placeholder for the non-debugging case.

FWIW, I intend to do the former for the Debian 5.22 packages, where
we would very much like to keep the 'debugperl' variant able to load
extensions built for the normal non-debugging perl.

(This is originally https://bugs.debian.org/810326)

---

---

Flags​:
  category=core
  severity=medium

---

Site configuration information for perl 5.22.1​:

Configured by Debian Project at Sat Dec 26 07​:28​:43 UTC 2015.

Summary of my perl5 (revision 5 version 22 subversion 1) configuration​:
 
  Platform​:
  osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi
  uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux '
  config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.22 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.22 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.22 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.22.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.22.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.22.1 -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
  ccversion='', gccversion='5.3.1 20151219', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
  perllibs=-ldl -lm -lpthread -lc -lcrypt
  libc=libc-2.21.so, so=so, useshrplib=true, libperl=libperl.so.5.22
  gnulibc_version='2.21'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches​:
  DEBPKG​:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
  DEBPKG​:debian/db_file_ver - http​://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
  DEBPKG​:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
  DEBPKG​:debian/enc2xs_inc - http​://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @​INC directories.
  DEBPKG​:debian/errno_ver - http​://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
  DEBPKG​:debian/libperl_embed_doc - http​://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
  DEBPKG​:fixes/respect_umask - Respect umask during installation
  DEBPKG​:debian/writable_site_dirs - Set umask approproately for site install directories
  DEBPKG​:debian/extutils_set_libperl_path - EU​:MM​: set location of libperl.a under /usr/lib
  DEBPKG​:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
  DEBPKG​:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
  DEBPKG​:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
  DEBPKG​:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
  DEBPKG​:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
  DEBPKG​:debian/mod_paths - Tweak @​INC ordering for Debian
  DEBPKG​:debian/prune_libs - http​://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
  DEBPKG​:fixes/net_smtp_docs - [rt.cpan.org #36038] http​://bugs.debian.org/100195 Document the Net​::SMTP 'Port' option
  DEBPKG​:debian/perlivp - http​://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
  DEBPKG​:debian/deprecate-with-apt - http​://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
  DEBPKG​:debian/squelch-locale-warnings - http​://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
  DEBPKG​:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
  DEBPKG​:debian/patchlevel - http​://bugs.debian.org/567489 List packaged patches for 5.22.1-3 in patchlevel.h
  DEBPKG​:debian/skip-kfreebsd-crash - http​://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
  DEBPKG​:fixes/document_makemaker_ccflags - http​://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
  DEBPKG​:debian/find_html2text - http​://bugs.debian.org/640479 Configure CPAN​::Distribution with correct name of html2text
  DEBPKG​:debian/perl5db-x-terminal-emulator.patch - http​://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
  DEBPKG​:debian/cpan-missing-site-dirs - http​://bugs.debian.org/688842 Fix CPAN​::FirstTime defaults with nonexisting site dirs if a parent is writable
  DEBPKG​:fixes/memoize_storable_nstore - [rt.cpan.org #77790] http​://bugs.debian.org/587650 Memoize​::Storable​: respect 'nstore' option not respected
  DEBPKG​:debian/regen-skip - Skip a regeneration check in unrelated git repositories
  DEBPKG​:debian/makemaker-pasthru - http​://bugs.debian.org/758471 Pass LD settings through to subdirectories
  DEBPKG​:fixes/pod_man_reproducible_date - http​://bugs.debian.org/759405 Support POD_MAN_DATE in Pod​::Man for the left-hand footer
  DEBPKG​:debian/locale-robustness - http​://bugs.debian.org/782068 [perl #124310] Make t/run/locale.t survive missing locales masked by LC_ALL
  DEBPKG​:fixes/podman-utc - http​://bugs.debian.org/780259 Make the embedded date from Pod​::Man reproducible
  DEBPKG​:fixes/podman-utc-docs - http​://bugs.debian.org/780259 Documentation and test suite updates for UTC fix
  DEBPKG​:fixes/podman-empty-date - http​://bugs.debian.org/780259 Support an empty POD_MAN_DATE environment variable
  DEBPKG​:fixes/podman-pipe - http​://bugs.debian.org/777405 Better errors for man pages from standard input
  DEBPKG​:debian/pod2man-customized - Update porting/customized.dat for pod2man modifications
  DEBPKG​:debian/makemaker-manext - http​://bugs.debian.org/247370 Make EU​::MakeMaker honour MANnEXT settings in generated manpage headers
  DEBPKG​:debian/makemaker_customized - Update t/porting/customized.dat for files patched in Debian
  DEBPKG​:debian/do-not-record-build-date - [6baa8db] http​://bugs.debian.org/774422 [perl #125830] Allow overriding the compile time in "perl -V" output
  DEBPKG​:fixes/podman-source-date-epoch - http​://bugs.debian.org/801621 Make Pod​::Man honor the SOURCE_DATE_EPOCH environment variable
  DEBPKG​:fixes/podman-source-date-epoch-cleanups - http​://bugs.debian.org/801621 Coding style and documentation for SOURCE_EPOCH_DATE
  DEBPKG​:fixes/podman-source-date-epoch-testfix - http​://bugs.debian.org/807086 Guard for building with SOURCE_DATE_EPOCH or POD_MAN_DATE set
  DEBPKG​:debian/devel-ppport-reproducibility - http​://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs
  DEBPKG​:fixes/encode-unicode-bom - http​://bugs.debian.org/798727 [rt.cpan.org #107043] Address https://rt.cpan.org/Public/Bug/Display.html?id=107043
  DEBPKG​:debian/encode-unicode-bom-doc - http​://bugs.debian.org/798727 Document Debian backport of Encode​::Unicode fix
  DEBPKG​:debian/kfreebsd-softupdates - http​://bugs.debian.org/796798 Work around Debian Bug#796798
  DEBPKG​:fixes/autodie-scope - http​://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub

---

@​INC for perl 5.22.1​:
  /etc/perl
  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1
  /usr/local/share/perl/5.22.1
  /usr/lib/x86_64-linux-gnu/perl5/5.22
  /usr/share/perl5
  /usr/lib/x86_64-linux-gnu/perl/5.22
  /usr/share/perl/5.22
  /usr/local/lib/site_perl
  /usr/lib/x86_64-linux-gnu/perl-base
  .

---

Environment for perl 5.22.1​:
  HOME=/home/niko
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LC_CTYPE=fi_FI.UTF-8
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/niko/bin​:/home/niko/bin​:/home/niko/bin​:/usr/local/bin​:/usr/bin​:/bin​:/usr/local/games​:/usr/games​:/sbin​:/usr/sbin​:/sbin​:/usr/sbin
  PERL_BADLANG (unset)
  SHELL=/bin/zsh

[p5pRT]

From @arc

Niko Tyni <perlbug-followup@​perl.org> wrote​:

The strict XS binary interface checking in Perl_xs_handshake(), introduced
in the 5.21 cycle, has pinpointed a long standing ABI incompatibility in
-DDEBUGGING vs. normal builds. According to at least commit f2b8894
such builds are intended to be binary compatible, but we're getting

% debugperl -e 'use Cwd'
Cwd.c​: loadable library and perl binaries are mismatched (got handshake key 0xdb00080, needed 0xdc80080)

where 'debugperl' is a 5.22.1 -DDEBUGGING one, configured otherwise
identically to the 'normal' perl that the extension is built for.

Thanks for the report. This is fixed as of
5ca5a62.

As far as I can tell, we can't fix this for any maint release, as it
would introduce a binary incompatibility between non-DEBUGGING
releases in the relevant maint. So I think Debian's builds of 5.22.x
will need to retain the local patch found in Debian #810326. But that
patch shouldn't be necessary at such time as 5.24 goes into Debian.

--
Aaron Crane ** http​://aaroncrane.co.uk/

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

@arc - Status changed from 'open' to 'resolved'

[p5pRT]

From @ilmari

Aaron Crane <arc@​cpan.org> writes​:

Niko Tyni <perlbug-followup@​perl.org> wrote​:

The strict XS binary interface checking in Perl_xs_handshake(), introduced
in the 5.21 cycle, has pinpointed a long standing ABI incompatibility in
-DDEBUGGING vs. normal builds. According to at least commit f2b8894
such builds are intended to be binary compatible, but we're getting

% debugperl -e 'use Cwd'
Cwd.c​: loadable library and perl binaries are mismatched (got handshake key 0xdb00080, needed 0xdc80080)

where 'debugperl' is a 5.22.1 -DDEBUGGING one, configured otherwise
identically to the 'normal' perl that the extension is built for.

Thanks for the report. This is fixed as of
5ca5a62.

This breaks builds without PERL_IMPLICIT_CONTEXT (i.e. threads) or
PERL_DEBUG_READONLY_COW defined, since struct perl_memory_debug_header
is not defined otherwise.

echo @​`sh cflags "optimize='-O2'" perlmini.o` -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
@​cc -c -DPERL_CORE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -O2 -Wall -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
In file included from perl.c​:33​:0​:
intrpvar.h​:787​:9​: error​: field ‘Imemory_debug_header’ has incomplete type
PERLVAR(I, memory_debug_header, struct perl_memory_debug_header)
  ^
perl.h​:5633​:41​: note​: in definition of macro ‘PERLVAR’
# define PERLVAR(prefix,var,type) type prefix##var;
  ^
makefile​:320​: recipe for target 'perlmini.o' failed
make​: *** [perlmini.o] Error 1

--
"I use RMS as a guide in the same way that a boat captain would use
a lighthouse. It's good to know where it is, but you generally
don't want to find yourself in the same spot." - Tollef Fog Heen

[p5pRT]

From @arc

Dagfinn Ilmari Mannsåker <ilmari@​ilmari.org> wrote​:

Aaron Crane <arc@​cpan.org> writes​:

Thanks for the report. This is fixed as of
5ca5a62.

This breaks builds without PERL_IMPLICIT_CONTEXT (i.e. threads) or
PERL_DEBUG_READONLY_COW defined, since struct perl_memory_debug_header
is not defined otherwise.

D'oh. Thanks for the report; fixed in 0419d97.

--
Aaron Crane ** http​://aaroncrane.co.uk/

[p5pRT]

From @bulk88

On Sun Jan 17 06​:54​:53 2016, arc wrote​:

Dagfinn Ilmari Mannsåker <ilmari@​ilmari.org> wrote​:

Aaron Crane <arc@​cpan.org> writes​:

Thanks for the report. This is fixed as of
5ca5a62.

This breaks builds without PERL_IMPLICIT_CONTEXT (i.e. threads) or
PERL_DEBUG_READONLY_COW defined, since struct
perl_memory_debug_header
is not defined otherwise.

D'oh. Thanks for the report; fixed in
0419d97.

Your fix does not fix ABI compatibility between a DEBUGGING XS module and non-DEBUGGING interp because of Perl_pad_sv. See https​://rt.perl.org/Ticket/Display.html?id=123007#txn-1314815 and https://rt.perl.org/Public/Bug/Display.html?id=122032 .

--
bulk88 ~ bulk88 at hotmail.com