ext/GDBM_File/t/gdbm.t fails with Perl 5.34 and gdbm 1.20

[jplesnik]

Module:
GDBM_File

Description

I am not able build Perl 5.34 on latest Fedora with new gdbm 1.20.
The test ../ext/GDBM_File/t/gdbm.t failed with error.

gdbm_firstkey: Item not found at ../../t/lib/dbmt_common.pl line 52.
Compilation failed in require at t/gdbm.t line 6.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 5.
../ext/GDBM_File/t/gdbm.t ............................................ 
Dubious, test returned 255 (wstat 65280, 0xff00)
All 5 subtests passed 

Steps to Reproduce

I don't have one-liner.

I installed all of Perl's dependencies (built with GDBM 1.19) and updated GDBM 1.20. Then I built Perl with the following command:

/bin/sh Configure -des -Doptimize=none '-Dccflags=-O2 -flto=auto -ffat-lto-objects -fexceptions \
-g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 \
-Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong \
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables \
-fstack-clash-protection -fcf-protection' '-Dldflags=-Wl,-z,relro -Wl,--as-needed  \
-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' '-Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro \
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' '-Dlddlflags=-shared -Wl,-z,relro \
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' -Dshrpdir=/usr/lib64 -DDEBUGGING=-g \
-Dversion=5.34.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc '-Dcf_by=Red Hat, Inc.' \
-Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5/5.34 \
-Dsitearch=/usr/local/lib64/perl5/5.34 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl \
-Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi \
'-Dlibpth=/usr/local/lib64 /lib64 /usr/lib64' -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace \
-Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio \
-Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly '-Dpager=/usr/bin/less -isr' -Dd_gethostent_r_proto \
-Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto \
-Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint

More information about build and environment could be found in logs
https://jplesnik.fedorapeople.org/perl-5.34-gdbm-1.20/

I was able to successfully rebuild perl 5.32.1 with GDBM 1.20

Expected behavior

The build would succeed.

Perl configuration

# perl -V
Summary of my perl5 (revision 5 version 34 subversion 0) configuration:
   
  Platform:
    osname=linux
    osvers=5.12.8-300.fc34.x86_64
    archname=x86_64-linux-thread-multi
    uname='linux buildvm-x86-22.iad2.fedoraproject.org 5.12.8-300.fc34.x86_64 #1 smp fri may 28 15:20:54 utc 2021 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Dldflags=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  -Dlddlflags=-shared -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.34.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5/5.34 -Dsitearch=/usr/local/lib64/perl5/5.34 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='  -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include'
    ccversion=''
    gccversion='11.1.1 20210531 (Red Hat 11.1.1-3)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='gcc'
    ldflags ='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib
    libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc
    libc=/lib/../lib64/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.33.9000'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '
    cccdlflags='-fPIC'
    lddlflags='-lpthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_SITECUSTOMIZE
    USE_THREAD_SAFE_LOCALE
  Built under linux
  Compiled at Jun 16 2021 00:00:00
  @INC:
    /usr/local/lib64/perl5/5.34
    /usr/local/share/perl5/5.34
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5

[jkeenan]

Module:
GDBM_File

Description

I am not able build Perl 5.34 on latest Fedora with new gdbm 1.20.
The test ../ext/GDBM_File/t/gdbm.t failed with error.

gdbm_firstkey: Item not found at ../../t/lib/dbmt_common.pl line 52.
Compilation failed in require at t/gdbm.t line 6.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 5.
../ext/GDBM_File/t/gdbm.t ............................................ 
Dubious, test returned 255 (wstat 65280, 0xff00)
All 5 subtests passed 

Can you paste the output of the following command, run from the top-level of this perl installation:

cd t; ./perl harness -v ../ext/GDBM_File/t/gdbm.t; cd -

Thank you very much.
Jim Keenan

[jplesnik]

@jkeenan

# cd t; ./perl harness -v ../ext/GDBM_File/t/gdbm.t; cd -
../ext/GDBM_File/t/gdbm.t .. 
ok 1 - use GDBM_File;
ok 2 - (eval q{(GDBM_WRCREAT, GDBM_WRITER)})[0]
ok 3 - (eval q{(GDBM_WRCREAT, GDBM_WRITER)})[1]
ok 4 - An object of class 'GDBM_File' isa 'GDBM_File'
ok 5
gdbm_firstkey: Item not found at ../../t/lib/dbmt_common.pl line 52.
Compilation failed in require at t/gdbm.t line 6.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 5.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 5 subtests passed 

Test Summary Report
-------------------
../ext/GDBM_File/t/gdbm.t (Wstat: 65280 Tests: 5 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.05 cusr  0.01 csys =  0.07 CPU)
Result: FAIL

[jkeenan]

Thanks. So that we can hone in on the failure (and possibly run a bisection), can you place the following program in ext/GDBM_File/ and then run it in a way similar to the above?

$ cat ext/GDBM_File/t/failure.t
#!./perl
use strict;
use warnings;

use Test::More;

$::Create_and_Write = '(GDBM_WRCREAT, GDBM_WRITER)';
our $DBM_Class = 'GDBM_File';
use_ok($DBM_Class);
diag ("GDBM_version: " . GDBM_File->GDBM_version);
my @ar = GDBM_File->GDBM_version;
diag(join('.' => @ar));

my ($create, $write) = eval $::Create_and_Write;
unlink <Op_dbmx.*>;
umask(0);

my %h;
isa_ok(tie(%h, $DBM_Class, 'Op_dbmx', $create, 0640), $DBM_Class);

my $Dfile = "Op_dbmx.pag";
if (! -e $Dfile) {
    ($Dfile) = <Op_dbmx*>;
}
my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,
$blksize,$blocks) = stat($Dfile);
is($mode & 0777, 0640, "mode 0640");

my $i = 0;
while (my ($key,$value) = each(%h)) {
    $i++;
}
is($i, 0, "no increment");

done_testing();

cd t; ./perl harness -v ../ext/GDBM_File/t/failure.t; cd -

[jkeenan]

It would also be helpful if you could configure and build perl with a much simpler set of switches to ./Configure, say:

sh ./Configure -des -Dusedevel -Duseithreads

That way we could rule out the problem being associated with one of the many switches in our config_args.

Thank you very much.
Jim Keenan

[jplesnik]

I used to build ./Configure as you recommended.

The output of the test is:

# cd t; ./perl harness -v ../ext/GDBM_File/t/failure.t; cd -
../ext/GDBM_File/t/failure.t .. 
ok 1 - use GDBM_File;
# GDBM_version: 1.20
# 1.20.0
ok 2 - An object of class 'GDBM_File' isa 'GDBM_File'
ok 3 - mode 0640
gdbm_firstkey: Item not found at t/failure.t line 30.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 3.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 3 subtests passed 

Test Summary Report
-------------------
../ext/GDBM_File/t/failure.t (Wstat: 65280 Tests: 3 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=3,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.05 cusr  0.01 csys =  0.07 CPU)
Result: FAIL

[Leont]

@graygnuorg do you have an idea?

[graygnuorg]

Yes, I do. This is because the code in GDBM_File.xs assumes gdbm error codes are defines, whereas they are enumerated constants.
The following PR fixes this: #18919

[jkeenan]

I used to build ./Configure as you recommended.

The output of the test is:

# cd t; ./perl harness -v ../ext/GDBM_File/t/failure.t; cd -
../ext/GDBM_File/t/failure.t .. 
ok 1 - use GDBM_File;
# GDBM_version: 1.20
# 1.20.0
ok 2 - An object of class 'GDBM_File' isa 'GDBM_File'
ok 3 - mode 0640
gdbm_firstkey: Item not found at t/failure.t line 30.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 3.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 3 subtests passed 

Test Summary Report
-------------------
../ext/GDBM_File/t/failure.t (Wstat: 65280 Tests: 3 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=3,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.05 cusr  0.01 csys =  0.07 CPU)
Result: FAIL

Thank you for that investigation. It confirms that a stripped-down version of the failing test file can be used for further investigation.

I downloaded https://ftp.gnu.org/gnu/gdbm/gdbm-1.20.tar.gz and manually installed it on FreeBSD-12 underneath /opt/local/gdbm (created for that purpose).

I then configured and built Perl 5 blead as follows and ran the test file:

$ sh ./Configure -des -Dusedevel -Duseithreads \
-Dlocincpth=/opt/local/gdbm/include \
-Dloclibpth=/opt/local/gdbm/lib

$ make

$ ./perl -Ilib -v | head -2 | tail -1
This is perl 5, version 35, subversion 2 (v5.35.2 (v5.35.1-25-gcf70408829)) built for amd64-freebsd-thread-multi

$ ./perl -Ilib -V:config_args
config_args='-des -Dusedevel -Duseithreads -Dlocincpth=/opt/local/gdbm/include -Dloclibpth=/opt/local/gdbm/lib';

$ ./perl -Ilib -Iext/GDBM_File/lib $P5P_DIR/gh-18915-gdbm-failure.t
ok 1 - use GDBM_File;
# GDBM_version: 1.20
# 1.20.0
ok 2 - An object of class 'GDBM_File' isa 'GDBM_File'
ok 3 - mode 0640
gdbm_firstkey: Item not found at $P5P_DIR/gh-18915-gdbm-failure.t line 30.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 3.

So @jplesnki's original failure report is reproducible.

I then did git checkout v5.32.0, configured, built and tested in the same way. I got a different, earlier error in the test file:

$ ./perl -Ilib -Iext/GDBM_File/lib $P5P_DIR/gh-18915-gdbm-failure.t
ok 1 - use GDBM_File;
Usage: GDBM_File::AUTOLOAD() at $P5P_DIR/gh-18915-gdbm-failure.t line 10.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 1.

I then examined the output of git log --format=fuller --reverse v5.32.0..HEAD -- ext/GDBM_File, honed in on some likely commits, and repeated the above process.

Result: the OP's failure was introduced in this commit:

commit 35cde4f7e7f6687315e63f6c984bb94b71564667
Author:     Sergey Poznyakoff <gray@gnu.org>
AuthorDate: Wed Jan 6 10:51:17 2021 +0200
Commit:     Tony Cook <tony@develop-help.com>
CommitDate: Wed Jan 6 23:44:47 2021 +0000

    GDBM_File: minor changes
    
    * ext/GDBM_File/GDBM_File.xs (rcvr_errfun): Use dTHX
    (gdbm_GDBM_version): Don't declare package.
    * ext/GDBM_File/typemap: Map gdbm_count_t to T_UV.
    
    * ext/GDBM_File/t/count.t: Use File::Temp to create db in
    a temporary directory.
    * ext/GDBM_File/t/fatal.t: Likewise.
    * ext/GDBM_File/t/opt.t: Likewise.

As I was studying this, @graygnuorg submitted a pull request which is under evaluation.

Thank you very much.
Jim Keenan

[jkeenan]

This should be fixed with merge of #18924 into blead just now.

Commits in reverse order:

* 5bc1e5fdd8 (HEAD -> blead, origin/blead, origin/HEAD, graygnuorg/gdbm-20210623, graygnuorg-gdbm-20210623-02) Fix definition of ITEM_NOT_FOUND for pre-1.13 versions.
* ea57297a58 Raise version number in ext/GDBM_File/GDBM_File.pm
* aacd2398e7 Fix GDBM_File to compile with version 1.20 and earlier

I will monitor this for several days before closing ticket.

Thank you very much.
Jim Keenan

[jplesnik]

I was able successfully build Perl 5.34 with GDBM 1.20. Thank you for the fix.

[jkeenan]

I was able successfully build Perl 5.34 with GDBM 1.20. Thank you for the fix.

Thanks for the feedback. Closing ticket as per schedule.