Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GDBM_FILE: gdbm_open requires a blocksize to be a power of two #13232

Closed
p5pRT opened this issue Sep 5, 2013 · 11 comments
Closed

GDBM_FILE: gdbm_open requires a blocksize to be a power of two #13232

p5pRT opened this issue Sep 5, 2013 · 11 comments

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Sep 5, 2013

Migrated from rt.perl.org#119623 (status was 'resolved')

Searchable as RT119623$

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 5, 2013

From @doughera88

Created by @doughera88

ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0.
This instructs gdbm to use the stat block size for the file system.
Unfortunately, gdbm will fail if that stat block size is not a power
of two. See [perl #119537] for some more details.

Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we
should retry with a 'default' blocksize, something simple like 1024. Or,
perhaps GDBM_File.xs ought to do something like stat(dirname(name)) and
then work out the "closest" power-of-two. (Note, though,that dirname()
may alter "name", so we'd have to first make a copy and then free it.)

[config info below is irrelvant]

Perl Info

Flags:
    category=library
    severity=low
    module=GDBM_File

Site configuration information for perl 5.18.0:

Configured by doughera at Mon May 13 07:10:40 EDT 2013.

Summary of my perl5 (revision 5 version 18 subversion 0) configuration:
   
  Platform:
    osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux
    uname='linux xxx 3.2.0-4-amd64 #1 smp debian 3.2.41-2 x86_64 gnulinux '
    config_args='-Dprefix=/opt/perl -Dscriptdir=/opt/perl/bin -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.7.2', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
    libs=-lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.13'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Locally applied patches:
    RC2


@INC for perl 5.18.0:
    /opt/perl/lib/site_perl/5.18.0/x86_64-linux
    /opt/perl/lib/site_perl/5.18.0
    /opt/perl/lib/5.18.0/x86_64-linux
    /opt/perl/lib/5.18.0
    /opt/perl/lib/site_perl/5.16.2
    /opt/perl/lib/site_perl/5.16.0
    /opt/perl/lib/site_perl/5.14.2
    /opt/perl/lib/site_perl/5.12.3
    /opt/perl/lib/site_perl/5.10.1
    /opt/perl/lib/site_perl/5.8.9
    /opt/perl/lib/site_perl
    .


Environment for perl 5.18.0:
    HOME=/home/doughera
    LANG (unset)
    LANGUAGE (unset)
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH=/usr/local/lib
    LOGDIR (unset)
    PATH=/home/doughera/script:/home/doughera/bin:/opt/perl/bin:/opt/dist/bin:/opt/gnuplot/bin:/opt/groff/bin:/usr/local/script:/usr/local/bin:/opt/dist/bin:/bin:/usr/bin:/usr/X11R6/bin:.
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 12, 2013

From @jkeenan

On Thu Sep 05 06​:46​:45 2013, doughera wrote​:

This is a bug report for perl from doughera@​lafayette.edu,
generated with the help of perlbug 1.39 running under perl 5.18.0.

-----------------------------------------------------------------
[Please describe your issue here]

ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0.
This instructs gdbm to use the stat block size for the file system.
Unfortunately, gdbm will fail if that stat block size is not a power
of two. See [perl #119537] for some more details.

Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we
should retry with a 'default' blocksize, something simple like 1024.
Or,
perhaps GDBM_File.xs ought to do something like stat(dirname(name))
and
then work out the "closest" power-of-two. (Note, though,that
dirname()
may alter "name", so we'd have to first make a copy and then free it.)

Andy,

Do you have any suggestions as to how we would test for the presence of this problem?

I think that would be a necessary first step toward a solution.

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 12, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 12, 2013

From @doughera88

On Wed, Dec 11, 2013 at 04​:06​:08PM -0800, James E Keenan via RT wrote​:

On Thu Sep 05 06​:46​:45 2013, doughera wrote​:

This is a bug report for perl from doughera@​lafayette.edu,
generated with the help of perlbug 1.39 running under perl 5.18.0.

-----------------------------------------------------------------
[Please describe your issue here]

ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0.
This instructs gdbm to use the stat block size for the file system.
Unfortunately, gdbm will fail if that stat block size is not a power
of two. See [perl #119537] for some more details.

Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we
should retry with a 'default' blocksize, something simple like 1024.
Or,
perhaps GDBM_File.xs ought to do something like stat(dirname(name))
and
then work out the "closest" power-of-two. (Note, though,that
dirname()
may alter "name", so we'd have to first make a copy and then free it.)

Andy,

Do you have any suggestions as to how we would test for the presence of this problem?

I think that would be a necessary first step toward a solution.

This problem only manifests itself on certain filesystems, which might
not match the filesystem on which perl is built and tested.
I don't see an easy and reliable way to test for it. I think that
checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then
retrying with 1024 would probably be good enough.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 31, 2014

From perl-diddler@tlinx.org

On Wed Dec 11 18​:56​:53 2013, doughera wrote​:

On Wed, Dec 11, 2013 at 04​:06​:08PM -0800, James E Keenan via RT wrote​:

Andy,

Do you have any suggestions as to how we would test for the presence
of this problem?

I think that would be a necessary first step toward a solution.

This problem only manifests itself on certain filesystems, which might
not match the filesystem on which perl is built and tested.
I don't see an easy and reliable way to test for it. I think that
checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then
retrying with 1024 would probably be good enough.


It may be doable with other file sytems, but xfs has parameters to
specify the strip size and width. I have stripes of 64K and had
a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe
size, with 4 data disks -- total width 256k. The striping of
those in a RAID0 with 3 units gave me the odd "optimal i/o size
of 3x256 => 756K.

To work around the problem I gave up on exposing the larger
optimal I/O and went with a 256K.

The linux "mkfs.xfs" Could be used to construct a virtual
drive on a ram disk or in a file. Wrote a quick bash script
to create such a disk on linux (assumes you are running as
root). It's a quick and simple script but should be
easily modified or used as a ref to build something more
complex...(attached)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 31, 2014

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 31, 2014

From @doughera88

On Thu, Jan 30, 2014 at 10​:20​:24PM -0800, Linda Walsh via RT wrote​:

On Wed Dec 11 18​:56​:53 2013, doughera wrote​:

On Wed, Dec 11, 2013 at 04​:06​:08PM -0800, James E Keenan via RT wrote​:

Andy,

Do you have any suggestions as to how we would test for the presence
of this problem?

I think that would be a necessary first step toward a solution.

This problem only manifests itself on certain filesystems, which might
not match the filesystem on which perl is built and tested.
I don't see an easy and reliable way to test for it. I think that
checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then
retrying with 1024 would probably be good enough.
-----------

It may be doable with other file sytems, but xfs has parameters to
specify the strip size and width. I have stripes of 64K and had
a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe
size, with 4 data disks -- total width 256k. The striping of
those in a RAID0 with 3 units gave me the odd "optimal i/o size
of 3x256 => 756K.

To work around the problem I gave up on exposing the larger
optimal I/O and went with a 256K.

The linux "mkfs.xfs" Could be used to construct a virtual
drive on a ram disk or in a file. Wrote a quick bash script
to create such a disk on linux (assumes you are running as
root). It's a quick and simple script but should be
easily modified or used as a ref to build something more
complex...(attached)

Thank you, but I don't know what you imagine we should do with this
script. I can't imagine including it in the perl test suite, and a
developer exploring a fix can reproduce the problem by simply redefining
GDBM_BLOCKSIZE in ext/GDBM_File.xs to something other than a power of 2.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 16, 2017

From zefram@fysh.org

Fixed in commit d1500e1.

-zefram

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 17, 2017

@xsawyerx - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 23, 2018

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 23, 2018

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.