Perl incorrectly limits maxmimum number of open files

[p5pRT]

Migrated from rt.perl.org#1826 (status was 'resolved')

Searchable as RT1826$

[p5pRT]

From jklein@eden.alerts.co.il

I'm working on a Solaris machine, and using ulimit, I upped the maximum
number of file descriptors to 700. I then ran a test program using open
and found that open() will only allow me to have up to 256 files open at a
time.

Okay, I thought "Perl open() is based on C's fopen() which only supports
fds up to 255".

So I tried FileHandle and found that it also failed above 256; I can live
with that.

With supreme confidence, I decided to run the test with sysopen. Imagine
my surprise when I found that still only 256 open fds were supported!!!

This seems to me to be a major bug. I searched the archives and looked
through the perlfaq, but there is no reference to this anywhere. Is this a
real problem or am I missing something very obvious?

PS I ran my program using truss and verified that the system call open()s
were succeeding and that Perl was deciding to fail the Perl sysopen()
call nonetheless.

Below is the program that I used to test this. The 3 open() lines are
mutually exclusive - ie. only uncomment one at a time.

Any help would be greatly appreciated,
Thanks,
Yossi Klein

---------------------

#!/usr/local/bin/perl

use FileHandle;
use Fcntl;

$i = 0;
while ($i < 350)
{
  $fd = "fd$i";
  $fname = '/dev/null';

  #$rc = open ($fd, "<$fname");

  #$rc = ($ffh[$i] = new FileHandle)->open($fname, "r");

  #$rc = sysopen($fd, $fname, O_RDONLY);

  if (!$rc)
  {
  die "open $i failed\n";
  }

  $i++;
}

Perl Info

Site configuration information for perl 5.00404:

Configured by jklein at Tue Dec 16 11:37:16 IST 1997.

Summary of my perl5 (5.0 patchlevel 4 subversion 4) configuration:
  Platform:
    osname=solaris, osvers=2.4, archname=sun4-solaris
    uname='sunos abel.alerts.co.il 5.4 generic_101945-41 sun4m sparc '
    hint=recommended, useposix=true, d_sigaction=define
    bincompat3=y useperlio=undef d_sfio=undef
  Compiler:
    cc='cc', optimize='-O', gccversion=
    cppflags='-I/usr/local/include'
    ccflags ='-I/usr/local/include'
    stdchar='unsigned char', d_stdstdio=define, usevfork=false
    voidflags=15, castflags=0, d_casti32=define, d_castneg=define
    intsize=4, alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib -L/opt/gnu/lib'
    libpth=/usr/local/lib /opt/gnu/lib /lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldb -ldl -lm -lc -lcrypt
    libc=/lib/libc.so, so=so
    useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-Kpic', lddlflags='-G -L/usr/local/lib -L/opt/gnu/lib'

Locally applied patches:
	


@INC for perl 5.00404:
	/usr/local/lib/perl5/sun4-solaris/5.00404
	/usr/local/lib/perl5
	/usr/local/lib/perl5/site_perl/sun4-solaris
	/usr/local/lib/perl5/site_perl
	/usr/local/lib/perl5/sun4-solaris
	.


Environment for perl 5.00404:
    HOME=/home/jklein
    LANG=en_US
    LC_COLLATE=en_US
    LC_CTYPE=en_US
    LC_MESSAGES=C
    LC_MONETARY=en_US
    LC_NUMERIC=en_US
    LC_TIME=en_US
    LD_LIBRARY_PATH=/usr/lib:/usr/dt/lib:/usr/ccs/lib:/opt/SUNWmotif/lib:/usr/openwin/lib:/opt/pure/cache/usr/lib:/home/jklein/rvplayer5.0:/news3.0/lib:/news3.0/sun7/lib
    LOGDIR (unset)
    PATH=/home/jklein/bin:/usr/local/bin:/usr/local/etc:/usr/lib/lp/postscript:/usr/lib/nis:/opt/gnu/bin:/bin:/usr/bin:/usr/ccs/bin:/usr/sbin:/opt/SUNWconn/bin:/opt/SUNWmotif/bin:/usr/openwin/bin:/usr/openwin/demo:/usr/bin/X11:/opt/SUNWspro/bin:/etc:/usr/etc:/usr/hosts:.:/usr/lang:/bin:/usr/bin:/usr/5bin:/usr/local/sybase/bin:/usr/snm/bin:/usr/local/frame/bin:/home/ddts/bin:/tools/java/bin:/opt/oracle7/bin:/opt/webalert/etc:/home/jklein/rvplayer5.0:/news3.0/bin:/news3.0/sun7/bin:/opt/news/bin
    PERL_BADLANG (unset)
    SHELL=/bin/ksh

[p5pRT]

From @chipdude

According to Yossi Klein​:

With supreme confidence, I decided to run the test with sysopen. Imagine
my surprise when I found that still only 256 open fds were supported!!!

Perhaps it's not Perl's fault. Perhaps your C library's fopen() and
fdopen(), that don't like large fd values. Try a C test program, and
let us know.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
  "Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From [Unknown Contact. See original ticket]

I know that fopen and fdopen don't support fds higher than 255; that's a
limitation of the FILE structure. sysopen(), however, calls open()
directly (PP2 - Page 229 as well as lots of other places) and as I said in
my email I've verified that the open() system calls beyond fd 256 are
succeeding, but for some reason Perl still fails the sysopen().

Yossi

On Wed, 24 Nov 1999, Chip Salzenberg wrote​:

According to Yossi Klein​:

With supreme confidence, I decided to run the test with sysopen. Imagine
my surprise when I found that still only 256 open fds were supported!!!

Perhaps it's not Perl's fault. Perhaps your C library's fopen() and
fdopen(), that don't like large fd values. Try a C test program, and
let us know.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
"Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From @chipdude

According to Yossi Klein​:

I know that fopen and fdopen don't support fds higher than 255; that's a
limitation of the FILE structure. sysopen(), however, calls open()
directly (PP2 - Page 229 as well as lots of other places) ....

But it then proceeds to call fdopen(). I'm quite sure of that.
I wrote sysopen(), you know. :-)

If you need more files open than 256, I suggest you (1) install and
use sfio, or else (2) use POSIX​::open and manipulate fds by hand.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
  "Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From [Unknown Contact. See original ticket]

Well in that case...

But, I still think that it's at least a documentation bug. There's nowhere
in any documenttation that I've seen (except for your email) that
indicates that sysopen() is limited by fdopen().

By the way, part of the problem I hit is that the EventServer.pm module
uses fileno() in order to validate that an argument to a callback function
is a valid fd. The fd is created via a Perl extension and happens to be
256, but EventServer.pm croaks because fileno(256) returns failure. I
guess technically this may be considered a bug in EventServer.pm. Does all
this mean that the Perl select() (you know which one :-) would also be
upset if given fds higher than 255?

Personally, I still think that it's a bug that sysopen() relies on
fdopen() and therefore is limited by 256, but since I guess no one else
has complained (and I don't have to guts or brains to even attempt any
type of a patch), I'll live with it. Maybe I'll just have to patch
EventServer.pm

Yossi

On Wed, 24 Nov 1999, Chip Salzenberg wrote​:

According to Yossi Klein​:

I know that fopen and fdopen don't support fds higher than 255; that's a
limitation of the FILE structure. sysopen(), however, calls open()
directly (PP2 - Page 229 as well as lots of other places) ....

But it then proceeds to call fdopen(). I'm quite sure of that.
I wrote sysopen(), you know. :-)

If you need more files open than 256, I suggest you (1) install and
use sfio, or else (2) use POSIX​::open and manipulate fds by hand.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
"Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From [Unknown Contact. See original ticket]

Chip Salzenberg <chip@​valinux.com> writes​:

According to Yossi Klein​:

With supreme confidence, I decided to run the test with
sysopen. Imagine my surprise when I found that still only 256 open fds
were supported!!!

Perhaps it's not Perl's fault. Perhaps your C library's fopen() and
fdopen(), that don't like large fd values. Try a C test program, and
let us know.

You have to use open() directly to access fds higher than 255 on Solaris,
but open() itself will work fine. It sounds like somewhere Perl is trying
to to stdio things with sysopen-opened file descriptors. That seems
wrong, but maybe it was intentional.

--
Russ Allbery (rra@​stanford.edu) <URL​:http​://www.eyrie.org/~eagle/>

[p5pRT]

From @chipdude

According to Yossi Klein​:

There's nowhere in any documenttation that I've seen (except for
your email) that indicates that sysopen() is limited by fdopen().

You're right, that should be documented. Would you please suggests a
sentence or two that would explain what you needed to know?

Personally, I still think that it's a bug that sysopen() relies on
fdopen() and therefore is limited by 256 [...]

But there is a way around it. Just start using sfio, like I
suggested. That bypasses your C library -- and please keep in
mind that you have come up against a KNOWN BUG in the C LIBRARY,
and Perl can't fix *everything* that is broken by other people.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
  "Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From [Unknown Contact. See original ticket]

On Wed, 24 Nov 1999, Chip Salzenberg wrote​:

According to Yossi Klein​:

There's nowhere in any documenttation that I've seen (except for
your email) that indicates that sysopen() is limited by fdopen().

You're right, that should be documented. Would you please suggests a
sentence or two that would explain what you needed to know?

I'm not eloquent, but how about something like​:

Please note that that sysopen relies on the fdopen() C library call.
fdopen() is known to fail beyond file descriptor 255 on many UNIX
systems. If you need more file descriptors than that, consider using
sfio() or POSIX​::open().

Personally, I still think that it's a bug that sysopen() relies on
fdopen() and therefore is limited by 256 [...]

But there is a way around it. Just start using sfio, like I
suggested. That bypasses your C library -- and please keep in
mind that you have come up against a KNOWN BUG in the C LIBRARY,
and Perl can't fix *everything* that is broken by other people.

I've been working in C so long that I don't even think of this as a bug,
but rather a limitation of the fopen-type calls. However, you do have a
valid point. Now, I just need a way to establish the validity of a file
descriptor greater than 255. Hopefully, select() will work. I'll write a
test program after sending out this email so you can get to sleep. (After
all, it IS after midnight in California although it's only 10AM here in
Israel :-).

Thanks for your valuable feedback.

Yossi Klein
News Alert LLC
(www.newsalert.com)

[p5pRT]

From @chipdude

According to Yossi Klein​:

Please note that that sysopen relies on the fdopen() C library call.
fdopen() is known to fail beyond file descriptor 255 on many UNIX
systems. If you need more file descriptors than that, consider using
sfio() or POSIX​::open().

Consider it added. Thanks. (BTW, sfio is a library, so its name
doesn't need parens.)

I've been working in C so long that I don't even think of this as a bug,
but rather a limitation of the fopen-type calls.

A feature is often a bug with seniority. :-(

Now, I just need a way to establish the validity of a file
descriptor greater than 255. Hopefully, select() will work.

It might, but I suggest scalar(POSIX​::fstat($fd)) . Simpler to use.

Thanks for your valuable feedback.

My pleasure.
--
Chip Salzenberg - a.k.a. - <chip@​valinux.com>
  "Fleagal. Bingo. Drooper. Snork. They're cops." //MST3K

[p5pRT]

From [Unknown Contact. See original ticket]

Yossi Klein <jklein@​eden.alerts.co.il> writes​:

This is a bug report for perl from jklein@​alerts.co.il,
generated with the help of perlbug 1.20 running under perl 5.00404.

With supreme confidence, I decided to run the test with sysopen. Imagine
my surprise when I found that still only 256 open fds were supported!!!

That is because sysopen() still needs a FILE * (when PerlIO == stdio)
for house keeping. POSIX​::open() should work, with its associated
read/write/close.

--
Nick Ing-Simmons <nik@​tiuk.ti.com>
Via, but not speaking for​: Texas Instruments Ltd.

[p5pRT]

From [Unknown Contact. See original ticket]

On Thu, Nov 25, 1999 at 09​:45​:56AM +0200, jklein@​eden.alerts.co.il wrote​:

By the way, part of the problem I hit is that the EventServer.pm module
uses fileno() in order to validate that an argument to a callback function
is a valid fd. The fd is created via a Perl extension and happens to be
256, but EventServer.pm croaks because fileno(256) returns failure. I
guess technically this may be considered a bug in EventServer.pm. Does all
this mean that the Perl select() (you know which one :-) would also be
upset if given fds higher than 255?

You're using EventServer.pm? Last time I checked, that modules was
hopelessly out of date. Have you taken a look at Event? I'm pretty
sure that Event can handle as many open files as the OS supports.

Hope this helps.

--
"Does `competition' have an abstract purpose?"
  via, but not speaking for Deutsche Bank