panic: top_env from embedded perl using 'do' with unused XS calls

[p5pRT]

Migrated from rt.perl.org#44051 (status was 'open')

Searchable as RT44051$

[p5pRT]

From perlbug@daveola.com

This is a bug report for perl from perlbug@​daveola.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

---

[Please enter your report here]

I'm getting a panic​: top_env

I've managed to reduce the test case to the following​:

  1. From a C program, create/start a perl interpreter object
  2. Load a perl script (using eval_pv "do ...")
  3. Setup an XS routine (which, incidentally, never gets used)
  4. Call a perl routine in the perl script
  5. From that perl routine, execute another perl script (using 'do file')
  6. That perl script needs to define four subs, two which must call the XS routine at least four times, and two which must call it at least once - though again note that none of the routines are called.

The attempt to 'do file' in step 5 causes the panic.

Here's the testcase and Makefile

[p5pRT]

From perlbug@daveola.com

bug.tar.gz

[p5pRT]

From @doy

I can't reproduce this with perl 5.16.0. Is this still an issue?

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From perlbug@daveola.com

On Fri Jun 22 10​:01​:49 2012, doy wrote​:

I can't reproduce this with perl 5.16.0. Is this still an issue?

I (Ubuntu) has finally updated to perl 5.14.2

The original issue is gone with 5.14.2, but it reveals new problems.

If we actually try to call a routine in the eval/do/require'd script
that calls any of the XS functions, then we have a problem.

I've made a simpler example. We have embedded perl that 'eval's another
script that has a function that calls an XS call. If we eval/do, then
when we try to call that function, perl complains about an 'Undefined
subroutine' even though the eval didn't have any errors.

If we change it to a require, then the require complains when it hits
the XS call saying that we've got too many arguments for the call, even
though it's the correct number. If we remove the arguments from the XS
call, then the require works and the XS call is made, but it will
complain about not getting enough arguments.

It seems the XS information is lost across the eval.

I've attached a new example tar.

--
David Ljung Madison
http​://GetDave.com/ http​://MarginalHacks.com/
http​://DaveSource.com/

[p5pRT]

From perlbug@daveola.com

bug.tar.gz

[p5pRT]

From perlbug@daveola.com

On Fri Jun 22 10​:01​:49 2012, doy wrote​:

I can't reproduce this with perl 5.16.0. Is this still an issue?

I (Ubuntu) has finally updated to perl 5.14.2

The original issue is gone with 5.14.2, but it reveals new problems.

If we actually try to call a routine in the eval/do/require'd script
that calls any of the XS functions, then we have a problem.

I've made a simpler example. We have embedded perl that 'eval's another
script that has a function that calls an XS call. If we eval/do, then
when we try to call that function, perl complains about an 'Undefined
subroutine' even though the eval didn't have any errors.

If we change it to a require, then the require complains when it hits
the XS call saying that we've got too many arguments for the call, even
though it's the correct number. If we remove the arguments from the XS
call, then the require works and the XS call is made, but it will
complain about not getting enough arguments.

It seems the XS information is lost across the eval.

I've attached a new example tar.

--
David Ljung Madison
http​://GetDave.com/ http​://MarginalHacks.com/
http​://DaveSource.com/

[p5pRT]

From perlbug@daveola.com

bug.tar.gz

[p5pRT]

From @bulk88

On Sat Nov 03 00​:27​:16 2012, daveola wrote​:

On Fri Jun 22 10​:01​:49 2012, doy wrote​:

I can't reproduce this with perl 5.16.0. Is this still an issue?

I (Ubuntu) has finally updated to perl 5.14.2

The original issue is gone with 5.14.2, but it reveals new problems.

If we actually try to call a routine in the eval/do/require'd script
that calls any of the XS functions, then we have a problem.

I've made a simpler example. We have embedded perl that 'eval's another
script that has a function that calls an XS call. If we eval/do, then
when we try to call that function, perl complains about an 'Undefined
subroutine' even though the eval didn't have any errors.

If we change it to a require, then the require complains when it hits
the XS call saying that we've got too many arguments for the call, even
though it's the correct number. If we remove the arguments from the XS
call, then the require works and the XS call is made, but it will
complain about not getting enough arguments.

It seems the XS information is lost across the eval.

I've attached a new example tar.

--
David Ljung Madison
http​://GetDave.com/ http​://MarginalHacks.com/
http​://DaveSource.com/

I got something else from the code you supplied. On Win32 Perl I got a
crash.
___________________________________________________
  ntdll.dll!_RtlpWaitForCriticalSection@​4() + 0x5b
  ntdll.dll!_RtlEnterCriticalSection@​4() + 0x46

perl517.dll!PerlIOUnix_refcnt_inc(int fd=0) Line 2374 C
  perl517.dll!PerlIOUnix_setfd(interpreter * my_perl=0x008647bc, _PerlIO
* * f=0x00913254, int fd=0, int imode=0) Line 2654 + 0x9 C
  perl517.dll!PerlIOUnix_open(interpreter * my_perl=0x008647bc,
_PerlIO_funcs * self=0x2819ee20, PerlIO_list_s * layers=0x00913104, long
n=0, const char * mode=0x281921d5, int fd=0, int imode=0, int perm=0,
_PerlIO * * f=0x00913254, int narg=0, sv * * args=0x00000000) Line
2735 + 0x15 C
  perl517.dll!PerlIOBuf_open(interpreter * my_perl=0x008647bc,
_PerlIO_funcs * self=0x2819efe0, PerlIO_list_s * layers=0x00913104, long
n=1, const char * mode=0x281921d4, int fd=0, int imode=0, int perm=0,
_PerlIO * * f=0x00000000, int narg=0, sv * * args=0x00000000) Line
3860 + 0x35 C
  perl517.dll!PerlIO_openn(interpreter * my_perl=0x008647bc, const char
* layers=0x00000000, const char * mode=0x281921d4, int fd=0, int
imode=0, int perm=0, _PerlIO * * f=0x00000000, int narg=0, sv * *
args=0x00000000) Line 1649 + 0x32 C
  perl517.dll!PerlIO_fdopen(int fd=0, const char * mode=0x281921d4)
Line 4884 + 0x1d C
  perl517.dll!PerlIO_stdstreams(interpreter * my_perl=0x008647bc) Line
1221 + 0xc C
  perl517.dll!PerlIO_resolve_layers(interpreter * my_perl=0x008647bc,
const char * layers=0x00000000, const char * mode=0x28189294, int
narg=1, sv * * args=0x0012fde4) Line 1539 + 0x9 C
  perl517.dll!PerlIO_openn(interpreter * my_perl=0x008647bc, const char
* layers=0x00000000, const char * mode=0x28189294, int fd=-1, int
imode=0, int perm=0, _PerlIO * * f=0x00000000, int narg=1, sv * *
args=0x0012fde4) Line 1620 + 0x19 C
  perl517.dll!PerlIO_open(const char * path=0x2818a0b0, const char *
mode=0x28189294) Line 4893 + 0x1d C
  perl517.dll!S_open_script(interpreter * my_perl=0x008647bc, const char
* scriptname=0x2818a0b0, char dosearch=0, char * suidscript=0x0012fe53)
Line 3705 + 0xe C
  perl517.dll!S_parse_body(interpreter * my_perl=0x008647bc, char * *
env=0x00000000, void (interpreter *)* xsinit=0x00000000) Line 2062 + 0x15 C
  perl517.dll!perl_parse(interpreter * my_perl=0x008647bc, void
(interpreter *)* xsinit=0x00000000, int argc=3, char * *
argv=0x0012ff50, char * * env=0x00000000) Line 1621 + 0x11 C
  main.exe!main() Line 28 + 0x14 C
  main.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23
___________________________________________________
PL_perlio_mutex is not set up correctly since DebugInfo should be a
pointer not null. DebugInfo is part of a link list debugging thing
Windows does in critical section structs.
___________________________________________________
- PL_perlio_mutex {DebugInfo=0x00000000 {Type=???
CreatorBackTraceIndex=??? CriticalSection=??? ...} LockCount=1
RecursionCount=0 ...} _RTL_CRITICAL_SECTION
+ DebugInfo 0x00000000 {Type=??? CreatorBackTraceIndex=???
CriticalSection=??? ...} _RTL_CRITICAL_SECTION_DEBUG *
  LockCount 1 long
  RecursionCount 0 long
  OwningThread 0x00000000 void *
  LockSemaphore 0x000007bc void *
  SpinCount 0 unsigned long
___________________________________________________

[p5pRT]

From @bulk88

On Wed Nov 07 21​:40​:01 2012, bulk88 wrote​:

I've attached a new example tar.

--
David Ljung Madison
http​://GetDave.com/ http​://MarginalHacks.com/
http​://DaveSource.com/

I got something else from the code you supplied. On Win32 Perl I got a
crash.
___________________________________________________
ntdll.dll!_RtlpWaitForCriticalSection@​4() + 0x5b
ntdll.dll!_RtlEnterCriticalSection@​4() + 0x46

perl517.dll!PerlIOUnix_refcnt_inc(int fd=0) Line 2374 C
perl517.dll!PerlIOUnix_setfd(interpreter * my_perl=0x008647bc, _PerlIO
* * f=0x00913254, int fd=0, int imode=0) Line 2654 + 0x9 C
perl517.dll!PerlIOUnix_open(interpreter * my_perl=0x008647bc,
_PerlIO_funcs * self=0x2819ee20, PerlIO_list_s * layers=0x00913104, long
n=0, const char * mode=0x281921d5, int fd=0, int imode=0, int perm=0,
_PerlIO * * f=0x00913254, int narg=0, sv * * args=0x00000000) Line
2735 + 0x15 C
perl517.dll!PerlIOBuf_open(interpreter * my_perl=0x008647bc,
_PerlIO_funcs * self=0x2819efe0, PerlIO_list_s * layers=0x00913104, long
n=1, const char * mode=0x281921d4, int fd=0, int imode=0, int perm=0,
_PerlIO * * f=0x00000000, int narg=0, sv * * args=0x00000000) Line
3860 + 0x35 C
perl517.dll!PerlIO_openn(interpreter * my_perl=0x008647bc, const char
* layers=0x00000000, const char * mode=0x281921d4, int fd=0, int
imode=0, int perm=0, _PerlIO * * f=0x00000000, int narg=0, sv * *
args=0x00000000) Line 1649 + 0x32 C
perl517.dll!PerlIO_fdopen(int fd=0, const char * mode=0x281921d4)
Line 4884 + 0x1d C
perl517.dll!PerlIO_stdstreams(interpreter * my_perl=0x008647bc) Line
1221 + 0xc C
perl517.dll!PerlIO_resolve_layers(interpreter * my_perl=0x008647bc,
const char * layers=0x00000000, const char * mode=0x28189294, int
narg=1, sv * * args=0x0012fde4) Line 1539 + 0x9 C
perl517.dll!PerlIO_openn(interpreter * my_perl=0x008647bc, const char
* layers=0x00000000, const char * mode=0x28189294, int fd=-1, int
imode=0, int perm=0, _PerlIO * * f=0x00000000, int narg=1, sv * *
args=0x0012fde4) Line 1620 + 0x19 C
perl517.dll!PerlIO_open(const char * path=0x2818a0b0, const char *
mode=0x28189294) Line 4893 + 0x1d C
perl517.dll!S_open_script(interpreter * my_perl=0x008647bc, const char
* scriptname=0x2818a0b0, char dosearch=0, char * suidscript=0x0012fe53)
Line 3705 + 0xe C
perl517.dll!S_parse_body(interpreter * my_perl=0x008647bc, char * *
env=0x00000000, void (interpreter *)* xsinit=0x00000000) Line 2062
+ 0x15 C
perl517.dll!perl_parse(interpreter * my_perl=0x008647bc, void
(interpreter *)* xsinit=0x00000000, int argc=3, char * *
argv=0x0012ff50, char * * env=0x00000000) Line 1621 + 0x11 C
main.exe!main() Line 28 + 0x14 C
main.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@​4() + 0x23
___________________________________________________
PL_perlio_mutex is not set up correctly since DebugInfo should be a
pointer not null. DebugInfo is part of a link list debugging thing
Windows does in critical section structs.
___________________________________________________
- PL_perlio_mutex {DebugInfo=0x00000000 {Type=???
CreatorBackTraceIndex=??? CriticalSection=??? ...} LockCount=1
RecursionCount=0 ...} _RTL_CRITICAL_SECTION
+ DebugInfo 0x00000000 {Type=??? CreatorBackTraceIndex=???
CriticalSection=??? ...} _RTL_CRITICAL_SECTION_DEBUG *
LockCount 1 long
RecursionCount 0 long
OwningThread 0x00000000 void *
LockSemaphore 0x000007bc void *
SpinCount 0 unsigned long
___________________________________________________

The crash on windows is caused by a missing (see perlembed )
"PERL_SYS_INIT3(&argc,&argv,&env);" and "PERL_SYS_TERM();" in main.c .
On threaded Win32 Perl, File IO definitely won't work without those 2
calls. I dont know what those 2 do under the hood on Unix, so that might
explain why there is more subtle behavior on Unix. Try adding those 2 as
described in perlembed and see if your problem goes away. I deleted the
files I used to do the above a while ago and forgot about this post so I
can't test the suggestion to see if it stops crashing on Win32 Perl.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From perlbug@daveola.com

The crash on windows is caused by a missing (see perlembed )
"PERL_SYS_INIT3(&argc,&argv,&env);" and "PERL_SYS_TERM();" in main.c .

I've added those to the second example (the one showing a missing
subroutine) and I still see the problem.

A third tar is attached.

[p5pRT]

From perlbug@daveola.com

bug3.tar.z

[p5pRT]

From perlbug@daveola.com

The crash on windows is caused by a missing (see perlembed )
"PERL_SYS_INIT3(&argc,&argv,&env);" and "PERL_SYS_TERM();" in main.c

It doesn't seem that the docs really explain what PERL_SYS_INIT3 does.
If I don't have the env, can I supply null or perhaps a pointer to an
empty array of char pointers? This code is being called from another
language, so we don't have a main or env in the C code.

[p5pRT]

From [Unknown Contact. See original ticket]

The crash on windows is caused by a missing (see perlembed )
"PERL_SYS_INIT3(&argc,&argv,&env);" and "PERL_SYS_TERM();" in main.c

It doesn't seem that the docs really explain what PERL_SYS_INIT3 does.
If I don't have the env, can I supply null or perhaps a pointer to an
empty array of char pointers? This code is being called from another
language, so we don't have a main or env in the C code.

[p5pRT]

From @bulk88

On Mon Mar 11 21​:08​:22 2013, daveola wrote​:

The crash on windows is caused by a missing (see perlembed )
"PERL_SYS_INIT3(&argc,&argv,&env);" and "PERL_SYS_TERM();" in main.c

It doesn't seem that the docs really explain what PERL_SYS_INIT3 does.
If I don't have the env, can I supply null or perhaps a pointer to an
empty array of char pointers? This code is being called from another
language, so we don't have a main or env in the C code.

Tried your new version. No crashes on 5.17.7. Used (win32 platform)
__________________________________________________________________

  cl -c -nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -GL -G7 -DWIN32
-D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T -MD -Zi -DNDEBUG
-O1 -GL -G7 "-IC​:\perl517\lib\CORE" main.c
  link -out​:main.exe -nologo -nodefaultlib -debug -opt​:ref,icf
-ltcg -libpath​:"c​:\perl517\lib\CORE" -machine​:x86 main.obj
C​:\perl517\lib\CORE\perl517.lib oldnames.lib kernel32.lib user32.lib
gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
__________________________________________________________________
to compile.

Running your code unmodified gave
__________________________________________________________________
C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>main
Evaluate builtin.pl
Setup the XS routine
Call builtinInit
do script.pl
Eval returns​: scriptpl and
Undefined subroutine &main​::fubar0 called at builtin.pl line 5.

C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>
___________________________________________________________________

In builtin.pl you wrote " my $ret = eval 'script.pl';" That doesn't do
what I think you think it does. You probably want a do() or a require().
The error is appropriate since script.pl was never read.

Changing the eval to a require so the line reads, " my $ret = require
'script.pl';", gives
____________________________________________________________________
C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>main
Evaluate builtin.pl
Setup the XS routine
Call builtinInit
do script.pl
Too many arguments for main​::iout at script.pl line 5, near ""Hi")"
Compilation failed in require at builtin.pl line 3.

C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>
____________________________________________________________________
"Too many arguments for " is an error from perl prototypes system.
Looking in main.c I see " newXSproto("iout",XS_iout,__FILE__,"");", if I
change the line to " newXSproto("iout",XS_iout,__FILE__,NULL);" and
recompile, I get
____________________________________________________________________
C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>main
Evaluate builtin.pl
Setup the XS routine
Call builtinInit
do script.pl
Eval returns​: 1 and
in fubar0
HiDone with script.pl
Done

C​:\Documents and Settings\Owner\Desktop\cpan libs\prtbug\bug3>
_____________________________________________________________________

I dont see a C bug in Perl here. You used "newXSproto" and gave a
prototype. If there is a bug, it is the POD for newXSproto at
http​://perldoc.perl.org/perlapi.html isnt listing a C prototype nor
describes the params that func/macro takes (I think that should be fixed).
--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From perlbug@daveola.com

In builtin.pl you wrote " my $ret = eval 'script.pl';"

Now that's humiliating. That's a typo from when I tried to simplify
the case down from the much larger case (which was using a 'do').

Looking in main.c I see "
newXSproto("iout",XS_iout,__FILE__,"");", if I
change the line to " newXSproto("iout",XS_iout,__FILE__,NULL);" and

Thanks for the fix on that.

If there is a bug, it is the POD for newXSproto at
http​://perldoc.perl.org/perlapi.html isnt listing a C prototype nor
describes the params that func/macro takes (I think that should be
fixed).

I agree with that. There's certainly documentation missing. Should a
separate bug be filed, and would it accomplish anything, or is this
probably always going to be under-documented?

Also - it seems near impossible to find out what is actually happening
in PERL_SYS_INIT3 and PERL_SYS_TERM, I've found some Windows code for
them, though I'm on UNIX. And the problem is I don't really think I
have a way to call PERL_SYS_TERM, since it should only be called on
the last ending perl, and I have no way of knowing when that is (the
main loop for this is in code that is not under my control, I'm writing
a C library which uses Perl.

Is it vital that I call PERL_SYS_TERM?

[p5pRT]

From perlbug@daveola.com

I wrote​:

I don't really think I have a way to call PERL_SYS_TERM, since it
should only be called on the last ending perl, and I have no way of
knowing when that is (the main loop for this is in code that is not
under my control, I'm writing a C library which uses Perl.

It seems I am not alone, Jan brings up similar issues with PERL_SYS_TERM​:

 
http​://www.nntp.perl.org/group/perl.perl5.porters/2007/01/msg120081.html

[p5pRT]

From @bulk88

On Tue Mar 12 15​:06​:14 2013, daveola wrote​:

If there is a bug, it is the POD for newXSproto at
http​://perldoc.perl.org/perlapi.html isnt listing a C prototype nor
describes the params that func/macro takes (I think that should be
fixed).

I agree with that. There's certainly documentation missing. Should a
separate bug be filed, and would it accomplish anything, or is this
probably always going to be under-documented?

It should not always be under-documented. A git patch is welcome to the
POD. Some things to do is look at newXS, see what it says about its
params. Look at the source code for newXS_flags (???), see if newXS and
newXSproto need more things added to their POD from looking at their
source. Like who owns the CV * newXS returns? Do you have to do a
refcnt_dec on it? (no).

Also - it seems near impossible to find out what is actually happening
in PERL_SYS_INIT3 and PERL_SYS_TERM, I've found some Windows code for
them, though I'm on UNIX. And the problem is I don't really think I
have a way to call PERL_SYS_TERM, since it should only be called on
the last ending perl, and I have no way of knowing when that is (the
main loop for this is in code that is not under my control, I'm writing
a C library which uses Perl.

Is it vital that I call PERL_SYS_TERM?

Yes. What it does is different on every OS. Basically it shuts down the
per process (not per interp) perl systems. INIT starts up the per
process stuff. Call each only once per process regardless of number
interps created/destroyed.

Make a C static counter. Each interp increases it, each interp decreases
it. Mutex it or use atomic ++/__ funcs. When it hits zero do the term.
If you go from 0 to 1, do an INIT.
--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From perlbug@daveola.com

It should not always be under-documented. A git patch is welcome to the
POD. Some things to do is look at newXS, see what it says about its
params. Look at the source code for newXS_flags (???), see if newXS and

If I understood the perlembed system I would happily do that patch,
but clearly I'm not the person to ask.

Make a C static counter... If you go from 0 to 1, do an INIT.

Is it okay to do a PERL_SYS_TERM if later other perl interpreters are
created (following a new PERL_SYS_INIT3?)