MY_CXT macros and modules with multiple object files

[p5pRT]

Migrated from rt.perl.org#20609 (status was 'open')

Searchable as RT20609$

[p5pRT]

From @salva

Created by @salva

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

  "static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

Bye,

- Salva

Perl Info

Flags:
    category=library
    severity=low

Site configuration information for perl v5.8.0:

Configured by salva at Fri Jan 24 21:02:39 GMT 2003.

Summary of my perl5 (revision 5.0 version 8 subversion 0)
configuration:
  Platform:
    osname=linux, osvers=2.4.20,
archname=i686-linux-thread-multi
    uname='linux cabo 2.4.20 #9 sat jan 11 13:16:56 gmt
2003 i686 unknown '
    config_args='-Dusethreads'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef
useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=undef
usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE
-DDEBUGGING -fno-strict-aliasing -I/usr/local/include',
    optimize='-g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='2.95.4 20011002 (Debian
prerelease)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8,
byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=4
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
    libc=/lib/libc-2.2.5.so, so=so, useshrplib=true,
libperl=libperl.so
    gnulibc_version='2.2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-rdynamic
-Wl,-rpath,/usr/local/perl/threads/lib/5.8.0/i686-linux-thread-multi/CORE'
    cccdlflags='-fpic', lddlflags='-shared
-L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.0:
   
/usr/local/perl/threads/lib/5.8.0/i686-linux-thread-multi
    /usr/local/perl/threads/lib/5.8.0
   
/usr/local/perl/threads/lib/site_perl/5.8.0/i686-linux-thread-multi
    /usr/local/perl/threads/lib/site_perl/5.8.0
    /usr/local/perl/threads/lib/site_perl
    .


Environment for perl v5.8.0:
    HOME=/home/salva
    LANG=C
    LANGUAGE (unset)
    LC_COLLATE=es_ES
    LC_CTYPE=es_ES
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
   
PATH=/usr/local/perl/threads/bin:~/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/local/mysql/bin:~svn/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @jkeenan

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

"static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

This post has not received a response since it was filed almost nine
years ago. grepping the source code suggests that the only module in
the core distribution which *might* qualify as having multiple .xs files
is Dynaloader.

Is there an XS-expert who could evaluate the poster's argument?

Thank you very much.
Jim Keenan

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @tonycoz

On Thu, Jan 19, 2012 at 07​:06​:40PM -0800, James E Keenan via RT wrote​:

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

"static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

This post has not received a response since it was filed almost nine
years ago. grepping the source code suggests that the only module in
the core distribution which *might* qualify as having multiple .xs files
is Dynaloader.

Is there an XS-expert who could evaluate the poster's argument?

It is a real (if uncommon) issue, and the solution makes sense.

It could be a problem for both multiple .xs files, but also if you use
API functions from other .c files in general.

Tony

[p5pRT]

From @tsee

On 01/20/2012 05​:40 AM, Tony Cook wrote​:

On Thu, Jan 19, 2012 at 07​:06​:40PM -0800, James E Keenan via RT wrote​:

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

"static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

This post has not received a response since it was filed almost nine
years ago. grepping the source code suggests that the only module in
the core distribution which *might* qualify as having multiple .xs files
is Dynaloader.

Is there an XS-expert who could evaluate the poster's argument?

It is a real (if uncommon) issue, and the solution makes sense.

It could be a problem for both multiple .xs files, but also if you use
API functions from other .c files in general.

That is my feeling, too. To wit, I think at least on every prolific XS
module author commonly uses multiple XS files. I think it was Florian.

--Steffen

[p5pRT]

From @bulk88

Glib has multiple XS files that compile to 1 dll.

[p5pRT]

From @craigberry

On Fri, Jan 20, 2012 at 12​:57 AM, Steffen Mueller <smueller@​cpan.org> wrote​:

On 01/20/2012 05​:40 AM, Tony Cook wrote​:

On Thu, Jan 19, 2012 at 07​:06​:40PM -0800, James E Keenan via RT wrote​:

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

  "static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

This post has not received a response since it was filed almost nine
years ago.  grepping the source code suggests that the only module in
the core distribution which *might* qualify as having multiple .xs files
is Dynaloader.

Is there an XS-expert who could evaluate the poster's argument?

It is a real (if uncommon) issue, and the solution makes sense.

It could be a problem for both multiple .xs files, but also if you use
API functions from other .c files in general.

That is my feeling, too. To wit, I think at least on every prolific XS
module author commonly uses multiple XS files. I think it was Florian.

I *think* I understand the question but I don't understand the
problem. If the OP is running a non-threaded perl, why does he care
what the thread context is? When you go to retrieve thread context in
a non-threaded perl you will get​:

% ./perl -Ilib Porting/expand-macro.pl dMY_CXT
`sh cflags "optimize='-O3'" try.c` -E try.c > try.i
  CCCMD = cc -DPERL_CORE -c -fno-common -DPERL_DARWIN
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include -std=c89 -O3 -Wall -ansi -W -Wextra
-Wdeclaration-after-statement -Wendif-labels -Wc++-compat
-Wwrite-strings
# 4 "dMY_CXT expands to"
extern int Perl___notused(void)

which I certainly hope returns the same answer when called from two
different C or XS files. It's actually in a perl built with threads
that these docs in perlxs.pod would be applicable​:

"Note that these macros will only work together within the same source
file; that is, a dMY_CTX in one source file will access a different
structure than a dMY_CTX in another source file."

I believe it might be possible to pass thread context around with
aMY_CXT/pMY_CXT between different source files but I've never tried
it. Interested parties might want to also look at what
PERL_GLOBAL_STRUCT does with interpreter context, specifically with
making it globally available.

[p5pRT]

From @salva

Hi, is nice to see that this issue is getting some attention!

________________________________
From​: Craig Berry via RT <perlbug-followup@​perl.org>
To​: sfandino@​yahoo.com
Sent​: Friday, January 20, 2012 4​:33 PM
Subject​: Re​: [perl #20609] MY_CXT macros and modules with multiple objectfiles

On Fri, Jan 20, 2012 at 12​:57 AM, Steffen Mueller <smueller@​cpan.org> wrote​:

On 01/20/2012 05​:40 AM, Tony Cook wrote​:

On Thu, Jan 19, 2012 at 07​:06​:40PM -0800, James E Keenan via RT wrote​:

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

  "static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

This post has not received a response since it was filed almost nine
years ago.  grepping the source code suggests that the only module in
the core distribution which *might* qualify as having multiple .xs files
is Dynaloader.

Is there an XS-expert who could evaluate the poster's argument?

It is a real (if uncommon) issue, and the solution makes sense.

It could be a problem for both multiple .xs files, but also if you use
API functions from other .c files in general.

That is my feeling, too. To wit, I think at least on every prolific XS
module author commonly uses multiple XS files. I think it was Florian.

I *think* I understand the question but I don't understand the
problem.  If the OP is running a non-threaded perl, why does he care
what the thread context is?

I cared because I wanted my module to support both threaded and non-threaded perls.

Under threaded perls, MY_CTX becomes a function call that gets the context from local thread storage and so it is accesible from any compilation unit (.xs and .c files).

On the other hand, under non-threaded perls, MY_CTX becomes an alias for the C structure declared as static by the START_MY_CXT macro, and so it is not accesible outside the compilation unit where it is declared (usually the .xs file).

% ./perl -Ilib Porting/expand-macro.pl dMY_CXT

`sh  cflags "optimize='-O3'" try.c` -E try.c > try.i
      CCCMD =  cc -DPERL_CORE -c -fno-common -DPERL_DARWIN
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include  -std=c89 -O3  -Wall -ansi -W -Wextra
-Wdeclaration-after-statement -Wendif-labels -Wc++-compat
-Wwrite-strings
# 4 "dMY_CXT expands to"
extern int Perl___notused(void)

which I certainly hope returns the same answer when called from two
different C or XS files.  It's actually in a perl built with threads
that these docs in perlxs.pod would be applicable​:

"Note that these macros will only work together within the same source
file; that is, a dMY_CTX in one source file will access a different
structure than a dMY_CTX in another source file."

I believe it might be possible to pass thread context around with
aMY_CXT/pMY_CXT between different source files but I've never tried
it.  Interested parties might want to also look at what

I don't think so.

Under non-threaded perls, aMY_CXT and pMY_CXT are defined as...

#  define pMY_CXT               void
#  define pMY_CXT_
#  define _pMY_CXT
#  define aMY_CXT
#  define aMY_CXT_
#  define _aMY_CXT

so the non visibility issue of the static structure remains.

[p5pRT]

From @craigberry

On Fri, Jan 20, 2012 at 11​:45 AM, Salvador Fandino <sfandino@​yahoo.com> wrote​:

Hi, is nice to see that this issue is getting some attention!

________________________________
From​: Craig Berry via RT <perlbug-followup@​perl.org>
To​: sfandino@​yahoo.com
Sent​: Friday, January 20, 2012 4​:33 PM
Subject​: Re​: [perl #20609] MY_CXT macros and modules with multiple
objectfiles

On Fri, Jan 20, 2012 at 12​:57 AM, Steffen Mueller <smueller@​cpan.org> wrote​:

On 01/20/2012 05​:40 AM, Tony Cook wrote​:

On Thu, Jan 19, 2012 at 07​:06​:40PM -0800, James E Keenan via RT wrote​:

On Wed Jan 29 11​:14​:24 2003, salva wrote​:

This is a bug report for perl from sfandino@​yahoo.com,
generated with the help of perlbug 1.34 running under perl
v5.8.0.

The MY_CXT family of macros to add local thread storage
to modules are designed to be used from a single object
(*.o) file.

i.e., imagine an extension module with two files file1.xs
and file2.xs, and that both modules need to access MY_CXT.

When the extension is compiled in a non-threaded perl,
MY_CXT gets defined as my_cxt, and my_cxt is

  "static my_cxt_t *my_cxt"

in file1.xs and so as it is "static" it is not accessible
from file2.c

To correct it, my_cxt should not be defined as static
anymore and a new macro has to be added to declare it
in other objects.

I *think* I understand the question but I don't understand the
problem.  If the OP is running a non-threaded perl, why does he care
what the thread context is?

I cared because I wanted my module to support both threaded and non-threaded
perls.

All of the thread-related macros are designed to support that, but
perhaps not in the way that you want.

Under threaded perls, MY_CTX becomes a function call that gets the context
from local thread storage and so it is accesible from any compilation unit
(.xs and .c files).

On the other hand, under non-threaded perls, MY_CTX becomes an alias for the
C structure declared as static by the START_MY_CXT macro, and so it is not
accesible outside the compilation unit where it is declared (usually the .xs
file).

That is true, but it usually doesn't matter. The thread-related
macros, as far as I understand them, are designed primarily to do
absolutely nothing in a non-threaded perl (but to do nothing safely,
which can be tricky).

But you want them to do something. I'm guessing you want to store
module-specific global data in the MY_CXT struct, which you are more
or less invited to do in the Three Blind Mice example here​:

<http​://perldoc.perl.org/perlxs.html#Safely-Storing-Static-Data-in-XS>

I hadn't been aware the docs encouraged people to make up their own
my_cxt_t structs. It sounds like what you want is something like
PERL_IMPLICIT_CONTEXT or PERL_GLOBAL_STRUCT, but without threads. I
have no idea whether that works or could be readily made to work.

[p5pRT]

From @bulk88

On Fri Jan 20 09​:46​:08 2012, salva wrote​:

I cared because I wanted my module to support both threaded and non-
threaded perls.

Under threaded perls, MY_CTX becomes a function call that gets the
context from local thread storage and so it is accesible from any
compilation unit (.xs and .c files).

Wrong, threaded MY_CXT STILL uses static data. 2 different compilation units will wind up with different my_cxt_index values which will translate to 2 different "my_cxt_t *" at the end. The compilation unit WITHOUT the "MY_CXT_INIT;" (also means the compilation unit without BOOT​:), will probably SEGV due to the -1.

from perl.h

---

# define START_MY_CXT static int my_cxt_index = -1;

---

On the other hand, under non-threaded perls, MY_CTX becomes an alias
for the C structure declared as static by the START_MY_CXT macro, and
so it is not accesible outside the compilation unit where it is
declared (usually the .xs file).

% ./perl -Ilib Porting/expand-macro.pl dMY_CXT

`sh  cflags "optimize='-O3'" try.c` -E try.c > try.i
      CCCMD =  cc -DPERL_CORE -c -fno-common -DPERL_DARWIN
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include  -std=c89 -O3  -Wall -ansi -W -Wextra
-Wdeclaration-after-statement -Wendif-labels -Wc++-compat
-Wwrite-strings
# 4 "dMY_CXT expands to"
extern int Perl___notused(void)

which I certainly hope returns the same answer when called from two
different C or XS files.  It's actually in a perl built with threads
that these docs in perlxs.pod would be applicable​:

"Note that these macros will only work together within the same
source
file; that is, a dMY_CTX in one source file will access a different
structure than a dMY_CTX in another source file."

I believe it might be possible to pass thread context around with
aMY_CXT/pMY_CXT between different source files but I've never tried
it.  Interested parties might want to also look at what

I don't think so.

Under non-threaded perls, aMY_CXT and pMY_CXT are defined as...

#  define pMY_CXT               void
#  define pMY_CXT_
#  define _pMY_CXT
#  define aMY_CXT
#  define aMY_CXT_
#  define _aMY_CXT

so the non visibility issue of the static structure remains.

You will have to declare an extern getter C function that does "dMY_CXT; return MY_CXT;" defined in the compilation unit with BOOT​:. If your compiler is advanced enough, ( http​://en.wikipedia.org/wiki/Interprocedural_optimization ) it might get auto-inlined. If on threaded MY_CXT, and if my_cxt_index is NOT declared static, there will be a disaster on ELF platforms due to my_cxt_index being an extern symbol and all XS modules, static linked or SO, that use MY_CXT, will be broken. The only solution would be for my_cxt_index to have a C symbol name compatible prefix unique to each XSPP based on "PACKAGE = " (or "MODULE = ", on a per binary/SO basis). A long long time ago (< ~5.9.3) before the more efficient current implementation ( as of http​://perl5.git.perl.org/perl.git/commitdiff/f16dd614412ea67a8eb64bb09a88fccdbd9db6b6?hp=85ce96a160e902929b94338ada20cf46b265d595 ), MY_CXT_KEY that each module defined (and still defines to this day because of docs), was used as the unique per module element, instead of my_cxt_index. But we never told anything that MY_CXT_KEY has to be a valid C identifier without double quotes (it has double quotes already and you can't de-stringify), so MY_CXT_KEY can't be used as the following invalid proposal

---

# define START_MY_CXT int MY_CXT_KEY##_my_cxt_index = -1;

---

After reading the old MY_CXT implementation that used a HV with MY_CXT_KEY, I think your feature request was a "bug" that the new MY_CXT implementation fixed (maybe davem can chime in since he wrote it and I dont have time to search the ML archives), since on old MY_CXT, the non-threaded was static, the thread was global to module due to the HV and MY_CXT_KEY (assuming you declared MY_CXT_KEY in a compilation unit, that didn't have BOOT​:). In new MY_CXT both the threaded and non-threaded, are compilation unit static. I'm leaning towards this ticket being rejected, unless someone (not me) wants to write a new GMY_CXT api (G for global) that uses symbol name unique my_cxt_index declared as extern, this GMY_CXT will still use PL_my_cxt_list as the new MY_CXT does.

I was going to bring up another API design question but the theory is impossible due to a check of my_cxt_index is -1 in Perl_my_cxt_init() and my_cxt_index isn't assigned to if its != -1. Will GMY_CXT have protection against multiple runs of GMY_CXT_INIT or is that nearly impossible to create such a scenario? The only scenario I see is if someone does a "require" inside an ithread, then BOOT​: of the DLL will run again and set my_cxt_index to a new value.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @iabyn

On Sat, Jan 25, 2014 at 05​:22​:36PM -0800, bulk88 via RT wrote​:

(maybe davem can chime in since he wrote it and I dont have time to
search the ML archives)

I haven't followed the thread in depth, but my understanding is that the
original intent of the MY_CXT mechanism was to replace *static* data with
per-thread static data when perl started getting threded; i.e. code that
used to have

  static struct foo my_foo;

wasn't thread-safe, as each thread would need its own copy of my_foo.

AFAIKT, the current MY_CXT mechanism does this fine.

If people want to access that "static" data from other source files,
then surely a simple accessor function is sufficient?

I have no objection to someone making the API non-static if they can think
of a clean way of doing it without harming efficiency or backwards
compatibility, but personally I'm not interested in doing so.

--
This email is confidential, and now that you have read it you are legally
obliged to shoot yourself. Or shoot a lawyer, if you prefer. If you have
received this email in error, place it in its original wrapping and return
for a full refund. By opening this email, you accept that Elvis lives.