Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically regenerate platform-specific modules #56828

Closed
Arfrever mannequin opened this issue Jul 23, 2011 · 27 comments
Closed

Automatically regenerate platform-specific modules #56828

Arfrever mannequin opened this issue Jul 23, 2011 · 27 comments

Comments

@Arfrever
Copy link
Mannequin

Arfrever mannequin commented Jul 23, 2011

BPO 12619
Nosy @malemburg, @loewis, @doko42, @amauryfa, @pitrou, @vstinner, @jwilk, @ned-deily, @djc, @ezio-melotti, @merwok, @bitdancer, @zware
Superseder
  • bpo-28027: Remove Lib/plat-/ files
  • Files
  • python-regenerate_platdir.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-09-09.16:41:42.138>
    created_at = <Date 2011-07-23.09:43:32.302>
    labels = []
    title = 'Automatically regenerate platform-specific modules'
    updated_at = <Date 2016-09-09.16:41:42.136>
    user = 'https://bugs.python.org/Arfrever'

    bugs.python.org fields:

    activity = <Date 2016-09-09.16:41:42.136>
    actor = 'zach.ware'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-09-09.16:41:42.138>
    closer = 'zach.ware'
    components = []
    creation = <Date 2011-07-23.09:43:32.302>
    creator = 'Arfrever'
    dependencies = []
    files = ['23423']
    hgrepos = []
    issue_num = 12619
    keywords = ['patch']
    message_count = 27.0
    messages = ['140952', '140957', '140958', '140962', '140969', '140984', '145653', '145659', '145667', '145759', '145762', '145948', '145953', '145956', '145958', '145961', '145970', '145971', '145977', '145981', '145987', '146021', '146027', '146360', '156157', '202627', '275335']
    nosy_count = 20.0
    nosy_names = ['lemburg', 'loewis', 'doko', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'jwilk', 'ned.deily', 'djc', 'ezio.melotti', 'eric.araujo', 'rpetrov', 'Arfrever', 'r.david.murray', 'Neurogeek', 'neologix', 'rosslagerwall', 'python-dev', 'Ramchandra Apte', 'zach.ware']
    pr_nums = []
    priority = 'normal'
    resolution = 'out of date'
    stage = 'resolved'
    status = 'closed'
    superseder = '28027'
    type = None
    url = 'https://bugs.python.org/issue12619'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Jul 23, 2011

    I suggest that platform-specific modules be automatically regenerated during installation. I'm attaching a patch.

    @merwok
    Copy link
    Member

    merwok commented Jul 23, 2011

    (Autotools/make newbie here) Why during install and not build?

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Jul 23, 2011

    I don't care when they will be regenerated.

    @doko42
    Copy link
    Member

    doko42 commented Jul 23, 2011

    is auto-generation wanted?
    are you sure that you can't end up with bad syntax?

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Jul 23, 2011

    py_compile.compile() can be used to verify syntax.

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Jul 23, 2011

    The new patch creates platform-specific modules at build time and verifies their syntax.

    @vstinner
    Copy link
    Member

    vstinner commented Oct 17, 2011

    is auto-generation wanted?

    For example, Lib/plat-linux/IN.py was regenerated at Nov 23 12:09:28 2002 (revision c2604d69aa5d) for the last time, and nobody complained. These modules are inconsistent. For example, the IN module contains constants of 64 bits platforms, whereas it can be used on 32 bits platforms.

    In my opinion, we should just drop these modules with h2py.py, I (re)opened a thread on python-dev.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 17, 2011

    -1 on auto-building. The header needed may not be available on the build platform, if it is not normally needed to build Python.

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Oct 17, 2011

    Regeneration of platform-specific modules fixes concerns about their outdateness and architecture differences (32-bit vs 64-bit, big endian vs little endian).

    Regeneration of given module could be performed only when corresponding header is available.

    Eventually regeneration of platform-specific modules could be controlled by an option of configure.

    @rpetrov
    Copy link
    Mannequin

    rpetrov mannequin commented Oct 17, 2011

    Related : bpo-1565071 and bpo-3990 . There is no reason to keep plat-xxx files if cannot be managed properly.

    @ned-deily
    Copy link
    Member

    ned-deily commented Oct 17, 2011

    What do you do for platforms like OS X where we support one set of binary files that contain multi-architecture C-files that can run as Intel-64, Intel-32 or PPC-32 on the same machine at user option at run time? For example, the Apple-suppled system Python on OS X 10.6 has all three and you can use arch to specify which to run (PPC is emulated on the Intel machines supported by 10.6):

    $ file /usr/bin/python2.6
    /usr/bin/python2.6: Mach-O universal binary with 3 architectures
    /usr/bin/python2.6 (for architecture x86_64):	Mach-O 64-bit executable x86_64
    /usr/bin/python2.6 (for architecture i386):	Mach-O executable i386
    /usr/bin/python2.6 (for architecture ppc7400):	Mach-O executable ppc

    The static IN.py currently shipped in plat-darwin is misleading at best. And you can't improve the situation by regenerating at installation time.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Oct 19, 2011

    Related : bpo-1565071 and bpo-3990 . There is no reason to keep plat-xxx files if cannot be managed properly.

    +1

    @malemburg
    Copy link
    Member

    malemburg commented Oct 19, 2011

    I don't see why these modules should be auto-generated. The constants
    in the modules hardly ever change and are also not affected by architecture
    differences (e.g. Mac OS X, Solaris, etc.) AFAICT.

    If you think they need to be auto-generated, you should make a case by
    example.

    Note that we cannot simply drop the modules. Some of the constants are
    needed for e.g. socket, ctypes or ldd programming.

    @vstinner
    Copy link
    Member

    vstinner commented Oct 19, 2011

    you should make a case by example

    Did you read comments of this issue and my email thread on python-dev? There are differents examples:

    • LONG_MAX is 9223372036854775807 even on 32 bits system
    • On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

    we cannot simply drop the modules. Some of the constants
    are needed for e.g. socket, ctypes or ldd programming.

    Ah? I removed all plat-* directories and ran the full test suite: no failure. The Python socket modules contain many constants (SOCK_, AF_, SO_*, ...):
    http://docs.python.org/library/socket.html#socket.AF_UNIX

    Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

    By "ldd", you mean "ld.so" (dlopen)? Yes, I agree that we need to expose dl constants. But the other constants are not used.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 19, 2011

    Victor, please accept that this entire infrastructure was originally added because there was a need for it, and the need did not go away. It's probably relevant only for a minority of applications, but you would make this minority's lifes much more complicated by removing the modules. If they don't hurt, why remove them?

    @pitrou
    Copy link
    Member

    pitrou commented Oct 19, 2011

    If they don't hurt, why remove them?

    Bogus constants don't hurt?

    @jwilk
    Copy link
    Mannequin

    jwilk mannequin commented Oct 19, 2011

    FYI, in Debian we have at least:
    one package using the CDROM module,
    3 packages using the IN module,
    14 packages using the DLFCN module.

    @malemburg
    Copy link
    Member

    malemburg commented Oct 19, 2011

    STINNER Victor wrote:

    STINNER Victor <victor.stinner@haypocalc.com> added the comment:

    > you should make a case by example

    Did you read comments of this issue and my email thread on python-dev?

    No.

    There are differents examples:

    • LONG_MAX is 9223372036854775807 even on 32 bits system
    • On Mac OS X, FAT programs contains 32 and 64 binaries, whereas constants are changed for 32 or 64 bits

    That's because the h2py.py script doesn't know anything about conditional
    definitions, e.g.

    PTRDIFF_MIN = (-9223372036854775807-1)
    PTRDIFF_MAX = (9223372036854775807)
    PTRDIFF_MIN = (-2147483647-1)
    PTRDIFF_MAX = (2147483647)

    Not all constants are really useful because of this, but some are, e.g.
    INT16_MAX, INT32_MAX, etc.

    > we cannot simply drop the modules. Some of the constants
    > are needed for e.g. socket, ctypes or ldd programming.

    Ah? I removed all plat-* directories and ran the full test suite: no failure.

    Right, the modules are not tested at all, AFAIK.

    The Python socket modules contain many constants (SOCK_, AF_, SO_*, ...):
    http://docs.python.org/library/socket.html#socket.AF_UNIX

    True, but you probably agree that it's easier to parse a header file
    into a Python module than to add each and every socket option on
    the planet to the C socket module, don't you ? :-)

    Which constants are used by the ctypes modules or can be used by modules using ctypes? Can you give examples? I listed usages of plat-* modules in the first message of my thread on python-dev.

    Not constants used by the ctypes, but constants which can be used
    with the ctypes module.

    By "ldd", you mean "ld.so" (dlopen)?

    Yes.

    Yes, I agree that we need to expose dl constants. But the other constants are not used.

    Not in the standard lib, that's true.

    Also note that the plat-* directories can contain platform specific code,
    e.g. the OS2 dirs have replacements for the pwd and grp modules.

    Finally, the list of standard files to include in those directories
    could be extended to cover more system level constants such as
    the ioctl or fcntl constants (not only the few defined in the C
    code, but all platform specific ones).

    @vstinner
    Copy link
    Member

    vstinner commented Oct 19, 2011

    I created the issue bpo-13226 to provide RTLD_* constants on all platforms (not only on Linux and sunos5).

    @vstinner
    Copy link
    Member

    vstinner commented Oct 19, 2011

    > There are differents examples:
    > - LONG_MAX is 9223372036854775807 even on 32 bits system
    > - On Mac OS X, FAT programs contains 32 and 64 binaries, whereas
    > constants are changed for 32 or 64 bits

    That's because the h2py.py script doesn't know anything about conditional
    definitions, e.g.

    PTRDIFF_MIN = (-9223372036854775807-1)
    PTRDIFF_MAX = (9223372036854775807)
    PTRDIFF_MIN = (-2147483647-1)
    PTRDIFF_MAX = (2147483647)

    Why may fix h2py.py, but regenerating plat-/.py files on build is not an
    option according to Martin (msg145659).

    And it doesn't solve the Mac OS X issue.

    Not all constants are really useful because of this, but some are, e.g.
    INT16_MAX, INT32_MAX, etc.

    INT16_MAX and INT32_MAX are not platform dependent. I'm not sure that Python
    should provide such constants.

    Right, the modules are not tested at all, AFAIK.

    One more reason to remove them. By the way, there are not documented.

    > The Python socket modules contain many constants (SOCK_, AF_, SO_*,
    > ...): http://docs.python.org/library/socket.html#socket.AF_UNIX

    True, but you probably agree that it's easier to parse a header file
    into a Python module than to add each and every socket option on
    the planet to the C socket module, don't you ? :-)

    It's not exactly the purpose of this issue. We are first trying to get correct
    constants.

    I don't think that it's really useful to expose all constants. I prefer to
    expose more constants on demand. When we add new constants, we try to use it,
    test it and document it (e.g. O_CLOEXEC).

    > Which constants are used by the ctypes modules or can be used by modules
    > using ctypes? Can you give examples? I listed usages of plat-* modules
    > in the first message of my thread on python-dev.

    Not constants used by the ctypes, but constants which can be used
    with the ctypes module.

    Sorry, I don't see which constants are useful with the ctypes module?

    > By "ldd", you mean "ld.so" (dlopen)?

    Yes.

    See my issue bpo-13226 for these constants.

    Also note that the plat-* directories can contain platform specific code,
    e.g. the OS2 dirs have replacements for the pwd and grp modules.

    (OS/2 has been deprecated in Python 3.3, see the PEP-11.)

    Except plat-os2emx, I don't see other platform specific code.

    @vstinner
    Copy link
    Member

    vstinner commented Oct 19, 2011

    FYI, in Debian we have at least:
    one package using the CDROM module,

    Is it cdsuite? (http://offog.org/code/cdsuite.html)

    3 packages using the IN module,

    I found policykit (it uses IN.INT_MAX). What are the 2 others? Which constants are used?

    14 packages using the DLFCN module.

    Oh, so many? Which projects? Which constants are used?

    @jwilk
    Copy link
    Mannequin

    jwilk mannequin commented Oct 20, 2011

    >FYI, in Debian we have at least:
    >one package using the CDROM module,
    Is it cdsuite? (http://offog.org/code/cdsuite.html)

    No, Freevo <http://freevo.sourceforge.net/\>.

    >3 packages using the IN module,
    I found policykit (it uses IN.INT_MAX).

    Not in Debian, apparently...

    What are the 2 others? Which constants are used?

    > 14 packages using the DLFCN module.

    Oh, so many? Which projects? Which constants are used?

    (That's only 12; we have two versions of Ecasound in the archive, and
    two binary packages with DLFCN-using code are produced from GDCM
    sources.)

    @vstinner
    Copy link
    Member

    vstinner commented Oct 20, 2011

    For Freevo: yes, it uses the CDROM module. But this module doesn't look to be perfect because Freevo has a fallback for FreeBSD and another for... Linux (because of missing CDROM.CDROM_DRIVE_STATUS?):
    -----------------------------
    try:
    from CDROM import *
    # test if CDROM_DRIVE_STATUS is there
    # (for some strange reason, this is missing sometimes)
    CDROM_DRIVE_STATUS
    except:
    if os.uname()[0] == 'FreeBSD':
    # FreeBSD ioctls - there is no CDROM.py...
    CDIOCEJECT = 0x20006318
    CDIOCCLOSE = 0x2000631c
    CDIOREADTOCENTRYS = 0xc0086305
    CD_LBA_FORMAT = 1
    CD_MSF_FORMAT = 2
    CDS_NO_DISC = 1
    CDS_DISC_OK = 4
    else:
    # see linux/cdrom.h and Documentation/ioctl/cdrom.txt
    CDROMEJECT = 0x5309
    CDROM_GET_CAPABILITY = 0x5331
    CDROMCLOSETRAY = 0x5319 # pendant of CDROMEJECT
    CDROM_SET_OPTIONS = 0x5320 # Set behavior options
    CDROM_CLEAR_OPTIONS = 0x5321 # Clear behavior options
    CDROM_SELECT_SPEED = 0x5322 # Set the CD-ROM speed
    CDROM_SELECT_DISC = 0x5323 # Select disc (for juke-boxes)
    CDROM_MEDIA_CHANGED = 0x5325 # Check is media changed
    CDROM_DRIVE_STATUS = 0x5326 # Get tray position, etc.
    CDROM_DISC_STATUS = 0x5327 # Get disc type, etc.
    CDROM_CHANGER_NSLOTS = 0x5328 # Get number of slots
    CDROM_LOCKDOOR = 0x5329 # lock or unlock door
    CDROM_DEBUG = 0x5330 # Turn debug messages on/off
    CDROM_GET_CAPABILITY = 0x5331 # get capabilities
    # CDROM_DRIVE_STATUS
    CDS_NO_INFO = 0
    CDS_NO_DISC = 1
    CDS_TRAY_OPEN = 2
    CDS_DRIVE_NOT_READY = 3
    CDS_DISC_OK = 4
    # capability flags
    CDC_CLOSE_TRAY = 0x1 # caddy systems _can't_ close
    CDC_OPEN_TRAY = 0x2 # but _can_ eject.
    CDC_LOCK = 0x4 # disable manual eject
    CDC_SELECT_SPEED = 0x8 # programmable speed
    CDC_SELECT_DISC = 0x10 # select disc from juke-box
    CDC_MO_DRIVE = 0x40000
    CDC_MRW = 0x80000
    CDC_MRW_W = 0x100000
    CDC_RAM = 0x200000
    # CDROM_DISC_STATUS
    CDS_AUDIO = 100
    CDS_DATA_1 = 101
    CDS_DATA_2 = 102
    CDS_XA_2_1 = 103
    CDS_XA_2_2 = 104
    CDS_MIXED = 105
    -----------------------------

    So Freevo does stil work if we remove the CDROM module. I still think that the Python standard library is not the right place for such constants. A third party module providing an API would be a better idea.

    socket.SO_PEERCRED has been added to Python 3.3 by c64216addd7f:

    "Add support for the send/recvmsg API to the socket module. Patch by David Watson and Heiko Wundram. (Closes bpo-6560)"

    socket.SO_PEERCRED is not listed in socket documentation.

    socket has no SO_BINDTODEVICE constant yet.

    RTLD_NOW, RTLD_GLOBAL and RTLD_LAZY will added to the posix module if my issue bpo-13226 is accepted.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 25, 2011

    New changeset 6159311f0f44 by Victor Stinner in branch 'default':
    Issue bpo-12619: Expose socket.SO_BINDTODEVICE constant
    http://hg.python.org/cpython/rev/6159311f0f44

    @RamchandraApte
    Copy link
    Mannequin

    RamchandraApte mannequin commented Mar 17, 2012

    +1 for this.

    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Nov 11, 2013

    See also bpo-19554.

    @zware
    Copy link
    Member

    zware commented Sep 9, 2016

    The platform-specific modules have been removed.

    @zware zware closed this as completed Sep 9, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    None yet
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants