Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-implementation of the stat module in C #55225

Closed
pitrou opened this issue Jan 26, 2011 · 50 comments
Closed

Re-implementation of the stat module in C #55225

pitrou opened this issue Jan 26, 2011 · 50 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@pitrou
Copy link
Member

pitrou commented Jan 26, 2011

BPO 11016
Nosy @loewis, @jcea, @pitrou, @vstinner, @giampaolo, @tiran, @bitdancer, @sdrees
Files
  • statmodule.patch
  • statmodule2.patch
  • statmodule3.patch
  • statmodule4.patch
  • stat_mode_overflow.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/jcea'
    closed_at = <Date 2013-07-02.00:26:09.592>
    created_at = <Date 2011-01-26.10:57:21.740>
    labels = ['type-feature', 'library']
    title = 'Re-implementation of the stat module in C'
    updated_at = <Date 2013-07-02.00:26:09.591>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2013-07-02.00:26:09.591>
    actor = 'christian.heimes'
    assignee = 'jcea'
    closed = True
    closed_date = <Date 2013-07-02.00:26:09.592>
    closer = 'christian.heimes'
    components = ['Library (Lib)']
    creation = <Date 2011-01-26.10:57:21.740>
    creator = 'pitrou'
    dependencies = []
    files = ['30153', '30648', '30650', '30666', '30675']
    hgrepos = []
    issue_num = 11016
    keywords = ['patch']
    message_count = 49.0
    messages = ['127100', '127103', '127105', '127107', '127108', '127110', '127111', '127114', '127117', '127118', '127137', '127138', '127567', '127579', '127592', '127671', '127672', '127673', '174739', '188300', '188321', '188519', '188526', '188547', '188550', '188551', '188561', '188610', '188611', '188832', '191486', '191492', '191503', '191596', '191640', '191659', '191663', '191665', '191681', '191682', '191683', '191692', '191698', '191699', '191704', '191730', '191736', '192166', '192168']
    nosy_count = 12.0
    nosy_names = ['loewis', 'jcea', 'pitrou', 'vstinner', 'movement', 'giampaolo.rodola', 'christian.heimes', 'Arfrever', 'r.david.murray', 'neologix', 'python-dev', 'dilettant']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue11016'
    versions = ['Python 3.4']

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 26, 2011

    Solaris has an additional kind of special files named doors. The standard headers define the following macro:

    #define      S_ISDOOR(mode)  (((mode)&0xF000) == 0xd000)

    Perhaps it would be nice to include the equivalent in the stat module.

    (although funnily even the "stat" command doesn't recognize them and displayed "weird file" instead:

    $ stat /var/run/syslog_door 
      File: `/var/run/syslog_door'
      Size: 0               Blocks: 0          IO Block: 0      weird file
    Device: 8bc0000h/146538496d     Inode: 44          Links: 1
    Access: (0644/Drw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2011-01-26 11:52:34.332492612 +0100
    Modify: 2011-01-26 11:52:34.332492612 +0100
    Change: 2011-01-26 11:52:34.332499428 +0100
    )

    @pitrou pitrou added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Jan 26, 2011
    @jcea
    Copy link
    Member

    jcea commented Jan 26, 2011

    Thanks for the feature request. It seems trivial to implement, but not trivial to test :).

    I assign this to myself. I will work on it after Mercurial migration is finished. Getting impatient :).

    @jcea jcea self-assigned this Jan 26, 2011
    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 26, 2011

    Thanks for the feature request. It seems trivial to implement, but not trivial to test :).

    You can test against an existing file, such as "/var/run/syslog_door"
    (given its name, it is unlikely to be anything other than a door)

    @jcea
    Copy link
    Member

    jcea commented Jan 26, 2011

    I do know, but when you are working inside a zone, I am not sure you can count of that file being always present. Syslog even could be disabled or not available inside a zone.

    I am thinking about how to manage OS's with no support for doors. Instead of conditionally compile the new STAT, I would rather have the function always available, but returning FALSE when the OS doesn't support doors. But what happen if you mount a Solaris filesystem in a nonsolaris machine?. Let say, a Solaris ZFS filesystem with doors, under linux/*bsd + ZFS?.

    So maybe would be better to not compile that function in those OS's, instead of "lying" in corner cases.

    Not decided yet. I accept ideas.

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 26, 2011

    I do know, but when you are working inside a zone, I am not sure you
    can count of that file being always present. Syslog even could be
    disabled or not available inside a zone.

    Then you can just skip the test.

    I am thinking about how to manage OS's with no support for doors.
    Instead of conditionally compile the new STAT,

    Lib/stat.py is a pure Python module. There is no compilation to do.
    You just have to add the pure Python version of the C macro.
    For the record, 0xd000 is (stat.S_IFSOCK + stat.S_IFIFO).

    I would rather have the function always available, but returning FALSE
    when the OS doesn't support doors.

    Right.

    But what happen if you mount a Solaris filesystem in a nonsolaris
    machine?. Let say, a Solaris ZFS filesystem with doors, under
    linux/*bsd + ZFS?.

    I guess S_ISDOOR would return True.

    @jcea
    Copy link
    Member

    jcea commented Jan 26, 2011

    Given the behaviour of Solaris DOORS, the flags selected are very cleverly chosen. Sensible engineering :).

    But now I am wondering... Which organization defines flags like S_IFSOCK or S_IFIFO?. Posix members?. I am worried about flag collision between OSs.

    For instance, if DOORS were a real flag defined in Solaris, how other OSs would know to not reuse that flag for some other purpose? :).

    Your comment is very valuable and solve my doubts about it. Implementation is truly trivial and could be supported everywhere.

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 26, 2011

    But now I am wondering... Which organization defines flags like
    S_IFSOCK or S_IFIFO?. Posix members?. I am worried about flag
    collision between OSs.

    They are defined unconditionally in Lib/stat.py.

    For instance, if DOORS were a real flag defined in Solaris, how other
    OSs would know to not reuse that flag for some other purpose? :).

    Which flag? S_ISDOOR uses an (unlikely) combination of existing flags.

    @jcea
    Copy link
    Member

    jcea commented Jan 26, 2011

    Antoine, I am not talking about python, I am talking about the UNIX standarization process. In particular, how a new flag (for filesystems) in an OS can be "skipped" and not being reused for some other purpuse in other OS.

    Off topic for python, but curious to know anyway :).

    @movement
    Copy link
    Mannequin

    movement mannequin commented Jan 26, 2011

    Jesus, yes, it's totally possible that POSIX might define:

    (stat.S_IFSOCK + stat.S_IFIFO)

    to mean something other than S_IDOOR. However, the POSIX committee
    is always careful to respect existing usage. It's vanishingly unlikely
    that any attempt to do this would actually happen, since it would
    be obviously unimplementable on the most popular UNIX (namely Solaris).

    @jcea
    Copy link
    Member

    jcea commented Jan 26, 2011

    I am looking at Linux :-).

    Anyway the feedback has been very useful. I will implement this as soon as mercurial switch is done.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jan 26, 2011

    I am thinking about how to manage OS's with no support for doors.
    Instead of conditionally compile the new STAT, I would rather have
    the function always available, but returning FALSE when the OS
    doesn't support doors. But what happen if you mount a Solaris
    filesystem in a nonsolaris machine?. Let say, a Solaris ZFS
    filesystem with doors, under linux/*bsd + ZFS?.

    So maybe would be better to not compile that function in those OS's,
    instead of "lying" in corner cases.

    I'd rather see this exposed from the posix module than the stat module
    (and see the stat module deprecated in the long term).
    As much other stuff, it only be defined if the system actually has the
    functionality. sys.path.isdoor could be defined on top of it, returning
    False on systems that don't have doors in the first place.

    For testing, if S_ISDOR is available but /var/run/name_service_door is
    not, the test should get skipped.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jan 26, 2011

    Am 26.01.2011 14:14, schrieb Jesús Cea Avión:

    Jesús Cea Avión <jcea@jcea.es> added the comment:

    Antoine, I am not talking about python, I am talking about the UNIX
    standarization process. In particular, how a new flag (for
    filesystems) in an OS can be "skipped" and not being reused for some
    other purpuse in other OS.

    There is no standards body defining the S_IFMT numerical values.
    POSIX defines the symbolic constants:

    http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html

    The Python stat module is flawed in assuming it knows the numeric values
    (although by tradition, it is correct on all systems we care about).

    @jcea
    Copy link
    Member

    jcea commented Jan 31, 2011

    Martin, what if C posix module (or whoever) would export the symbolic constants, and update "stat.py" to use those symbolic constants?.

    Do you think that would be an improvement?.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jan 31, 2011

    Martin, what if C posix module (or whoever) would export the symbolic
    constants, and update "stat.py" to use those symbolic constants?.

    Do you think that would be an improvement?.

    Improvement compared to what? The status quo? Certainly.
    Compared to a path where the stat module is deprecated?
    Certainly not.

    @jcea
    Copy link
    Member

    jcea commented Jan 31, 2011

    Martin, I guess "stat" deprecation could require a few years and it would be an extra incompatibility burden between 2.7 and 3.x.

    Beside the symbolic constants, why would you see "stat" deprecated?.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jan 31, 2011

    Am 31.01.2011 12:46, schrieb Jesús Cea Avión:

    Jesús Cea Avión <jcea@jcea.es> added the comment:

    Martin, I guess "stat" deprecation could require a few years and it
    would be an extra incompatibility burden between 2.7 and 3.x.

    Beside the symbolic constants, why would you see "stat" deprecated?.

    Because it doesn't provide any useful functionality, and confuses
    the users. The ST_* constants should have been deprecated long ago,
    since people really should use the st_ members of stat_result. In
    addition, stat_result should stop behaving like a tuple.

    The other constants and functions (S_*) are plainly non-portable.
    Apparently, contributor have the illusion that the values they put
    into the file are somehow standardized, when they are actually not.
    So having the module available makes people like you wonder how
    to best support it (see msg127107 and msg127110). Better remove
    it so that it doesn't confuse you anymore.

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 31, 2011

    Apparently, contributor have the illusion that the values they put
    into the file are somehow standardized, when they are actually not.

    Are there actual bugs with that or is it merely a theoretical concern?

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jan 31, 2011

    > Apparently, contributor have the illusion that the values they put
    > into the file are somehow standardized, when they are actually not.

    Are there actual bugs with that or is it merely a theoretical concern?

    Neither, nor (i.e. it's in the middle). I'm not aware of an actual
    problem, but I very much doubt that SF_SNAPSHOT really has the same
    meaning on all systems (IOW, chances are high that some system has
    allocated the same bit to mean something different).

    Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
    0xe000 as S_IFWHT (whiteout), but Solaris defines it as
    S_IFPORT (event port).

    @jcea
    Copy link
    Member

    jcea commented Nov 4, 2012

    After reviewing the code, I agree with Martin v. Löwis that "stat.py" is a joke. Neither can we trust the numerical constants (each OS is free to choose a different meaning) neither the code in "filemode()" can cope with modes with more than a bit active (like DOORS).

    I think all this code should be moved to a C module, able to access the underlining real constant definitions.

    What do you think?.

    Time to deprecate this and move the functionality into posix module, as suggested by Martin?.

    @tiran
    Copy link
    Member

    tiran commented May 3, 2013

    The stat module (or whatever it is going to be in Python 3.4) is missing more checks than S_ISDOOR:

    # Solaris
    S_ISDOOR()
    S_ISPORT()
    
    # POSIX 1.b real-time extension
    S_ISMSG()
    S_ISSEM()
    S_ISSHM()

    # whiteout, translucent file systems
    S_ISWHT

    @tiran
    Copy link
    Member

    tiran commented May 3, 2013

    I have done some research. The POSIX 1.b extensions aren't used on any supported system (neither Linux nor BSD, Solaris or AIX).

    random832@fastmail.us has compiled a list of even more file types:

    Heirloom toolchest "ls" supports:
    http://heirloom.cvs.sourceforge.net/viewvc/heirloom/heirloom/ls/ls.c?revision=1.9&view=markup
    http://heirloom.cvs.sourceforge.net/viewvc/heirloom/heirloom/ls/ls.1?revision=1.5&view=markup
    S_IFNWK HP-UX network special file
    S_IFNAM XENIX special named file
    S_INSEM XENIX semaphore subtype of IFNAM (looked up from s->rdev)
    S_INSHD XENIX shared data subtype of IFNAM " " " "

    Of these, GNU coreutils ls only supports doors and whiteouts.

    Chasing after a random hunch (something about AIX), I found these:

    http://cd.textfiles.com/transameritech2/EXTRAS/JOVE-4.6/ASK.C
    S_ISHIDDEN Hidden Directory [aix]
    S_ISCDF Context Dependent Files [hpux]
    S_ISNWK Network Special [hpux]

    http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00084.html
    S_ISMPX AIX "MPX" file (multiplex device?)

    https://github.com/gagern/gnulib/blob/master/tests/test-sys_stat.c
    has a massive pile of macros with no comments
    S_ISCTG S_ISMPB S_ISMPX S_ISNAM S_ISNWK S_ISOFD S_ISOFL S_ISPORT

    http://lists.gnu.org/archive/html/bug-gnulib/2004-08/msg00017.html
    S_ISOFD Cray DMF (data migration facility): off line, with data
    S_ISOFL Cray DMF (data migration facility): off line, with no data
    S_ISCTG Contiguous
    (It's possible that these may not be file types)

    http://doiso.googlecode.com/svn/trunk/Source/mkisofs-1.12b5/include/statdefs.h
    S_ISMPC UNUSED multiplexed c
    S_ISNAM Named file (XENIX)
    S_ISMPB UNUSED multiplexed b
    S_ISCNT Contiguous file
    S_ISSHAD Solaris shadow inode

    http://www.opensource.apple.com/source/gnutar/gnutar-450/gnutar/lib/sys_stat_.h
    S_ISMPB /* V7 */
    S_ISPORT /* Solaris 10 and up */
    S_TYPEISSEM S_TYPEISSHM - macros to check the XENIX IFNAM types
    mentioned above
    S_TYPEISMQ S_TYPEISTMO

    @tiran
    Copy link
    Member

    tiran commented May 6, 2013

    I have created a C implementation of the stat module for Python 3.4. It implements all current features, handling for DOOR, PORT, WHT and a fix for bpo-17913.

    The first half of the file has lots of #ifndef checks. I'm not sure if they are really required. I guess they are required for the tarfile module on Windows.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented May 6, 2013

    Christian, could you post it as a mercurial diff for review?
    Also, it would maybe be better to rename this issue to "rewrite stat module in C".

    Shouldn't we also remove the Python version?

    @bitdancer
    Copy link
    Member

    bitdancer commented May 6, 2013

    We've been *adding* python implementations for other modules, so I don't see why we would remove this one.

    @pitrou
    Copy link
    Member Author

    pitrou commented May 6, 2013

    We've been *adding* python implementations for other modules, so I
    don't see why we would remove this one.

    Well, the one reason is that the C constants aren't accessible from
    Python code. Once the constants are there, the rest is so trivial
    that it doesn't really deserve a pure Python alternative, IMHO.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented May 6, 2013

    We've been *adding* python implementations for other modules, so I don't see why we would remove this one.

    Because it's buggy, and cannot be implemented correctly in python.

    @bitdancer
    Copy link
    Member

    bitdancer commented May 6, 2013

    "Cannot be implemented correctly in Python" is a darned good reason :)

    @vstinner
    Copy link
    Member

    vstinner commented May 6, 2013

    If we use the C language, I would prefer to only expose what does really exist, and not add "fake" functions like:

    +#ifndef S_ISDOOR
    +# define S_ISDOOR(mode) 0
    +#endif

    So in:

    +static PyMethodDef stat_methods[] = {
    + {"S_ISDIR", stat_S_ISDIR, METH_O, stat_S_ISDIR_doc},
    + {"S_ISCHR", stat_S_ISCHR, METH_O, stat_S_ISCHR_doc},
    + {"S_ISBLK", stat_S_ISBLK, METH_O, stat_S_ISBLK_doc},
    + {"S_ISREG", stat_S_ISREG, METH_O, stat_S_ISREG_doc},
    + {"S_ISFIFO", stat_S_ISFIFO, METH_O, stat_S_ISFIFO_doc},
    + {"S_ISLNK", stat_S_ISLNK, METH_O, stat_S_ISLNK_doc},
    + {"S_ISSOCK", stat_S_ISSOCK, METH_O, stat_S_ISSOCK_doc},
    + {"S_ISDOOR", stat_S_ISDOOR, METH_O, stat_S_ISDOOR_doc},
    + {"S_ISPORT", stat_S_ISPORT, METH_O, stat_S_ISPORT_doc},
    + {"S_ISWHT", stat_S_ISWHT, METH_O, stat_S_ISWHT_doc},
    + {"S_IMODE", stat_S_IMODE, METH_O, stat_S_IMODE_doc},
    + {"S_IFMT", stat_S_IFMT, METH_O, stat_S_IFMT_doc},
    + {"filemode", stat_filemode, METH_O, stat_filemode_doc},
    + {NULL, NULL} /* sentinel */
    +};

    I expect something similar to what is done in posixmodule.c:

    static PyMethodDef stat_methods[] = {
        {"S_ISDIR",         stat_S_ISDIR,  METH_O, stat_S_ISDIR_doc},
        ...
    #ifdef S_ISDOOR
        {"S_ISDOOR",        stat_S_ISDOOR, METH_O, stat_S_ISDOOR_doc},
    #endif
        {"filemode",        stat_filemode, METH_O, stat_filemode_doc},
        {NULL,              NULL}           /* sentinel */
    };

    @vstinner
    Copy link
    Member

    vstinner commented May 6, 2013

    I have created a C implementation of the stat module for Python 3.4.

    Oh, your C module is called "stat", as the "stat" module implemented in Python (Lib/stat.py). What is your plan? Replace completly the Python module with the C module? It may be nice to keep the Python module for compatibility with other Python implementations, and test both implementations.

    @tiran
    Copy link
    Member

    tiran commented May 10, 2013

    AP has started to review my patch. I'm not yet sure how we should handle constants and functions when the platform doesn't provide them. For example Windows doesn't have any S_IS*() function like macros and doesn't provide a S_IFBLK constant.

    We have three possibilities here:

    1. omit the constant / function (breaks b/w compatibility)
    2. add constant with a value of 0 / function that returns false (may break b/w compatibility)
    3. default to current value from Lib/stat.py

    I'm against 1) because it breaks backwards compatibility and makes the module harder to use. I like to follow 3) for all S_I*, SF_* and UF_* macros and functions that are currently provided by the stat module and 2) for new constants and functions such as S_ISDOOR().

    @tiran tiran changed the title Add S_ISDOOR to the stat module stat module in C May 10, 2013
    @tiran
    Copy link
    Member

    tiran commented Jun 19, 2013

    New patch with extensive tests for the stat module. The tests are testing both the new C module and the old stat.py module.

    @tiran
    Copy link
    Member

    tiran commented Jun 19, 2013

    The latest patch comes with more comments and documentation updates.

    The C implementation defines all existing constants to the hard coded values from stat.py if the platform doesn't provide them. The approach keeps maximum backward compatibility with the old stat module.

    I have also added the new constants and functions to the pure Python implementation for maximum compatibility with other Python implementation.

    @tiran tiran changed the title stat module in C Re-implementation of the stat module in C Jun 19, 2013
    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Jun 20, 2013

    I still fail to understand why you just don't ditch the Python implementation.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 21, 2013

    New changeset f8ff61f44aca by Christian Heimes in branch '3.3':
    Add tests for untested features of the 'stat' module (part of issue bpo-11016)
    http://hg.python.org/cpython/rev/f8ff61f44aca

    New changeset d15aee50e4a0 by Christian Heimes in branch 'default':
    Add tests for untested features of the 'stat' module (part of issue bpo-11016)
    http://hg.python.org/cpython/rev/d15aee50e4a0

    @tiran
    Copy link
    Member

    tiran commented Jun 22, 2013

    New patch

    People demand a _stat module in C and now they are getting a _stat module in C.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 22, 2013

    New changeset 420f70a22b9d by Christian Heimes in branch 'default':
    Issue bpo-11016: Add C implementation of the stat module as _stat
    http://hg.python.org/cpython/rev/420f70a22b9d

    @vstinner
    Copy link
    Member

    vstinner commented Jun 22, 2013

    buf[9] is not initialized in stat_filemode(). Use a shorter buffer (9
    bytes) or set it to NUL.
    Le 22 juin 2013 21:05, "Roundup Robot" <report@bugs.python.org> a écrit :

    Roundup Robot added the comment:

    New changeset 420f70a22b9d by Christian Heimes in branch 'default':
    Issue bpo-11016: Add C implementation of the stat module as _stat
    http://hg.python.org/cpython/rev/420f70a22b9d

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue11016\>


    @tiran
    Copy link
    Member

    tiran commented Jun 22, 2013

    All 10 chars are set:

    buf[0] = filetype(mode);
    fileperm(mode, &buf[1]);

    buf[0] is set by filetype().
    fileperm() sets 9 chars in buf[1] to buf[9].

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 22, 2013

    New changeset e5427b0b2bf7 by Victor Stinner in branch 'default':
    Issue bpo-11016: Try to fix compilaton of the new _stat.c module on Windows
    http://hg.python.org/cpython/rev/e5427b0b2bf7

    @vstinner
    Copy link
    Member

    vstinner commented Jun 22, 2013

    fileperm() sets 9 chars in buf[1] to buf[9].

    Ah ok, fine, I missed the "&buf[1]" hack.

    @vstinner
    Copy link
    Member

    vstinner commented Jun 22, 2013

    New changeset e5427b0b2bf7 by Victor Stinner in branch 'default':
    Issue bpo-11016: Try to fix compilaton of the new _stat.c module on Windows
    http://hg.python.org/cpython/rev/e5427b0b2bf7

    @christian: Can you please review this commit?

    By the way, mode_t is also defined in import.c:

    #ifdef MS_WINDOWS
    /* for stat.st_mode */
    typedef unsigned short mode_t;
    /* for _mkdir */
    #include <direct.h>
    #endif

    And stat_filemode() should detect integer overflow. mode_t is a 32-bit unsigned integer on Linux, and now a 16-bit integer on Windows, whereas stat_filemode() uses an unsigned long (which 32 bit on Windows, and 32 or 64 bits on Linux).

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2013

    And stat_filemode() should detect integer overflow.

    Attached stat_mode_overflow.patch should fix this issue.

    (I would also suggest to inline fileperm() into stat_filemode(), or pass buf instead of &buf[1]. But you may not agree, as you want :-))

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2013

    My changeset e5427b0b2bf7 is not enough: "import _stat" still fail on Windows. See for example:

    http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/2164/steps/test/logs/stdio

    ======================================================================
    ERROR: test_directory (test.test_stat.TestFilemodeCStat)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_stat.py", line 128, in test_directory
        st_mode, modestr = self.get_mode()
      File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_stat.py", line 67, in get_mode
        modestr = self.statmod.filemode(st_mode)
    AttributeError: 'NoneType' object has no attribute 'filemode'

    ======================================================================
    ERROR: test_mode (test.test_stat.TestFilemodeCStat)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_stat.py", line 119, in test_mode
        st_mode, modestr = self.get_mode()
      File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_stat.py", line 67, in get_mode
        modestr = self.statmod.filemode(st_mode)
    AttributeError: 'NoneType' object has no attribute 'filemode'

    ======================================================================
    ERROR: test_module_attributes (test.test_stat.TestFilemodeCStat)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_stat.py", line 169, in test_module_attributes
        modvalue = getattr(self.statmod, key)
    AttributeError: 'NoneType' object has no attribute 'ST_DEV'

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2013

    test_stat.test_devices() fail on Solaris: it looks like os.devnull is a symlink. You should probably use os.stat() instead of os.lstat() for this specific test.

    http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20%28cc%2C%2032b%29%20%5BSB%5D%203.x/builds/708/steps/test/logs/stdio

    ======================================================================
    FAIL: test_devices (test.test_stat.TestFilemodeCStat)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/cpython/buildslave/cc-32/3.x.snakebite-sol10-sparc-cc-32/build/Lib/test/test_stat.py", line 157, in test_devices
        self.assertEqual(modestr[0], 'c')
    AssertionError: 'l' != 'c'
    - l
    + c

    @tiran
    Copy link
    Member

    tiran commented Jun 23, 2013

    I have addressed the Windows build issue in http://hg.python.org/cpython/rev/838f04e5a690 and the failing test on Solaris in http://hg.python.org/cpython/rev/6c23ca1982b3 (also 2.7 and default).

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 23, 2013

    New changeset 44c8a9d80595 by Victor Stinner in branch 'default':
    Issue bpo-11016: Detect integer conversion on conversion from Python int to C mode_t
    http://hg.python.org/cpython/rev/44c8a9d80595

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jun 23, 2013

    New changeset 75bc0ae02bcd by Christian Heimes in branch 'default':
    Issue bpo-11016: Don't define macros and constants that are already set by pyport.h
    http://hg.python.org/cpython/rev/75bc0ae02bcd

    @vstinner
    Copy link
    Member

    vstinner commented Jul 2, 2013

    Can we re-close this issue? Or is there still something to do?

    @tiran
    Copy link
    Member

    tiran commented Jul 2, 2013

    Let's close it.

    @tiran tiran closed this as completed Jul 2, 2013
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @QGB
    Copy link

    QGB commented Jun 27, 2022

    [[S_ISDIR , S_ISCHR , S_ISBLK , S_ISREG , S_ISFIFO, S_ISLNK , S_ISSOCK, S_ISDOOR, S_ISPORT, S_ISWHT , S_IMODE , S_IFMT  , filemode],
     (1       , 0       , 15597   , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 493     , 16384   , drwxr-xr-x),
     (1       , 7       , 2       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 360     , 16384   , dr-xr-x---),
     (1       , 9       , 25      , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 1023    , 16384   , drwxrwxrwt),
     (1       , 11      , 60      , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 365     , 16384   , dr-xr-xr-x),
     (1       , 20      , 1062    , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 448     , 16384   , drwx------),
     (1       , 62      , 27      , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 488     , 16384   , drwxr-x---),
     (0       , 118     , 7587    , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 493     , 32768   , -rwxr-xr-x),
     (0       , 126     , 107495  , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 420     , 32768   , -rw-r--r--),
     (0       , 153     , 847     , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 292     , 32768   , -r--r--r--),
     (0       , 237     , 2671    , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 384     , 32768   , -rw-------),
     (0       , 263     , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 288     , 32768   , -r--r-----),
     (0       , 518     , 361     , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 416     , 32768   , -rw-r-----),
     (0       , 3954    , 9       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 448     , 32768   , -rwx------),
     (0       , 4064    , 383     , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 484     , 32768   , -rwxr--r--),
     (0       , 4138    , 4       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 32768   , ----------),
     (1       , 4221    , 2       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 480     , 16384   , drwxr-----),
     (1       , 4423    , 79      , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 509     , 16384   , drwxrwxr-x),
     (1       , 4473    , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 489     , 16384   , drwxr-x--x),
     (0       , 14378   , 24      , 0       , 0       , 0       , 1       , 0       , 0       , 0       , 438     , 49152   , srw-rw-rw-),
     (0       , 14746   , 1       , 0       , 0       , 0       , 1       , 0       , 0       , 0       , 511     , 49152   , srwxrwxrwx),
     (0       , 14758   , 4       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 256     , 32768   , -r--------),
     (0       , 14832   , 757     , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 436     , 32768   , -rw-rw-r--),
     (1       , 15107   , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 457     , 16384   , drwx--x--x),
     (0       , 15108   , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 432     , 32768   , -rw-rw----),
     (1       , 15142   , 4       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 504     , 16384   , drwxrwx---),
     (1       , 15246   , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 472     , 16384   , drwx-wx---),
     (1       , 15249   , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 0       , 456     , 16384   , drwx--x---),
     (0       , 15736   , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 1389    , 32768   , -r-xr-sr-x),
     (0       , 15930   , 6       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 2505    , 32768   , -rws--x--x),
     (0       , 15949   , 28      , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 2541    , 32768   , -rwsr-xr-x),
     (0       , 16125   , 10      , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 1517    , 32768   , -rwxr-sr-x),
     (0       , 16153   , 3       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 1097    , 32768   , ---x--s--x),
     (0       , 16169   , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 2121    , 32768   , ---s--x--x),
     (0       , 16320   , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 73      , 32768   , ---x--x--x),
     (0       , 16453   , 2       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 2120    , 32768   , ---s--x---),
     (0       , 16939   , 13      , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 365     , 32768   , -r-xr-xr-x),
     (0       , 16954   , 9       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 488     , 32768   , -rwxr-x---),
     (0       , 39106   , 39      , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 509     , 32768   , -rwxrwxr-x),
     (0       , 84315   , 1       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 1481    , 32768   , -rwx--s--x),
     (0       , 84334   , 1       , 1       , 0       , 0       , 0       , 0       , 0       , 0       , 2536    , 32768   , -rwsr-x---)]
    
    

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants