Re-implementation of the stat module in C

[pitrou]

BPO	11016
Nosy	@loewis, @jcea, @pitrou, @vstinner, @giampaolo, @tiran, @bitdancer, @sdrees
Files	statmodule.patch statmodule2.patch statmodule3.patch statmodule4.patch stat_mode_overflow.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jcea'
closed_at = <Date 2013-07-02.00:26:09.592>
created_at = <Date 2011-01-26.10:57:21.740>
labels = ['type-feature', 'library']
title = 'Re-implementation of the stat module in C'
updated_at = <Date 2013-07-02.00:26:09.591>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2013-07-02.00:26:09.591>
actor = 'christian.heimes'
assignee = 'jcea'
closed = True
closed_date = <Date 2013-07-02.00:26:09.592>
closer = 'christian.heimes'
components = ['Library (Lib)']
creation = <Date 2011-01-26.10:57:21.740>
creator = 'pitrou'
dependencies = []
files = ['30153', '30648', '30650', '30666', '30675']
hgrepos = []
issue_num = 11016
keywords = ['patch']
message_count = 49.0
messages = ['127100', '127103', '127105', '127107', '127108', '127110', '127111', '127114', '127117', '127118', '127137', '127138', '127567', '127579', '127592', '127671', '127672', '127673', '174739', '188300', '188321', '188519', '188526', '188547', '188550', '188551', '188561', '188610', '188611', '188832', '191486', '191492', '191503', '191596', '191640', '191659', '191663', '191665', '191681', '191682', '191683', '191692', '191698', '191699', '191704', '191730', '191736', '192166', '192168']
nosy_count = 12.0
nosy_names = ['loewis', 'jcea', 'pitrou', 'vstinner', 'movement', 'giampaolo.rodola', 'christian.heimes', 'Arfrever', 'r.david.murray', 'neologix', 'python-dev', 'dilettant']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11016'
versions = ['Python 3.4']

[pitrou]

Solaris has an additional kind of special files named doors. The standard headers define the following macro:

#define      S_ISDOOR(mode)  (((mode)&0xF000) == 0xd000)

Perhaps it would be nice to include the equivalent in the stat module.

(although funnily even the "stat" command doesn't recognize them and displayed "weird file" instead:

$ stat /var/run/syslog_door 
  File: `/var/run/syslog_door'
  Size: 0               Blocks: 0          IO Block: 0      weird file
Device: 8bc0000h/146538496d     Inode: 44          Links: 1
Access: (0644/Drw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-01-26 11:52:34.332492612 +0100
Modify: 2011-01-26 11:52:34.332492612 +0100
Change: 2011-01-26 11:52:34.332499428 +0100
)

[jcea]

Thanks for the feature request. It seems trivial to implement, but not trivial to test :).

I assign this to myself. I will work on it after Mercurial migration is finished. Getting impatient :).

[pitrou]

Thanks for the feature request. It seems trivial to implement, but not trivial to test :).

You can test against an existing file, such as "/var/run/syslog_door"
(given its name, it is unlikely to be anything other than a door)

[jcea]

I do know, but when you are working inside a zone, I am not sure you can count of that file being always present. Syslog even could be disabled or not available inside a zone.

I am thinking about how to manage OS's with no support for doors. Instead of conditionally compile the new STAT, I would rather have the function always available, but returning FALSE when the OS doesn't support doors. But what happen if you mount a Solaris filesystem in a nonsolaris machine?. Let say, a Solaris ZFS filesystem with doors, under linux/*bsd + ZFS?.

So maybe would be better to not compile that function in those OS's, instead of "lying" in corner cases.

Not decided yet. I accept ideas.

[pitrou]

I do know, but when you are working inside a zone, I am not sure you
can count of that file being always present. Syslog even could be
disabled or not available inside a zone.

Then you can just skip the test.

I am thinking about how to manage OS's with no support for doors.
Instead of conditionally compile the new STAT,

Lib/stat.py is a pure Python module. There is no compilation to do.
You just have to add the pure Python version of the C macro.
For the record, 0xd000 is (stat.S_IFSOCK + stat.S_IFIFO).

I would rather have the function always available, but returning FALSE
when the OS doesn't support doors.

Right.

But what happen if you mount a Solaris filesystem in a nonsolaris
machine?. Let say, a Solaris ZFS filesystem with doors, under
linux/*bsd + ZFS?.

I guess S_ISDOOR would return True.

[jcea]

Given the behaviour of Solaris DOORS, the flags selected are very cleverly chosen. Sensible engineering :).

But now I am wondering... Which organization defines flags like S_IFSOCK or S_IFIFO?. Posix members?. I am worried about flag collision between OSs.

For instance, if DOORS were a real flag defined in Solaris, how other OSs would know to not reuse that flag for some other purpose? :).

Your comment is very valuable and solve my doubts about it. Implementation is truly trivial and could be supported everywhere.

[pitrou]

But now I am wondering... Which organization defines flags like
S_IFSOCK or S_IFIFO?. Posix members?. I am worried about flag
collision between OSs.

They are defined unconditionally in Lib/stat.py.

For instance, if DOORS were a real flag defined in Solaris, how other
OSs would know to not reuse that flag for some other purpose? :).

Which flag? S_ISDOOR uses an (unlikely) combination of existing flags.

[jcea]

Antoine, I am not talking about python, I am talking about the UNIX standarization process. In particular, how a new flag (for filesystems) in an OS can be "skipped" and not being reused for some other purpuse in other OS.

Off topic for python, but curious to know anyway :).

[movement]

Jesus, yes, it's totally possible that POSIX might define:

(stat.S_IFSOCK + stat.S_IFIFO)

to mean something other than S_IDOOR. However, the POSIX committee
is always careful to respect existing usage. It's vanishingly unlikely
that any attempt to do this would actually happen, since it would
be obviously unimplementable on the most popular UNIX (namely Solaris).

[jcea]

I am looking at Linux :-).

Anyway the feedback has been very useful. I will implement this as soon as mercurial switch is done.

[loewis]

I am thinking about how to manage OS's with no support for doors.
Instead of conditionally compile the new STAT, I would rather have
the function always available, but returning FALSE when the OS
doesn't support doors. But what happen if you mount a Solaris
filesystem in a nonsolaris machine?. Let say, a Solaris ZFS
filesystem with doors, under linux/*bsd + ZFS?.

So maybe would be better to not compile that function in those OS's,
instead of "lying" in corner cases.

I'd rather see this exposed from the posix module than the stat module
(and see the stat module deprecated in the long term).
As much other stuff, it only be defined if the system actually has the
functionality. sys.path.isdoor could be defined on top of it, returning
False on systems that don't have doors in the first place.

For testing, if S_ISDOR is available but /var/run/name_service_door is
not, the test should get skipped.

[loewis]

Am 26.01.2011 14:14, schrieb Jesús Cea Avión:

Jesús Cea Avión <jcea@jcea.es> added the comment:

Antoine, I am not talking about python, I am talking about the UNIX
standarization process. In particular, how a new flag (for
filesystems) in an OS can be "skipped" and not being reused for some
other purpuse in other OS.

There is no standards body defining the S_IFMT numerical values.
POSIX defines the symbolic constants:

http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html

The Python stat module is flawed in assuming it knows the numeric values
(although by tradition, it is correct on all systems we care about).