epoll and kqueue wrappers for the select module

[tiran]

BPO	1657
Nosy	@gvanrossum, @akuchling, @terryjreedy, @gpshead, @giampaolo, @tiran, @tpn, @intgr
Files	test_kqueue.diff trunk_select_epoll_kqueue9.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2010-07-10.23:43:25.091>
created_at = <Date 2007-12-19.06:57:08.176>
labels = ['type-feature', 'docs']
title = 'epoll and kqueue wrappers for the select module'
updated_at = <Date 2010-07-10.23:43:25.087>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2010-07-10.23:43:25.087>
actor = 'terry.reedy'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2010-07-10.23:43:25.091>
closer = 'terry.reedy'
components = ['Documentation']
creation = <Date 2007-12-19.06:57:08.176>
creator = 'christian.heimes'
dependencies = []
files = ['9020', '9535']
hgrepos = []
issue_num = 1657
keywords = []
message_count = 38.0
messages = ['58800', '58801', '58804', '58813', '58814', '58867', '58878', '58914', '58920', '58929', '58938', '58941', '58944', '58947', '58961', '58962', '58976', '58982', '58994', '59486', '59487', '60176', '60222', '60266', '62899', '63128', '63129', '64137', '64158', '64207', '64209', '64287', '64292', '64294', '64299', '89588', '99763', '109945']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'akuchling', 'terry.reedy', 'gregory.p.smith', 'jimjjewett', 'therve', 'giampaolo.rodola', 'christian.heimes', 'trent', 'intgr', 'Erik Gorset']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1657'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[tiran]

UPDATE:

• Better API with register(), unregister() and modify() instead of control()
• Some documentation

[therve]

Cool, thanks for working on that. Just for the record, I don't really
understand the workflow: why closing the other ticket as duplicate and
not post the patch on the old one? But whatever.

For this patch, I don't see the benefit of putting it in the select
module, instead of a separate module. Is there a specific reason?

Looking at the code, I don't have many remarks. pyepoll_new may leak if
epoll_create fails. I think that allowing threading around epoll_ctl is
useless, but I may be wrong.

Thanks again for working on it.

[tiran]

For this patch, I don't see the benefit of putting it in the select
module, instead of a separate module. Is there a specific reason?

There is at least one, but probably several other modules named "epoll"
or "_epoll" in the wild. These modules implement an epoll interface with
Pyrex, ctypes or C. All of them have a slightly different API. I don't
want to break 3rd party software.

The select module already contains an interface to select and poll. IMO
it's the best place.

Looking at the code, I don't have many remarks. pyepoll_new may leak
if epoll_create fails. I think that allowing threading around
epoll_ctl is useless, but I may be wrong.

Thanks, I've fixed the problem in pyepoll_new.

The Linux kernel protects every call to epoll_ctl with a mutex. It
*could* block and therefor I'm allowing threads around the epoll_ctl() call.

[tiran]

I mistakenly removed the wrong message. Here is the original msg:

The patch implements Linux's epoll interface
(http://linux.die.net/man/4/epoll). My patch doesn't introduce a new
module like http://bugs.python.org/issue1675118 and it wraps the epoll
control fd in an object like Twisted's _epoll.pyd interface.

My interface is almost identical to Twisted's API except for some names.
I named my control function "control" instead of "_control" and the
constants are all named "select.EPOLL_SPAM" instead of "SPAM".

Missing:
Documentation

[tiran]

Added license header to test_epoll.
Some C code cleanups

[tiran]

First draft of a kqueue wrapper loosely based on Twisted's _kqueue.c
from the kqreactor branch

[tiran]

UPDATE:

• misc cleanups
• added missing constants, see man kqueue(2)

[therve]

Some remarks:

• the name of the function used for PyArg_ParseTupleAndKeywords in
  register, modify, unregister is set to control instead of the good name.
• there is a leak in pyepoll_new if the parsing of arguments fails.
• the indentation is sometimes tabs, sometimes spaces. That should be
  good to unify this (to tabs I guess, since the select module used tabs
  before).
• it seems there is an unrelated change in sunau.py
• I don't think the stdlib unittest module has skip support. You have
  to find another way to skip the tests if the modules aren't present.

I've been able to port the epollreactor to your implementation and run
the whole twisted tests with it, so I don't think there are outstanding
problems. The code is fairly simple anyway.

That's it for epoll and general remarks. I'll look at kqueue asap. Thanks!

[tiran]

Some remarks:

• the name of the function used for PyArg_ParseTupleAndKeywords in
  register, modify, unregister is set to control instead of the good name.

Fixed

• there is a leak in pyepoll_new if the parsing of arguments fails.

Fixed

• the indentation is sometimes tabs, sometimes spaces. That should be
  good to unify this (to tabs I guess, since the select module used tabs
  before).

Fixed except for switch() and goto. I find the 4 space indention of the
case and the goto lables easier to read.

• it seems there is an unrelated change in sunau.py

Fixed

• I don't think the stdlib unittest module has skip support. You have
  to find another way to skip the tests if the modules aren't present.

Fixed ;)

I've been able to port the epollreactor to your implementation and run
the whole twisted tests with it, so I don't think there are outstanding
problems. The code is fairly simple anyway.

That's it for epoll and general remarks. I'll look at kqueue asap. Thanks!

Thansk for the code review and your remarks. I've fixed the problems
locally. I've also fixed a problem in unregister when fd is already
closed and I'm using PyFile_AsFileDescriptor(). It supports ints and
objects with a fileno() method.

I'm waiting for your test of kqueue before I upload the patch.

[tiran]

Here is the promised patch. I've added a small optimization. The objects
now keep a buffer to reduce the malloc overhead.

[tiran]

I've fixed a bug with the preallocated buffer and in the repr() of kevent.

[therve]

Here I go for kqueue:

• the docstring of test_kqueue.py is wrong
• the tests are a bit light. It would be good the have a test like
  test_control_and_wait in test_epoll.
• the kqueue_queue_control (and the pyepoll_poll) are now completely
  wrong! You should not limit to FD_SETSIZE, these 2 systems are there
  because they're able to handle for fds than that. Also, this buffer
  thing looks like a premature optimization. I'm unable to tell if it's
  correct or not.
• the NETDEV and related flags aren't defined under OS X 10.4. I guess
  there are flags for freebsd, but kqueue should build on OS X too.

I've been able to use this module for the twisted reactor, so the
functionality is OK. But thsi FD_SETSIZE limit is a huge problem.