Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Rewrite docs for Sphinx and compile into a reference manual.

  • Loading branch information...
commit 0460c2601baaf8f2f2b15e1dbde77d2a232eed4c 1 parent a55712d
@ehuss ehuss authored
View
2  .gitignore
@@ -5,5 +5,7 @@ build/
coro/_coro.[ch]
coro/oserrors.[ch]
coro/clocks/tsc_time.c
+coro/dns/packet.c
+coro/event_queue.cpp
*.pyc
*.so
View
177 coro/__init__.py
@@ -20,113 +20,7 @@
# $Header: //prod/main/ap/shrapnel/coro/__init__.py#31 $
-"""Coroutine threading library.
-
-Introduction
-============
-Shrapnel is a cooperative threading library.
-
-Getting Started
-===============
-When your process starts up, you must spawn a thread to do some work, and then
-start the event loop. The event loop runs forever processing events until the
-process exits. An example::
-
- import coro
-
- def main():
- print 'Hello world!'
- # This will cause the process to exit.
- coro.set_exit(0)
-
- coro.spawn(main)
- coro.event_loop()
-
-Coroutines
-==========
-Every coroutine thread is created with either the `new` function (which does
-NOT automatically start the thread) or the `spawn` function (which DOES
-automatically start it).
-
-Every thread has a unique numeric ID. You may also set the name of the thread
-when you create it.
-
-Timeouts
-========
-The shrapnel timeout facility allows you to execute a function which will be
-interrupted if it does not finish within a specified period of time. The
-`coro.TimeoutError` exception will be raised if the timeout expires. See the
-`with_timeout` docstring for more detail.
-
-If the event loop is not running (such as in a non-coro process), a custom
-version of `with_timeout` is installed that will operate using SIGALRM so that
-you may use `with_timeout` in code that needs to run in non-coro processes
-(though this is not recommended and should be avoided if possible).
-
-Thread Local Storage
-====================
-There is a thread-local storage interface available for storing global data that
-is thread-specific. You instantiate a `ThreadLocal` instance and you can
-assign attributes to it that will be specific to that thread. See the
-`ThreadLocal` docs for more detail.
-
-Signal Handlers
-===============
-By default when you start the event loop, two signal handlers are installed
-(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
-You can change this behavior by setting `install_signal_handlers` to False
-before starting the event loop.
-
-See `coro.signal_handler` for more detail on setting coro signal handlers.
-
-Selfishness
-===========
-Certain socket operations are allowed to try to execute without blocking if
-they are able to (such as send/receiving data on a local socket or on a
-high-speed network). However, there is a limit to the number of times a thread
-is allowed to do this. The default is 4. The default may be changed
-(`set_selfishness`) and the value on a per-thread may be changed
-(`coro.coro.set_max_selfish_acts`).
-
-Time
-====
-Shrapnel uses the `tsc_time` module for handling time. It uses the TSC
-value for a stable and high-resolution unit of time. See that module's
-documentation for more detail.
-
-A thread is always created when you start the event loop that will
-resynchronize the TSC relationship to accomodate any clock drift (see
-`tick_updater` and `tsc_time.update_time_relation`).
-
-Exception Notifier
-==================
-When a thread exits due to an exception, by default a stack trace is printed to
-stderr. You may install your own callback to handle this situation. See the
-`set_exception_notifier` function for more detail.
-
-Debug Output
-============
-The shrapnel library provides a mechanism for printing debug information to
-stderr. The `print_stderr` function will print a string with a timestamp
-and the thread number. The `write_stderr` function writes the string verbatim.
-
-Shrapnel keeps a reference to the "real" stderr (in `saved_stderr`) and the
-`print_stderr` and `write_stderr` functions always use the real stderr value. A
-particular reason for doing this is the backdoor module replaces sys.stderr and
-sys.stdout, but we do not want debug output to go to the interactive session.
-
-Profiling
-=========
-Shrapnel has its own profiler that is coro-aware. See `coro.profiler` for
-details on how to run the profiler.
-
-:Variables:
- - `all_threads`: A dictionary of all live coroutine objects. The key is
- the coroutine ID, and the value is the coroutine object.
- - `saved_stderr`: The actual stderr object for the process. This normally
- should not be used. An example of why this exists is because the
- backdoor replaces sys.stderr while executing user code.
-"""
+"""Coroutine threading library."""
from coro._coro import *
from coro._coro import _yield
@@ -166,13 +60,12 @@ def default_exception_notifier():
class InParallelError (Exception):
- """An error occurred in the `in_parallel` function.
+ """An error occurred in the :func:`in_parallel` function.
- :IVariables:
- - `result_list`: A list of ``(status, result)`` tuples. ``status`` is
- either `SUCCESS` or `FAILURE`. For success, the result is the return
+ :ivar result_list: A list of ``(status, result)`` tuples. ``status`` is
+ either :data:`SUCCESS` or :data:`FAILURE`. For success, the result is the return
value of the function. For failure, it is the output from
- `sys.exc_info`.
+ ``sys.exc_info``.
"""
def __init__(self, result_list):
@@ -195,17 +88,14 @@ def in_parallel (fun_arg_list):
This will block until all functions have returned or raised an exception.
- If one or more functions raises an exception, then the `InParallelError`
+ If one or more functions raises an exception, then the :exc:`InParallelError`
exception will be raised.
- :Parameters:
- - `fun_arg_list`: A list of ``(fun, args)`` tuples.
+ :param fun_arg_list: A list of ``(fun, args)`` tuples.
- :Return:
- Returns a list of return values from the functions.
+ :returns: A list of return values from the functions.
- :Exceptions:
- - `InParallelError`: One or more of the functions raised an exception.
+ :raises InParallelError: One or more of the functions raised an exception.
"""
# InParallelError, [(SUCCESS, result0), (FAILURE, exc_info1), ...]
@@ -257,14 +147,11 @@ def tick_updater():
def waitpid (pid):
"""Wait for a process to exit.
- :Parameters:
- - `pid`: The process ID to wait for.
+ :param pid: The process ID to wait for.
- :Return:
- Returns a tuple ``(pid, status)`` of the process.
+ :returns: A tuple ``(pid, status)`` of the process.
- :Exceptions:
- - `SimultaneousError`: Something is already waiting for this process
+ :raises SimultaneousError: Something is already waiting for this process
ID.
"""
if UNAME == "Linux":
@@ -290,14 +177,11 @@ def waitpid (pid):
def get_thread_by_id (thread_id):
"""Get a coro thread by ID.
- :Parameters:
- - `thread_id`: The thread ID.
+ :param thread_id: The thread ID.
- :Return:
- Returns the coroutine object.
+ :returns: The coroutine object.
- :Exceptions:
- - `KeyError`: The coroutine does not exist.
+ :raises KeyError: The coroutine does not exist.
"""
return all_threads[thread_id]
@@ -305,11 +189,9 @@ def where (co):
"""Return a string indicating where the given coroutine thread is currently
running.
- :Parameters:
- - `co`: The coroutine object.
+ :param co: The coroutine object.
- :Return:
- Returns a string displaying where the coro thread is currently
+ :returns: A string displaying where the coro thread is currently
executing.
"""
f = co.get_frame()
@@ -318,8 +200,7 @@ def where (co):
def where_all():
"""Get a dictionary of where all coroutines are currently executing.
- :Return:
- Returns a dictionary mapping the coroutine ID to a tuple of ``(name,
+ :returns: A dictionary mapping the coroutine ID to a tuple of ``(name,
coro, where)`` where ``where`` is a string representing where the
coroutine is currently running.
"""
@@ -339,13 +220,11 @@ def spawn (fun, *args, **kwargs):
Additional arguments and keyword arguments will be passed to the given function.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
- - `thread_name`: The name of the thread. Defaults to the name of the
+ :param fun: The function to call when the coroutine starts.
+ :param thread_name: The name of the thread. Defaults to the name of the
function.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
@@ -364,13 +243,11 @@ def new (fun, *args, **kwargs):
This will not start the coroutine. Call the ``start`` method on the
coroutine to schedule it to run.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
- - `thread_name`: The name of the thread. Defaults to the name of the
+ :param fun: The function to call when the coroutine starts.
+ :param thread_name: The name of the thread. Defaults to the name of the
function.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
@@ -457,8 +334,7 @@ def install_thread_emulation():
def coro_is_running():
"""Determine if the coro event loop is running.
- :Return:
- Returns True if the event loop is running, otherwise False.
+ :returns: True if the event loop is running, otherwise False.
"""
return event_loop_is_running
@@ -468,8 +344,7 @@ def sigterm_handler (*_unused_args):
def event_loop (timeout=30):
"""Start the event loop.
- :Parameters:
- - `timeout`: The amount of time to wait for kevent to return
+ :param timeout: The amount of time to wait for kevent to return
events. You should probably *not* set this value.
"""
global event_loop_is_running, with_timeout, sleep_relative
View
196 coro/_coro.pyx
@@ -1,15 +1,15 @@
-# Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
-#
+ # Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
+#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
-#
+#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
-#
+#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
@@ -19,14 +19,13 @@
# SOFTWARE.
# -*- Mode: Pyrex -*-
+#cython: embedsignature=True
"""Pyrex module for coroutine implementation.
Module variables defined below are available only from Pyrex. Python-accessible
variables are documented in the top level of the coro package ``__init__.py``.
-:Variables:
- - `_ticks_per_sec`: Number of CPU ticks per second (uint64_t).
"""
__coro_version__ = "$Id: //prod/main/ap/shrapnel/coro/_coro.pyx#114 $"
@@ -82,6 +81,7 @@ cdef extern from "Python.h":
# ================================================================================
# global variables
# ================================================================================
+# Number of CPU ticks per second (uint64_t).
cdef uint64_t _ticks_per_sec
cdef object ticks_per_sec
_ticks_per_sec = tsc_time_module.ticks_per_sec
@@ -155,10 +155,6 @@ cdef enum:
import sys
-class Exit (Exception):
- "exception used to exit the event loop"
- pass
-
class ScheduleError (Exception):
"attempt to schedule an already-scheduled coroutine"
pass
@@ -206,6 +202,12 @@ cdef int default_selfishness
default_selfishness = 4
def set_selfishness(n):
+ """Set the global default selfishness limit.
+
+ This sets the default for every new coroutine.
+
+ :param n: The new limit.
+ """
global default_selfishness
default_selfishness = n
@@ -214,11 +216,10 @@ live_coros = 0
cdef public class coro [ object _coro_object, type _coro_type ]:
- """XXX
+ """The coroutine object.
- :IVariables:
- - `top`: A `call_stack` object used by the profiler. NULL if the
- profiler is not enabled or if this is the first call of the coroutine.
+ Do not create this object directly. Use either :func:`new` or
+ :func:`spawn` to create one.
"""
cdef machine_state state
@@ -232,7 +233,8 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef size_t stack_size
cdef PyFrameObject * frame
cdef void * saved_exception_data[6]
- # used only by the profiler
+ # used only by the profiler, a call_stack object. NULL if the profiler is
+ # not enabled or if this is the first call of the coroutine.
cdef call_stack * top
cdef int saved_recursion_depth
cdef int selfish_acts, max_selfish_acts
@@ -488,45 +490,39 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef _schedule (self, value):
"""Schedule this coroutine to run.
- :Parameters:
- - `value`: The value to resume the coroutine with. Note that
- "interrupting" the coroutine resumes it with a special
- `exception` value which is checked when this coro is resumed.
+ :param value: The value to resume the coroutine with. Note that
+ "interrupting" the coroutine resumes it with a special
+ ``exception`` value which is checked when this coro is resumed.
- :Exceptions:
- - `DeadCoroutine`: The coroutine is dead (it has already exited).
- - `ScheduleError`: The coroutine is already scheduled to run.
- - `ScheduleError`: Attempted to schedule the currently running coro.
+ :raises DeadCoroutine: The coroutine is dead (it has already exited).
+ :raises ScheduleError: The coroutine is already scheduled to run.
+ :raises ScheduleError: Attempted to schedule the currently running coro.
"""
the_scheduler._schedule (self, value)
cdef _unschedule (self):
"""Unschedule this coroutine.
- :Return:
- Returns True if it was successfully unscheduled, False if not.
+ :returns: True if it was successfully unscheduled, False if not.
"""
return the_scheduler._unschedule (self)
def schedule (self, value=None):
"""Schedule this coroutine to run.
- :Parameters:
- - `value`: The value to resume the coroutine with. Defaults to
- None.
+ :param value: The value to resume the coroutine with. Defaults to
+ None.
- :Exceptions:
- - `DeadCoroutine`: The coroutine is dead (it has already exited).
- - `ScheduleError`: The coroutine is already scheduled to run.
- - `ScheduleError`: Attempted to schedule the currently running coro.
+ :raises DeadCoroutine: The coroutine is dead (it has already exited).
+ :raises ScheduleError: The coroutine is already scheduled to run.
+ :raises ScheduleError: Attempted to schedule the currently running coro.
"""
return self._schedule (value)
def start (self):
"""Start the coroutine for the first time.
- :Exceptions:
- - `ScheduleError`: The coro is already started.
+ :raises ScheduleError: The coro is already started.
"""
if self.started:
raise ScheduleError(self)
@@ -549,20 +545,18 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef __interrupt (self, the_exception):
"""Schedule the coro to resume with an exception.
- :Parameters:
- - `the_exception`: The exception to raise (may be class or instance).
+ :param the_exception: The exception to raise (may be class or instance).
- :Exceptions:
- - `DeadCoroutine`: The coroutine is dead (it has already exited).
- - `ScheduleError`: The coroutine is already scheduled to run.
- - `ScheduleError`: Attempted to interrupt the currently running coro.
+ :raises DeadCoroutine: The coroutine is dead (it has already exited).
+ :raises ScheduleError: The coroutine is already scheduled to run.
+ :raises ScheduleError: Attempted to interrupt the currently running coro.
"""
self._schedule (exception (the_exception))
def shutdown (self):
"""Shut down this coroutine.
- This will raise the `Shutdown` exception on this thread.
+ This will raise the :exc:`Shutdown` exception on this thread.
This method will not fail. If the thread is already dead, then it is
ignored. If the thread hasn't started, then it is canceled.
@@ -573,22 +567,19 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
def raise_exception (self, the_exception, force=True, cancel_start=False):
"""Schedule this coroutine to resume with an exception.
- :Parameters:
- - `the_exception`: The exception to raise. May be an Exception
- class or instance.
- - `force`: If True, will force the exception to be raised, even if
+ :param the_exception: The exception to raise. May be an Exception class or instance.
+ :param force: If True, will force the exception to be raised, even if
the coroutine is already scheduled. Defaults to True.
- - `cancel_start`: If True, will cancel the coroutine if it has not
+ :param cancel_start: If True, will cancel the coroutine if it has not
started, yet. If False, and the couroutine has not started, then
- it will rise `NotStartedError`. Defaults to False.
+ it will rise :exc:`NotStartedError`. Defaults to False.
- :Exceptions:
- - `DeadCoroutine`: The coroutine is dead (it has already exited).
- - `ScheduleError`: The coroutine is already scheduled to run (and
- `force` was set to False).
- - `ScheduleError`: Attempted to raise an exception on the currently
+ :raises DeadCoroutine: The coroutine is dead (it has already exited).
+ :raises ScheduleError: The coroutine is already scheduled to run (and
+ ``force`` was set to False).
+ :raises ScheduleError: Attempted to raise an exception on the currently
running coro.
- - `NotStartedError`: The coroutine has not started, yet.
+ :raises NotStartedError: The coroutine has not started, yet.
"""
IF CORO_DEBUG:
# Minstack coro used to take an "exception value" as the second
@@ -645,8 +636,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
def set_name (self, name):
"""Set the name of this coroutine thread.
- :Parameters:
- - `name`: The name of the thread.
+ :param name: The name of the thread.
"""
self.name = name
return self
@@ -660,8 +650,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
If no name has been specified, then a name is generated.
- :Return:
- Returns the coroutine name.
+ :returns: The coroutine name.
"""
return self.name
@@ -679,8 +668,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
When a coroutine is created, it defaults to 4.
- :Parameters:
- - `maximum`: The maximum number of selfish acts.
+ :param maximum: The maximum number of selfish acts.
"""
if maximum > 32768:
raise ValueError('Value too large.')
@@ -719,8 +707,7 @@ def get_live_coros():
Note that this includes coroutines that have not started or have exited,
but not deallocated, yet.
- :Return:
- Returns the number of live coroutine objects.
+ :returns: The number of live coroutine objects.
"""
global live_coros
return live_coros
@@ -757,11 +744,9 @@ def set_exception_notifier (new_func):
due to an exception. The default exception notifier simply prints the name
of the coroutine and a traceback of where the exception was raised.
- :Parameters:
- - `new_func`: The exception notifier to call. It takes no arguments.
+ :param new_func: The exception notifier to call. It takes no arguments.
- :Return:
- Returns the old exception notifier.
+ :returns: The old exception notifier.
"""
global exception_notifier
old_func = exception_notifier
@@ -867,12 +852,11 @@ class SimultaneousError (Exception):
"""Two threads attempted a conflicting blocking operation (e.g., read() on
the same descriptor).
- :IVariables:
- - `co`: The coroutine that is trying to block on an event.
- - `other`: The coroutine or function that is already waiting on the
+ :ivar co: The coroutine that is trying to block on an event.
+ :ivar other: The coroutine or function that is already waiting on the
event.
- - `event`: The event that it is trying to block on. For kqueue, this
- is normally a `kevent_key` object.
+ :ivar event: The event that it is trying to block on. For kqueue, this
+ is normally a ``kevent_key`` object.
"""
def __init__(self, co, other, object event):
@@ -1000,11 +984,9 @@ cdef public class sched [ object sched_object, type sched_type ]:
cdef _unschedule (self, coro co):
"""Unschedule this coroutine.
- :Parameters:
- - `co`: The coroutine to unschedule.
+ :param co: The coroutine to unschedule.
- :Return:
- Returns True if it was successfully unscheduled, False if not.
+ :returns: True if it was successfully unscheduled, False if not.
"""
cdef int i
for i from 0 <= i < len(self.pending):
@@ -1038,15 +1020,12 @@ cdef public class sched [ object sched_object, type sched_type ]:
The default latency warning threshold is 0.2 seconds. This will allow
you to change the threshold by multiplying the 0.2 value.
- :Parameters:
- - `factor`: The latency threshold multiplier. May be a number from
+ :param factor: The latency threshold multiplier. May be a number from
0 to 300. A value of 0 disables latency warnings.
- :Return:
- Returns the old multipler factor.
+ :returns: The old multipler factor.
- :Exceptions:
- - `ValueError`: The factor is too small or too large.
+ :raises ValueError: The factor is too small or too large.
"""
if factor < 0 or factor > 300:
raise ValueError('Latency factor must be a number from 0 to 300.')
@@ -1075,25 +1054,22 @@ cdef public class sched [ object sched_object, type sched_type ]:
Nested timeouts will be handled correctly. If an outer timeout fires
first, then only the outer ``except TimeoutError`` exception handler
will catch it. An exception handlers on the inside will be skipped
- becaue the actual exception is the `Interrupted` exception until it
+ becaue the actual exception is the :exc:`Interrupted` exception until it
gets to the original ``with_timeout`` frame.
Nested timeouts that are set to fire at the exact same time are not
defined which one will fire first.
- Care must be taken to *never* catch the `Interrupted` exception within
+ Care must be taken to *never* catch the :exc:`Interrupted` exception within
code that is wrapped with a timeout.
- :Parameters:
- - `delta`: The number of seconds to wait before raising a timeout.
+ :param delta: The number of seconds to wait before raising a timeout.
Should be >= 0. Negative value will be treated as 0.
- - `function`: The function to call.
+ :param function: The function to call.
- :Return:
- Returns the return value of the function.
+ :returns: The return value of the function.
- :Exceptions:
- - `TimeoutError`: The function did not return within the specified
+ :raises TimeoutError: The function did not return within the specified
timeout.
"""
cdef timebomb tb
@@ -1127,8 +1103,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
cdef sleep (self, uint64_t when):
"""Sleep until a specific point in time.
- :Parameters:
- - `when`: The TSC value when you want the coroutine to wake up.
+ :param when: The TSC value when you want the coroutine to wake up.
"""
cdef event e
IF CORO_DEBUG:
@@ -1150,8 +1125,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
Your thread may continue running (with the interrupt rescheduled to try
again later), or it may be interrupted.
- :Parameters:
- - `delta`: The number of seconds to sleep.
+ :param delta: The number of seconds to sleep.
"""
cdef uint64_t when
# Two lines to avoid Pyrex Python conversion.
@@ -1160,7 +1134,10 @@ cdef public class sched [ object sched_object, type sched_type ]:
self.sleep (when)
def sleep_absolute (self, uint64_t when):
- """This is an alias for the `sleep` method."""
+ """Sleep until a specific point in time.
+
+ :param when: The TSC value when you want the coroutine to wake up.
+ """
self.sleep (when)
cdef schedule_ready_events (self, uint64_t now):
@@ -1231,8 +1208,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
def event_loop (self, timeout=30):
"""Start the event loop.
- :Parameters:
- - `timeout`: The amount of time to wait for kevent to return
+ :param timeout: The amount of time to wait for kevent to return
events. You should probably *not* set this value. Defaults to 30
seconds.
"""
@@ -1301,11 +1277,10 @@ def yield_slice():
def schedule (coro co, value=None):
"""Schedule a coroutine to run.
- See `coro.schedule` for more detail.
+ See :meth:`coro.schedule` for more detail.
- :Parameters:
- - `co`: The coroutine to schedule.
- - `value`: The value to resume the coroutine with. Defaults to None.
+ :param co: The coroutine to schedule.
+ :param value: The value to resume the coroutine with. Defaults to None.
"""
return co._schedule (value)
@@ -1355,8 +1330,7 @@ def print_stderr (s):
This will print the thread id, followed by a timestamp, followed by the
string. If the string does not end with a newline, one will be added.
- :Parameters:
- - `s`: A string to print.
+ :param s: A string to print.
"""
try:
current_thread = current()
@@ -1394,11 +1368,9 @@ def spawn (fun, *args, **kwargs):
Additional arguments and keyword arguments will be passed to the given function.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
+ :param fun: The function to call when the coroutine starts.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
return _spawn (fun, args, kwargs)
@@ -1410,11 +1382,9 @@ def new (fun, *args, **kwargs):
This will not start the coroutine. Call the ``start`` method on the
coroutine to schedule it to run.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
+ :param fun: The function to call when the coroutine starts.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
id = get_coro_id()
co = coro (fun, args, kwargs, id)
@@ -1427,8 +1397,7 @@ def set_exit(exit_code=0):
Note that if any other coroutines are scheduled to run, they will be given
a chance to run before exiting.
- :Parameters:
- - `exit_code`: The exit code of the process. Defaults to 0.
+ :param exit_code: The exit code of the process. Defaults to 0.
"""
global _exit
global _exit_code
@@ -1441,8 +1410,7 @@ def set_print_exit_string(val):
By default, the string will be printed.
- :Parameters:
- - `val`: Whether or not "Exiting" should be printed when the event loop
+ :param val: Whether or not "Exiting" should be printed when the event loop
exits.
"""
global _print_exit_string
View
38 coro/profiler.py
@@ -25,17 +25,18 @@
Introduction
============
This profiler is coro-aware. It produces output to a binary file on disk. You
-then use the `coro.print_profile` module to convert it to an HTML file.
+then use the :mod:`coro.print_profile` module to convert it to an HTML file.
Using The Profiler
==================
-There are two ways to run the profiler. One is to use the `go` function where
-you give it a python function to run. Profiling will start and call the
-function, and then the profiler will automatically stop when the function
-exits.
+There are two ways to run the profiler. One is to use the
+:func:`coro.profiler.go` function where you give it a python function to run.
+Profiling will start and call the function, and then the profiler will
+automatically stop when the function exits.
-The other method is to call `start` to start the profiler and `stop` when you
-want to stop profiling. This can be conveniently done from the backdoor.
+The other method is to call :func:`coro.profiler.start` to start the profiler
+and :func:`coro.profiler.stop` when you want to stop profiling. This can be
+conveniently done from the backdoor.
Rendering Output
================
@@ -49,12 +50,13 @@
Profiler Types
==============
The profiler supports different ways of gathering statistics. This is done by
-specifying the "bench" object to use (see `go` and `start`). They default to
-the "rusage" method of gathering statistics about every function call (see the
-getrusage man page for more detail). If you want a higher performance profile,
-you can use the `coro.bench` object instead which simply records TSC values for
-every function call. If you want to define your own method of gathering
-statistics, subclass `coro.bench` and implement your own techniques.
+specifying the "bench" object to use (see :func:`go` and :func:`start`). They
+default to the "rusage" method of gathering statistics about every function
+call (see the getrusage man page for more detail). If you want a higher
+performance profile, you can use the :class:`coro.bench` object instead which
+simply records TSC values for every function call. If you want to define your
+own method of gathering statistics, subclass :class:`coro.bench` and implement
+your own techniques.
"""
@@ -98,14 +100,12 @@ def go (fun, *args, **kwargs):
This will display the results to stdout after the function is finished.
- :Parameters:
- - `fun`: The function to call.
+ :param fun: The function to call.
- :Keywords:
- - `profile_filename`: The name of the file to save the profile data.
+ :keyword profile_filename: The name of the file to save the profile data.
Defaults to '/tmp/coro_profile.bin'.
- - `profile_bench`: The bench object type to use. Defaults to
- `coro.rusage_bench`.
+ :keyword profile_bench: The bench object type to use. Defaults to
+ :class:`coro.rusage_bench`.
"""
if kwargs.has_key('profile_filename'):
profile_filename = kwargs['profile_filename']
View
345 coro/socket.pyx
@@ -207,20 +207,19 @@ cdef _readv_compute(size_list, buffer_tuple, int n, int received, iovec * iov,
into consideration what has been received so far will create an iovec array
for the readv function.
- :Parameters:
- - `size_list`: A Python object that should be a sequence of integers
+ :param size_list: A Python object that should be a sequence of integers
that indicate which blocks are being requested.
- - `buffer_tuple`: A tuple of Python strings (should be already
+ :param buffer_tuple: A tuple of Python strings (should be already
allocated and should be the same length as size_list).
- - `n`: The length of size_list and buffer_tuple.
- - `received`: The number of bytes received so far.
- - `iov`: The ``iovec`` array. This should have `n` elements.
- - `left`: OUTPUT: The number of bytes left to read.
- - `iov_pos`: OUTPUT: The number of elements added to ``iov``.
- - `complete_index`: The index of the last element in the buffer tuple
+ :param n: The length of size_list and buffer_tuple.
+ :param received: The number of bytes received so far.
+ :param iov: The ``iovec`` array. This should have ``n`` elements.
+ :param left: OUTPUT: The number of bytes left to read.
+ :param iov_pos: OUTPUT: The number of elements added to ``iov``.
+ :param complete_index: The index of the last element in the buffer tuple
that has been *completely* received. -1 if nothing has been
completely received.
- - `partial_index`: The index of the element in the buffer tuple that
+ :param partial_index: The index of the element in the buffer tuple that
has partially received some data. -1 if none of the elements have
partial data.
"""
@@ -254,7 +253,8 @@ cdef _readv_compute(size_list, buffer_tuple, int n, int received, iovec * iov,
cdef public class sock [ object sock_object, type sock_type ]:
- """Coro socket object.
+ """
+ Coro socket object.
This is typically used for network sockets, but can also be used for
coro-safe IO on any file descriptor that supports kqueue non-blocking
@@ -262,20 +262,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
The constructor takes the following parameters:
- - `domain`: The socket domain family, defaults to AF_INET (see `AF`).
- - `stype`: The socket type, defaults to SOCK_STREAM (see `SOCK`).
- - `protocol`: The socket protocol (normally not used, defaults to 0).
- - `fd`: The file descriptor to use. Creates a new socket file
+ :param domain: The socket domain family, defaults to AF_INET (see :class:`AF`).
+ :param stype: The socket type, defaults to SOCK_STREAM (see :class:`SOCK`).
+ :param protocol: The socket protocol (normally not used, defaults to 0).
+ :param fd: The file descriptor to use. Creates a new socket file
descriptor if not specified.
- :IVariables:
- - `fd`: The file descriptor number. Set to -1 when the socket is
+ :ivar fd: The file descriptor number. Set to -1 when the socket is
closed.
- - `orig_fd`: The original file descriptor number. This is left for
+ :ivar orig_fd: The original file descriptor number. This is left for
debugging purposes to determine which file descriptor was in use
before the socket was closed.
- - `domain`: The socket domain (AF_INET, AF_UNIX, AF_UNSPEC).
- - `stype`: The socket type (SOCK_STREAM, SOCK_DGRAM)
+ :ivar domain: The socket domain (AF_INET, AF_UNIX, AF_UNSPEC).
+ :ivar stype: The socket type (SOCK_STREAM, SOCK_DGRAM)
"""
cdef public int fd, orig_fd, domain, stype
@@ -323,8 +322,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def get_fileno (self):
"""Get the current file descriptor.
- :Return:
- Returns the current file descriptor number. Returns -1 if the
+ :returns: The current file descriptor number. Returns -1 if the
socket is closed.
"""
warnings.warn('socket.get_fileno() is deprecated, use fileno() instead.', DeprecationWarning)
@@ -333,8 +331,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def fileno (self):
"""Get the current file descriptor.
- :Return:
- Returns the current file descriptor number. Returns -1 if the
+ :returns: The current file descriptor number. Returns -1 if the
socket is closed.
"""
return self.fd
@@ -362,19 +359,16 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getsockopt (self, int level, int optname, socklen_t buflen=0):
"""Get a socket option.
- :Parameters:
- - `level`: The socket level to get (see `SOL`).
- - `optname`: The socket option to get (see `SO`).
- - `buflen`: The size of the buffer needed to retrieve the value. If
+ :param level: The socket level to get (see :class:`SOL`).
+ :param optname: The socket option to get (see :class:`SO`).
+ :param buflen: The size of the buffer needed to retrieve the value. If
not specified, it assumes the result is an integer and will
return an integer. Otherwise, it will create a new string with
the result, and you may use the struct module to decode it.
- :Return:
- Returns an integer if `buflen` is zero, otherwise returns a string.
+ :returns: An integer if ``buflen`` is zero, otherwise returns a string.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int flag, r
cdef socklen_t flagsize
@@ -397,13 +391,11 @@ cdef public class sock [ object sock_object, type sock_type ]:
def setsockopt (self, int level, int optname, value):
"""Set a socket option.
- :Parameters:
- - `level`: The socket level to set (see `SOL`).
- - `optname`: The socket option to set (see `SO`).
- - `value`: The value to set. May be an integer, or a struct-packed string.
+ :param level: The socket level to set (see :class:`SOL`).
+ :param optname: The socket option to set (see :class:`SO`).
+ :param value: The value to set. May be an integer, or a struct-packed string.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int flag, r
cdef socklen_t optlen
@@ -421,8 +413,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
It is safe to call this if the socket is already closed.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int r
if self.fd != -1:
@@ -442,15 +433,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call write to ensure all data has been sent. This
will raise OSError if it is unable to send all data.
- :Parameters:
- - `data`: The data to send.
+ :param data: The data to send.
- :Return:
- Returns the number of bytes sent, which should always be the length
- of `data`.
+ :returns: The number of bytes sent, which should always be the length
+ of ``data``.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef char * buffer
cdef int r, left, sent
@@ -482,35 +470,32 @@ cdef public class sock [ object sock_object, type sock_type ]:
def sendall(self, data):
"""Send all data.
- This is an alias for the `send` method.
+ This is an alias for the :meth:`send` method.
"""
return self.send(data)
def write (self, data):
"""Write data.
- This is an alias for the `send` method.
+ This is an alias for the :meth:`send` method.
"""
return self.send(data)
def sendto (self, data, address, int flags=0):
"""Send data to a specific address.
- :Parameters:
- - `data`: The data to send.
- - `address`: The address to send to. For unix-domain sockets, this
+ :param data: The data to send.
+ :param address: The address to send to. For unix-domain sockets, this
is a string. For IP sockets, this is a tuple ``(IP, port)``
where IP is a string.
Port is always an integer.
- - `flags`: sendto flags to use (defaults to 0) (see sendto(2)
+ :param flags: sendto flags to use (defaults to 0) (see sendto(2)
manpage).
- :Return:
- Returns the number of bytes sent which may be less than the send
+ :returns: The number of bytes sent which may be less than the send
requested.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef char * buffer
cdef sockaddr_storage sa
@@ -536,23 +521,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
else:
return r
- # - `flags`: recv flags to use (defaults to 0) (see recv(2) manpage).
def recv (self, int buffer_size):
"""Receive data.
This may return less data than you request if the socket buffer is not
- large enough. Use `recv_exact` to ensure you receive exactly the
+ large enough. Use :meth:`recv_exact` to ensure you receive exactly the
amount requested.
- :Parameters:
- - `buffer_size`: The number of bytes to receive.
+ :param buffer_size: The number of bytes to receive.
- :Return:
- Returns a string of data. Returns the empty string when the end of
+ :returns: A string of data. Returns the empty string when the end of
the stream is reached.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef buffer
cdef int r, new_buffer_size
@@ -584,7 +565,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def read (self, buffer_size):
"""Read data.
- This is an alias for the `recv` method.
+ This is an alias for the :meth:`recv` method.
"""
return self.recv(buffer_size)
@@ -594,20 +575,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
This may return less data than you request if the socket buffer is not
large enough.
- :Parameters:
- - `buffer_size`: The number of bytes to receive.
- - `flags`: Socket flags to set (defaults to 0) (see recvfrom(2)
+ :param buffer_size: The number of bytes to receive.
+ :param flags: Socket flags to set (defaults to 0) (see recvfrom(2)
manpage).
- :Return:
- Returns a tuple ``(data, address)`` where data is a string and
+ :returns: A tuple ``(data, address)`` where data is a string and
address is the address of the remote side (string for unix-domain,
tuple of ``(IP, port)`` for IP where IP is a string and port is an
integer). Data is the empty string when the end of the stream is
reached.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef buffer
cdef sockaddr_storage sa
@@ -654,15 +632,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call read until all data is received.
- :Parameters:
- - `bytes`: The number of bytes to receive.
+ :param bytes: The number of bytes to receive.
- :Return:
- Returns the data as a string.
+ :returns: The data as a string.
- :Exceptions:
- - `OSError`: OS-level error.
- - `EOFError`: Not all data could be read. The first argument
+ :raises OSError: OS-level error.
+ :raises EOFError: Not all data could be read. The first argument
includes any partial data read as a string.
"""
cdef char * p, * p0
@@ -696,16 +671,13 @@ cdef public class sock [ object sock_object, type sock_type ]:
of the stream is reached before all data is received, then the result
tuple will only contain the elements competely or partially received.
- :Parameters:
- - `size_list`: A sequence of integers that indicates the buffer
+ :param size_list: A sequence of integers that indicates the buffer
sizes to read.
- :Return:
- Returns a tuple of strings corresponding to the sizes requested in
- `size_list`.
+ :returns: A tuple of strings corresponding to the sizes requested in
+ ``size_list``.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int n, i
cdef int iov_pos
@@ -791,15 +763,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call writev until all data is sent. If it is
unable to send all data, it will raise an OSError exception.
- :Parameters:
- - `data`: A sequence of strings to write.
+ :param data: A sequence of strings to write.
- :Return:
- Returns the number of bytes sent which should always be the sum of
+ :returns: The number of bytes sent which should always be the sum of
the lengths of all the strings in the data sequence.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef char * buffer
cdef int r, left, size, sent
@@ -867,20 +836,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
This is for the Python buffer interface. If you don't know what that
is, move along. This method is for Python socket compatibility.
- :Parameters:
- - `buffer`: A writeable Python buffer object. Must be a contiguous
+ :param buffer: A writeable Python buffer object. Must be a contiguous
segment.
- - `nbytes`: Number of bytes to read. Must be less than or equal to
+ :param nbytes: Number of bytes to read. Must be less than or equal to
the size of the buffer. Defaults to 0 which means the size of
- `buffer`.
- - `flags`: Flags for the recv system call (see recv(2) manpage).
+ ``buffer``.
+ :param flags: Flags for the recv system call (see recv(2) manpage).
Defaults to 0.
- :Return:
- Returns the number of bytes read.
+ :returns: The number of bytes read.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef void *cbuf
cdef Py_ssize_t cbuflen
@@ -917,22 +883,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
This is for the Python buffer interface. If you don't know what that
is, move along. This method is for Python socket compatibility.
- :Parameters:
- - `buffer`: A writeable Python buffer object. Must be a contiguous
+ :param buffer: A writeable Python buffer object. Must be a contiguous
segment.
- - `nbytes`: Number of bytes to read. Must be less than or equal to
+ :param nbytes: Number of bytes to read. Must be less than or equal to
the size of the buffer. Defaults to 0 which means the size of
- `buffer`.
- - `flags`: Flags for the recv system call (see recvfrom(2) manpage).
+ ``buffer``.
+ :param flags: Flags for the recv system call (see recvfrom(2) manpage).
Defaults to 0.
- :Return:
- Returns a tuple ``(nbytes, address)`` where ``bytes`` is the number
+ :returns: A tuple ``(nbytes, address)`` where ``bytes`` is the number
of bytes read and ``address`` then it is the address of the remote
side.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef void *cbuf
@@ -972,18 +935,16 @@ cdef public class sock [ object sock_object, type sock_type ]:
cdef parse_address (self, object address, sockaddr_storage * sa, socklen_t * addr_len, bint resolve=False):
"""Parse a Python socket address and set the C structure values.
- :Parameters:
- - `address`: The Python address to parse. For IP, it should be a
+ :param address: The Python address to parse. For IP, it should be a
``(IP, port)`` tuple where the IP is a string. Use the empty
string to indicate INADDR_ANY.
The port should always be a host-byte-order integer.
For Unix-domain sockets, the address should be a string.
- - `sa`: OUTPUT: The sockaddr_storage C-structure to store the
+ :param sa: OUTPUT: The sockaddr_storage C-structure to store the
result in.
- - `addr_len`: OUTPUT: The size of the structure placed into `sa`.
+ :param addr_len: OUTPUT: The size of the structure placed into ``sa``.
- :Exceptions:
- - `ValueError`: The value could not be parsed.
+ :raises ValueError: The value could not be parsed.
"""
cdef sockaddr_in * sin
cdef sockaddr_in6 *sin6
@@ -1053,12 +1014,10 @@ cdef public class sock [ object sock_object, type sock_type ]:
cdef object unparse_address (self, sockaddr_storage *sa, socklen_t addr_len):
"""Unpack a C-socket address structure and generate a Python address object.
- :Parameters:
- - `sa`: The sockaddr_storage structure to unpack.
- - `addr_len`: The length of the `sa` structure.
+ :param sa: The sockaddr_storage structure to unpack.
+ :param addr_len: The length of the ``sa`` structure.
- :Return:
- Returns a ``(IP, port)`` tuple for IP addresses where IP is a
+ :returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string in canonical format for the given address family . Returns a
string for UNIX-domain sockets. Returns None for unknown socket
domains.
@@ -1087,13 +1046,11 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will block until there is data available to be read.
- :Return:
- Returns the amount "readable". For different sockets, this may be
+ :returns: The amount "readable". For different sockets, this may be
different values, see the EVFILT_READ section of the kevent manpage
for details.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return self._wait_for_read()
@@ -1102,11 +1059,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will block until it is possible to write to the socket.
- :Return:
- Returns the number of bytes writeable on the socket.
+ :returns: The number of bytes writeable on the socket.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return self._wait_for_write()
@@ -1119,14 +1074,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
cpdef connect_addr (self, address, bint resolve=False):
"""Connect the socket.
- :Parameters:
- - `address`: The address to connect to. For IP, it should be a
+ :param address: The address to connect to. For IP, it should be a
``(IP, port)`` tuple where the IP is a string.
The port should always be a host-byte-order integer. For
Unix-domain sockets, the address should be a string.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
@@ -1152,15 +1105,13 @@ cdef public class sock [ object sock_object, type sock_type ]:
def bind (self, address):
"""Bind the socket.
- :Parameters:
- - `address`: The address to bind to. For IP, it should be a
+ :param address: The address to bind to. For IP, it should be a
``(IP, port)`` tuple where the IP is a string. Use the empty
string to indicate INADDR_ANY.
The port should always be a host-byte-order integer.
For Unix-domain sockets, the address should be a string.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
@@ -1176,11 +1127,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
def listen (self, backlog):
"""Set the socket to listen for connections.
- :Parameters:
- - `backlog`: The maximum size of the queue for pending connections.
+ :param backlog: The maximum size of the queue for pending connections.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int r
r = listen (self.fd, backlog)
@@ -1190,14 +1139,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
def accept (self):
"""Accept a connection.
- :Return:
- Returns a tuple ``(socket, address)`` where ``socket`` is a socket
+ :returns: A tuple ``(socket, address)`` where ``socket`` is a socket
object and ``address`` is an ``(IP, port)`` tuple for IP
addresses or a string for UNIX-domain sockets. IP addresses are
returned as strings.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
@@ -1227,20 +1174,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
def accept_many (self, int max=0):
"""Accept multiple connections.
- This will accept up to `max` connections for any connections available
+ This will accept up to ``max`` connections for any connections available
on the listen queue. This will block if there are no connections
waiting.
- :Parameters:
- - `max`: The maximum number of connections to accept. If not
+ :param max: The maximum number of connections to accept. If not
specified, defaults to infinity (accept all pending connections).
- :Return:
- Returns a list of ``(socket, address)`` tuples (see `accept` method
+ :returns: A list of ``(socket, address)`` tuples (see :meth:`accept` method
for information on return format).
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
@@ -1280,11 +1224,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
def shutdown (self, int how):
"""Shutdown the socket.
- :Parameters:
- - `how`: How to shut down the socket (see the shutdown(2) manpage).
+ :param how: How to shut down the socket (see the shutdown(2) manpage).
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int r
r = shutdown (self.fd, how)
@@ -1296,12 +1238,10 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getpeername (self):
"""Get the remote-side address.
- :Return:
- Returns a ``(IP, port)`` tuple for IP addresses where IP is a
+ :returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string. Returns a string for UNIX-domain sockets.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
@@ -1318,8 +1258,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getsockname (self):
"""Get the local address of the socket.
- :Return:
- Returns a ``(IP, port)`` tuple for IP addresses where IP is a
+ :returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string or an empty string for INADDR_ANY. Returns a
string for UNIX-domain sockets (empty string if not bound).
"""
@@ -1340,17 +1279,15 @@ cdef public class sock [ object sock_object, type sock_type ]:
The mode and bufsize arguments are as for the built-in open() function.
- The underlying socket is duplicated via `sock.dup` to emulate Python's
+ The underlying socket is duplicated via ``sock.dup`` to emulate Python's
reference counting behavior.
- :Parameters:
- - `mode`: The mode of the file, defaults to 'r'.
- - `bufsize`: The buffer size (0 is no buffering, 1 is line
+ :param mode: The mode of the file, defaults to 'r'.
+ :param bufsize: The buffer size (0 is no buffering, 1 is line
buffering, greater than 1 is the explicit buffer size).
Defaults to -1 (does not change the default buffering).
- :Return:
- Returns a file-like object that wraps the socket.
+ :returns: A file-like object that wraps the socket.
"""
# Probably unwise to access an underscore private value from the
# socket module, but it should work OK for the foreseeable future.
@@ -1366,8 +1303,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def dup(self):
"""Duplicate the socket object using the OS dup() call.
- :Return:
- Returns a new sock instance that holds the new file descriptor.
+ :returns: A new sock instance that holds the new file descriptor.
"""
cdef sock new_sock
cdef int new_fd
@@ -1410,8 +1346,7 @@ def get_live_sockets():
"""Get the number of live socket objects. This includes socket objects
that are closed.
- :Return:
- Returns the number of socket objects.
+ :returns: The number of socket objects.
"""
global live_sockets
return live_sockets
@@ -1476,7 +1411,7 @@ cdef class file_sock(sock):
The constructor takes one argument:
- - ``fileobj``: A Python-like file object. Currently only needs to
+ :param fileobj: A Python-like file object. Currently only needs to
implement the ``fileno`` method.
When the object is deallocated, the file descriptor is closed.
@@ -1499,7 +1434,7 @@ cdef class fd_sock(sock):
The constructor takes one argument:
- - ``fd``: A file descriptor.
+ :param fd: A file descriptor.
When the object is deallocated, the file descriptor is closed.
"""
@@ -1511,55 +1446,45 @@ cdef class fd_sock(sock):
def tcp_sock():
"""Create a streaming IPv4 socket.
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (AF_INET, SOCK_STREAM)
def udp_sock():
"""Create a datagram IPv4 socket.
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (AF_INET, SOCK_DGRAM)
def tcp6_sock():
"""Create a streaming IPv6 socket.
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (AF_INET6, SOCK_STREAM)
def udp6_sock():
"""Create a datagram IPv6 socket.
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (AF_INET6, SOCK_DGRAM)
def unix_sock():
"""Create a streaming unix-domain socket.
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (AF_UNIX, SOCK_STREAM)
@@ -1569,26 +1494,19 @@ def make_socket (int domain, int stype):
This is a backwards-compatibility wrapper around the sock object
constructor.
- :Parameters:
- - `domain`: The socket domain family (see `AF`).
- - `stype`: The socket type (see `SOCK`).
+ :param domain: The socket domain family (see :class:`AF`).
+ :param stype: The socket type (see :class:`SOCK`).
- :Return:
- Returns a socket object.
+ :returns: A socket object.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
return sock (domain, stype)
def has_ipv6():
"""Whether or not this system can create an IPv6 socket.
- :Return:
- Returns True if this system can create an IPv6 socket, False otherwise
-
- :Exceptions:
- - None
+ :returns: True if this system can create an IPv6 socket, False otherwise
"""
cdef int s
@@ -1603,16 +1521,13 @@ def has_ipv6():
def socketpair(int domain=AF_UNIX, int stype=SOCK_STREAM, int protocol=0):
"""Create an unnamed pair of connected sockets.
- :Parameters:
- - `domain`: The socket domain family (defaults to AF_UNIX).
- - `stype`: The socket type (defaults to SOCK_STREAM).
- - `protocol`: The socket protocol (normally not used, defaults to 0).
+ :param domain: The socket domain family (defaults to AF_UNIX).
+ :param stype: The socket type (defaults to SOCK_STREAM).
+ :param protocol: The socket protocol (normally not used, defaults to 0).
- :Return:
- Returns a tuple of 2 connected sockets.
+ :returns: A tuple of 2 connected sockets.
- :Exceptions:
- - `OSError`: OS-level error.
+ :raises OSError: OS-level error.
"""
cdef int sv[2]
cdef int rc
View
150 coro/sync.pyx
@@ -49,22 +49,19 @@ class LockError (Exception):
# Semaphore
# ===========================================================================
-"""A semaphore is a locking primitive that corresponds with a set of
-resources. A semphore is essentially a counter. Whenever a resource is
-aquired, the count is lowered. If the count goes below 0, then it blocks
-until it goes above zero. Once you are done with a resource, you raise
-the counter."""
-
cdef class semaphore:
- """Semaphore lock object.
+ """
+ A semaphore is a locking primitive that corresponds with a set of
+ resources. A semphore is essentially a counter. Whenever a resource is
+ aquired, the count is lowered. If the count goes below 0, then it blocks
+ until it goes above zero. Once you are done with a resource, you raise
+ the counter.
- The constructor takes one parameter, the value to start the semaphore with.
+ :param value: The value to start the semaphore with (an integer).
- :IVariables:
- - `avail`: The current value of the semaphore. Also available via
- __int__.
- - `_waiting`: A fifo of ``(value, co)`` tuples of coroutines waiting
+ :ivar avail: The current value of the semaphore. Also available via __int__.
+ :ivar _waiting: A fifo of ``(value, co)`` tuples of coroutines waiting
for the semaphore. ``value`` is the value being requested, and ``co``
is the coroutine object. (C only.)
"""
@@ -87,8 +84,7 @@ cdef class semaphore:
if the requested number of resource elements are not available (if the
value would go negative).
- :Parameters:
- - `value`: The number of resource elements.
+ :param value: The number of resource elements.
"""
cdef coro me
me = the_scheduler._current
@@ -116,8 +112,7 @@ cdef class semaphore:
def release(self, int value):
"""Release a number of resource elements.
- :Parameters:
- - `value`: The number of resource elements to release (add to the
+ :param value: The number of resource elements to release (add to the
sempahore).
"""
cdef coro ci
@@ -174,21 +169,18 @@ cdef class semaphore:
cdef class inverted_semaphore:
- """Inverted semaphore.
-
+ """
An inverted semaphore works very much like a regular semaphore, except
threads block _until_ the value reaches zero. For example, if you want a
thread to wait for 1 or more events to finish, you can have each event
raise the value (always nonblocking) and have your waiter thread call
block_till_zero.
- The constructor takes one optional parameter, the value to start the
- semaphore with. It defaults to 0.
+ :param value: The value to start the semaphore with. It defaults to 0.
- :IVariables:
- - `value`: The value of the inverted semaphore. Also available via
+ :ivar value: The value of the inverted semaphore. Also available via
__int__.
- - `_waiting`: A fifo of coroutine objects waiting for the semaphore to
+ :ivar _waiting: A fifo of coroutine objects waiting for the semaphore to
reach zero. (C only).
"""
@@ -207,8 +199,7 @@ cdef class inverted_semaphore:
This never blocks.
- :Parameters:
- - `value`: The number of resource elements to acquire (add to the
+ :param value: The number of resource elements to acquire (add to the
semaphore). Defaults to 1.
"""
self.value = self.value + value
@@ -218,8 +209,7 @@ cdef class inverted_semaphore:
This never blocks. This may wake up waiting threads.
- :Parameters:
- - `value`: The number of resource elements to release (subtract
+ :param value: The number of resource elements to release (subtract
from the semaphore). Defaults to 1.
"""
cdef coro co
@@ -258,16 +248,16 @@ cdef class inverted_semaphore:
cdef class mutex:
- """Mutual Exclusion lock object.
+ """
+ Mutual Exclusion lock object.
A single thread may acquire the mutex multiple times, but it must release
the lock an equal number of times.
- :IVariables:
- - `_locked`: Count of how many locks on the mutex are currently held.
- - `_owner`: The coroutine object that owns the lock (None if no owner).
+ :ivar _locked: Count of how many locks on the mutex are currently held.
+ :ivar _owner: The coroutine object that owns the lock (None if no owner).
(C only.)
- - `_waiting`: A fifo of coroutine objects waiting for the lock.
+ :ivar _waiting: A fifo of coroutine objects waiting for the lock.
"""
cdef public int _locked
@@ -290,8 +280,7 @@ cdef class mutex:
A coro thread may lock the mutex multiple times. It must call unlock
the same number of times to release it.
- :Return:
- Returns True if it blocked, False if the mutex was acquired
+ :returns: True if it blocked, False if the mutex was acquired
immediately.
"""
cdef coro me
@@ -317,8 +306,7 @@ cdef class mutex:
def trylock(self):
"""Try to lock the mutex.
- :Return:
- Returns True if it is already locked by another coroutine thread.
+ :returns: True if it is already locked by another coroutine thread.
Returns False if the lock was successfully acquired.
"""
cdef coro me
@@ -333,20 +321,17 @@ cdef class mutex:
def locked (self):
"""Determine if the mutex is currently locked.
- :Return:
- Returns True if the mutex is locked, otherwise False.
+ :returns: True if the mutex is locked, otherwise False.
"""
return (self._locked > 0)
def has_lock (self, thread=None):
"""Determine if a particular coroutine has the lock.
- :Parameters:
- - `thread`: The coroutine object to check if it owns the lock. If
+ :param thread: The coroutine object to check if it owns the lock. If
not specified, defaults to the current thread.
- :Return:
- Returns True if the specified thread has the lock, otherwise
+ :returns: True if the specified thread has the lock, otherwise
returns False.
"""
if thread is None:
@@ -358,8 +343,7 @@ cdef class mutex:
The thread unlocking must be the thread that initially locked it.
- :Return:
- Returns True if another thread was waiting for the lock, otherwise
+ :returns: True if another thread was waiting for the lock, otherwise
it returns False.
"""
cdef coro me, co
@@ -397,7 +381,8 @@ cdef class mutex:
cdef class rw_lock:
- """A many-reader single-writer lock.
+ """
+ A many-reader single-writer lock.
This lock allows multiple "readers" to own the lock simultaneously. A
"writer" can only acquire a lock if there are no other "readers" or
@@ -411,14 +396,13 @@ cdef class rw_lock:
way around (holding a read lock and trying to acquire a write lock will
cause a deadlock).
- :IVariables:
- - `_writer`: Count of the number of write locks. (C only.)
- - `_writer_id`: Thread ID of the current write lock owner (0 if there
+ :ivar _writer: Count of the number of write locks. (C only.)
+ :ivar _writer_id: Thread ID of the current write lock owner (0 if there
is no owner). (C only.)
- - `_reader`: Count of the number of read locks. (C only.)
- - `_waiting_writers`: A fifo of coroutine objects waiting for a write
+ :ivar _reader: Count of the number of read locks. (C only.)
+ :ivar _waiting_writers: A fifo of coroutine objects waiting for a write
lock. (C only.)
- - `_waiting_readers`: A fifo of coroutine objects waiting for a read
+ :ivar _waiting_readers: A fifo of coroutine objects waiting for a read
lock. (C only.)
"""
@@ -443,7 +427,7 @@ cdef class rw_lock:
thread.
A coro thread may acquire multiple read locks, but it must call
- `read_unlock` an equal number of times.
+ :meth:`read_unlock` an equal number of times.
"""
cdef coro me
me = the_scheduler._current
@@ -463,12 +447,11 @@ cdef class rw_lock:
def try_read_lock(self):
"""Attempt to acquire a read lock.
- This is the same as `read_lock` except it does not block if it cannot
+ This is the same as :meth:`read_lock` except it does not block if it cannot
acquire the lock.
- :Return:
- Returns True if it cannot acquire the lock.
- Returns False if it successfully acquired the lock.
+ :returns: True if it cannot acquire the lock.
+ False if it successfully acquired the lock.
"""
cdef coro me
me = the_scheduler._current
@@ -484,7 +467,7 @@ cdef class rw_lock:
This blocks if there are any other readers or writers holding the lock.
A coro thread may acquire multiple write locks, but it must call
- `write_unlock` an equal number of times.
+ :meth:`write_unlock` an equal number of times.
Attempting to acquire a read lock while holding a write lock will cause
a deadlock.
@@ -518,12 +501,11 @@ cdef class rw_lock:
def try_write_lock(self):
"""Attempt to acquire a write lock.
- This is the same as `write_lock` except it does not block if it cannot
+ This is the same as :meth:`write_lock` except it does not block if it cannot
acquire the lock.
- :Return:
- Returns True if it cannot acquire the lock.
- Returns False if it successfully acquired the lock.
+ :returns: True if it cannot acquire the lock.
+ False if it successfully acquired the lock.
"""
cdef coro me
me = the_scheduler._current
@@ -602,13 +584,11 @@ cdef class rw_lock:
cdef class condition_variable:
- """Condition variable.
-
+ """
This locking primitive provides a method to "trigger" an event for other
threads.
- :IVariables:
- - `_waiting`: A fifo of coroutine objects waiting for the lock. (C only.)
+ :ivar _waiting: A fifo of coroutine objects waiting for the lock. (C only.)
"""
cdef readonly _fifo _waiting
@@ -634,8 +614,7 @@ cdef class condition_variable:
def wait (self):
"""Wait for the condition variable to be triggered.
- :Return:
- Returns the arguments given to the wake call (defaults to the empty
+ :returns: The arguments given to the wake call (defaults to the empty
tuple).
"""
return self._wait()
@@ -663,20 +642,17 @@ cdef class condition_variable:
If there are no threads waiting, this does nothing.
- :Parameters:
- - `args`: The arguments to wake the thread with. Defaults to the
+ :param args: The arguments to wake the thread with. Defaults to the
empty tuple.
- :Return:
- Returns True if a thread was awoken, False if not.
+ :returns: True if a thread was awoken, False if not.
"""
return self._wake_one (args)
def wake_all (self, args=()):
"""Wake all waiting threads.
- :Parameters:
- - `args`: The arguments to wake the thread with. Defaults to the
+ :param args: The arguments to wake the thread with. Defaults to the
empty tuple.
"""
cdef coro co
@@ -690,13 +666,11 @@ cdef class condition_variable:
def wake_n (self, int count, args=()):
"""Wake a specific number of threads.
- :Parameters:
- - `count`: The number of threads to wake up.
- - `args`: The arguments to wake the thread with. Defaults to the
+ :param count: The number of threads to wake up.
+ :param args: The arguments to wake the thread with. Defaults to the
empty tuple.
- :Return:
- Returns the total number of threads actually awoken.
+ :returns: The total number of threads actually awoken.
"""
cdef coro co
cdef int total
@@ -715,8 +689,7 @@ cdef class condition_variable:
def raise_all (self, the_exception):
"""Raise an exception on all waiting threads.
- :Parameters:
- - `the_exception`: The exception to raise on all waiting threads.
+ :param the_exception: The exception to raise on all waiting threads.
"""
cdef coro co
while self._waiting.size:
@@ -732,13 +705,13 @@ cdef class condition_variable:
cdef class fifo:
- """First-in First-Out container.
+ """
+ First-in First-Out container.
This uses a linked list.
- :IVariables:
- - `fifo`: The fifo object. (C only.)
- - `cv`: A condition variable. (C only.)
+ :ivar fifo: The fifo object. (C only.)
+ :ivar cv: A condition variable. (C only.)
"""
cdef _fifo fifo
@@ -754,8 +727,7 @@ cdef class fifo:
def push (self, thing):
"""Push an object to the end of the FIFO.
- :Parameters:
- - `thing`: The thing to add to the FIFO.
+ :param thing: The thing to add to the FIFO.
"""
self.fifo._push (thing)
self.cv.wake_one()
@@ -765,8 +737,7 @@ cdef class fifo:
This blocks if the FIFO is empty.
- :Return:
- Returns the next object from the FIFO.
+ :returns: The next object from the FIFO.
"""
while self.fifo.size == 0:
self.cv._wait()
@@ -778,8 +749,7 @@ cdef class fifo:
This will block if the fifo is empty and wait until there is an element
to pop.
- :Return:
- Returns a list of objects. Returns an empty list if the FIFO is
+ :returns: A list of objects. Returns an empty list if the FIFO is
empty.
"""
cdef int i
View
1  docs/.gitignore
@@ -0,0 +1 @@
+_build/
View
6 docs/conf.py
@@ -91,12 +91,12 @@
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
-html_theme = 'sphinxdoc'
+html_theme = 'default'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
-#html_theme_options = {}
+html_theme_options = {'collapsiblesidebar': True}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
@@ -240,3 +240,5 @@
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
+
+autodoc_default_flags = ['members', 'show-inheritance']
View
8 docs/index.rst
@@ -8,11 +8,9 @@ Welcome to Shrapnel's documentation!
Contents:
-.. toctree::
- :maxdepth: 1
-
- tutorial
-
+* :doc:`Installation <installation>`
+* :doc:`Tutorial <tutorial>`
+* :doc:`Reference Manual <ref/index>`
Indices and tables
View
47 docs/installation.rst
@@ -0,0 +1,47 @@
+============
+Installation
+============
+
+Supported Platforms
+===================
+Shrapnel currently works on FreeBSD, Linux, and Mac OS X with x86 32- or 64-bit platforms.
+It supports Python 2.7 (TODO: and 2.6?).
+
+Prerequisites
+=============
+pip
+---
+To make installation easy, you can use `pip <http://www.pip-installer.org/>`_.
+This is a tool which will fetch Python packages from `PyPi
+<http://pypi.python.org/>`_ and install them.
+
+Visit http://www.pip-installer.org/en/latest/installing.html for information
+on how to install pip if you don't already have it installed.
+
+Cython
+------
+You need version 0.12.1 or newer of `Cython <http://cython.org/>`_. If you
+already have Cython installed, you can check your current version by running
+``cython -V``.
+
+To install Cython, run:
+
+ pip install cython
+
+Distribute
+----------
+You need version 0.6.16 or newer of `distribute <http://pypi.python.org/pypi/distribute>`_.
+Distribute is a build and packaging tool for Python (a replacement for setuptools).
+
+To install distribute, run:
+
+ pip install distribute
+
+Shrapnel
+--------
+Finally, you can install Shrapnel, run:
+
+ pip install shrapnel
+
+Alternatively you can download it from https://github.com/ironport/shrapnel
+and do the usual ``python setup.py install`` procedure.
View
19 docs/ref/clocks.rst
@@ -0,0 +1,19 @@
+======
+Clocks
+======
+
+Shrapnel needs to keep track of time to manage scheduling of sleeps and
+timeouts. Because Shrapnel is intended to support thousands of coroutines,
+and each coroutine may be making many timeout calls per second, Shrapnel needs
+to use a timing facility that is relatively high performance. It also needs
+one that is monotonic, so it does not need to deal with system clock changes.
+
+The ``clocks`` subpackage is intended to provide a variety of different time
+facilities. Currently it only supports using the x86 TSC timer. This is a
+timer built in to the CPU, and thus is very fast.
+
+TSC Time
+========
+Support for TSC time is implemented in the ``coro.clocks.tsc_time`` module.
+
+.. automodule:: coro.clocks.tsc_time
View
107 docs/ref/coroutines.rst
@@ -0,0 +1,107 @@
+==========
+Coroutines
+==========
+
+The central concept of Shrapnel is the coroutine. You can think of a coroutine
+like it is a thread. When it runs out of work to do, it yields and allows other
+coroutines to run. Scheduling of coroutines is handled by the scheduler which
+runs an "event loop".
+
+Event Loop
+==========
+
+The event loop is a loop that runs forever until the program ends. Every
+Shrapnel program needs to start the event loop as one of the first things it
+does. A typical example would be::
+
+ import coro
+
+ def main():
+ print 'Hello world!'
+ # This will cause the process to exit.
+ coro.set_exit(0)
+
+ if __name__ == '__main__':
+ coro.spawn(main)
+ coro.event_loop()
+
+Coroutines
+==========
+
+Every coroutine thread is created with either the :func:`new` function (which
+does NOT automatically start the thread) or the :func:`spawn` function (which
+DOES automatically start it).
+
+Every thread has a unique numeric ID. You may also set the name of the thread
+when you create it.
+
+.. autoclass:: coro.coro
+
+Timeouts
+========
+The shrapnel timeout facility allows you to execute a function which will be
+interrupted if it does not finish within a specified period of time. The
+:class:`TimeoutError` exception will be raised if the timeout expires. See the
+:func:`with_timeout` docstring for more detail.
+
+If the event loop is not running (such as in a non-coro process), a custom
+version of `with_timeout` is installed that will operate using SIGALRM so that
+you may use `with_timeout` in code that needs to run in non-coro processes
+(though this is not recommended and should be avoided if possible).
+
+.. autofunction:: coro.with_timeout
+
+Parallel Execution
+==================
+XXX
+
+.. autofunction:: coro.in_parallel
+.. autoexception:: coro.InParallelError
+
+Thread Local Storage
+====================
+There is a thread-local storage interface available for storing global data that
+is thread-specific. You instantiate a :class:`ThreadLocal` instance and you can
+assign attributes to it that will be specific to that thread. From a design
+perspective, it is generally discouraged to use thread-local storage. But
+nonetheless, it can be useful at times.
+
+.. autoclass:: coro.ThreadLocal
+
+Functions
+=========
+The coro module defines the following functions:
+
+.. autofunction:: coro.get_thread_by_id
+.. autofunction:: coro.coro_is_running
+.. autofunction:: coro.event_loop
+.. autofunction:: coro.new
+.. autofunction:: coro.spawn