Permalink
Browse files

Merge branch 'master' into zstack

Conflicts:
	coro/_coro.pyx
  • Loading branch information...
2 parents 7ba1e19 + 5187b9c commit e921e61e01463f74f8479ed8544ec0ab9457e9ca @samrushing samrushing committed May 2, 2012
Showing with 6,045 additions and 12,203 deletions.
  1. +2 −0 .gitignore
  2. +6 −0 README.rst
  3. +26 −151 coro/__init__.py
  4. +95 −127 coro/_coro.pyx
  5. +1 −0 coro/dns/__init__.py
  6. +516 −0 coro/dns/packet.pyx
  7. +78 −0 coro/dns/stub_resolver.py
  8. +102 −0 coro/dns/test/tpack.py
  9. +0 −183 coro/event_queue.cc
  10. +0 −46 coro/event_queue.h
  11. +77 −83 coro/event_queue.pyx
  12. +8 −0 coro/http/__init__.py
  13. +150 −0 coro/http/client.py
  14. +79 −0 coro/http/demo/grid.py
  15. +45 −0 coro/http/demo/session.py
  16. +27 −0 coro/http/demo/tclient.py
  17. +20 −0 coro/http/demo/tlslite_server.py
  18. +195 −0 coro/http/handlers.py
  19. 0 {coroutine/httpd → coro/http}/http_date.py
  20. +155 −0 coro/http/protocol.py
  21. +549 −0 coro/http/server.py
  22. +64 −0 coro/http/session_handler.py
  23. +85 −251 coro/linux_poller.pyx
  24. +19 −19 coro/profiler.py
  25. +116 −0 coro/read_stream.py
  26. +180 −225 coro/socket.pyx
  27. +60 −90 coro/sync.pyx
  28. +3 −0 coro/zstack_lz4.pyx
  29. +0 −955 coroutine/httpd/coro_httpd.py
  30. +0 −113 coroutine/httpd/mime_type_table.py
  31. +0 −9,859 coroutine/httpd/rfc2616.txt
  32. +0 −82 coroutine/httpd/session_handler.py
  33. +494 −0 distribute_setup.py
  34. +1 −0 docs/.gitignore
  35. +153 −0 docs/Makefile
  36. +244 −0 docs/conf.py
  37. +10 −2 docs/coro.rst
  38. +22 −0 docs/index.rst
  39. +47 −0 docs/installation.rst
  40. +19 −0 docs/ref/clocks.rst
  41. +107 −0 docs/ref/coroutines.rst
  42. +105 −0 docs/ref/debugging.rst
  43. +8 −0 docs/ref/dns.rst
  44. +12 −0 docs/ref/emulation.rst
  45. +22 −0 docs/ref/index.rst
  46. +31 −0 docs/ref/oserrors.rst
  47. +5 −0 docs/ref/profiling.rst
  48. +30 −0 docs/ref/selfishness.rst
  49. +17 −0 docs/ref/signals.rst
  50. +44 −0 docs/ref/sockets.rst
  51. +17 −0 docs/ref/synchronization.rst
  52. +825 −0 docs/shrapnel.svg
  53. +23 −0 docs/style.css
  54. +663 −0 docs/tutorial.rst
  55. +46 −0 docs/tutorial/proxy.py
  56. +4 −0 docs/tutorial/t0.py
  57. +25 −0 docs/tutorial/t1.py
  58. +26 −0 docs/tutorial/t2.py
  59. +32 −0 docs/tutorial/t3.py
  60. +253 −0 docs/tutorial/worms.py
  61. +39 −17 setup.py
  62. +63 −0 test/test_event_queue.py
View
@@ -5,5 +5,7 @@ build/
coro/_coro.[ch]
coro/oserrors.[ch]
coro/clocks/tsc_time.c
+coro/dns/packet.c
+coro/event_queue.cpp
*.pyc
*.so
View
@@ -8,6 +8,12 @@ around FreeBSD's kqueue() system call. It's designed for
single-process servers that can handle 10,000+ simultaneous network
connections.
+API Documentation
+=================
+
+See http://ironport.github.com/shrapnel/
+
+
Short History Of Python Coroutine Implementations
=================================================
View
@@ -20,113 +20,7 @@
# $Header: //prod/main/ap/shrapnel/coro/__init__.py#31 $
-"""Coroutine threading library.
-
-Introduction
-============
-Shrapnel is a cooperative threading library.
-
-Getting Started
-===============
-When your process starts up, you must spawn a thread to do some work, and then
-start the event loop. The event loop runs forever processing events until the
-process exits. An example::
-
- import coro
-
- def main():
- print 'Hello world!'
- # This will cause the process to exit.
- coro.set_exit(0)
-
- coro.spawn(main)
- coro.event_loop()
-
-Coroutines
-==========
-Every coroutine thread is created with either the `new` function (which does
-NOT automatically start the thread) or the `spawn` function (which DOES
-automatically start it).
-
-Every thread has a unique numeric ID. You may also set the name of the thread
-when you create it.
-
-Timeouts
-========
-The shrapnel timeout facility allows you to execute a function which will be
-interrupted if it does finish within a specified period of time. The
-`coro.TimeoutError` exception will be raised if the timeout expires. See the
-`with_timeout` docstring for more detail.
-
-If the event loop is not running (such as in a non-coro process), a custom
-version of `with_timeout` is installed that will operate using SIGALRM so that
-you may use `with_timeout` in code that needs to run in non-coro processes
-(though this is not recommended and should be avoided if possible).
-
-Thread Local Storage
-====================
-There is a tread-local storage interface available for storing global data this
-is thread-specific. You instantiate a `ThreadLocal` instance and you can
-assign attributes to it that will be specific to that thread. See the
-`ThreadLocal` docs for more detail.
-
-Signal Handlers
-===============
-By default when you start the event loop, two signal handlers are installed
-(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
-You can change this behavior by setting `install_signal_handlers` to False
-before starting the event loop.
-
-See `coro.signal_handler` for more detail on setting coro signal handlers.
-
-Selfishness
-===========
-Certain socket operations are allowed to try to execute without blocking if
-they are able to (such as send/receiving data on a local socket or on a
-high-speed network). However, there is a limit to the number of times a thread
-is allowed to do this. The default is 4. The default may be changed
-(`set_selfishness`) and the value on a per-thread may be changed
-(`coro.coro.set_max_selfish_acts`).
-
-Time
-====
-Shrapnel uses the `tsc_time` module for handling time. It uses the TSC
-value for a stable and high-resolution unit of time. See that module's
-documentation for more detail.
-
-A thread is always created when you start the event loop that will
-resynchronize the TSC relationship to accomodate any clock drift (see
-`tick_updater` and `tsc_time.update_time_relation`).
-
-Exception Notifier
-==================
-When a thread exits due to an exception, by default a stack trace is printed to
-stderr. You may install your own callback to handle this situation. See the
-`set_exception_notifier` function for more detail.
-
-Debug Output
-============
-The shrapnel library provides a mechanism for printing debug information to
-stderr. The `print_stderr` function will print a string with a timestamp
-and the thread number. The `write_stderr` function writes the string verbatim.
-
-Shrapnel keeps a reference to the "real" stderr (in `saved_stderr`) and the
-`print_stderr` and `write_stderr` functions always use the real stderr value. A
-particular reason for doing this is the backdoor module replaces sys.stderr and
-sys.stdout, but we do not want debug output to go to the interactive session.
-
-Profiling
-=========
-Shrapnel has its own profiler that is coro-aware. See `coro.profiler` for
-details on how to run the profiler.
-
-:Variables:
- - `all_threads`: A dictionary of all live coroutine objects. The key is
- the coroutine ID, and the value is the coroutine object.
- - `saved_stderr`: The actual stderr object for the process. This normally
- should not be used. An example of why this exists is because the
- backdoor replaces sys.stderr while executing user code.
-"""
+"""Coroutine threading library."""
from coro._coro import *
from coro._coro import _yield
@@ -166,13 +60,12 @@ def default_exception_notifier():
class InParallelError (Exception):
- """An error occurred in the `in_parallel` function.
+ """An error occurred in the :func:`in_parallel` function.
- :IVariables:
- - `result_list`: A list of ``(status, result)`` tuples. ``status`` is
- either `SUCCESS` or `FAILURE`. For success, the result is the return
+ :ivar result_list: A list of ``(status, result)`` tuples. ``status`` is
+ either :data:`SUCCESS` or :data:`FAILURE`. For success, the result is the return
value of the function. For failure, it is the output from
- `sys.exc_info`.
+ ``sys.exc_info``.
"""
def __init__(self, result_list):
@@ -195,17 +88,14 @@ def in_parallel (fun_arg_list):
This will block until all functions have returned or raised an exception.
- If one or more functions raises an exception, then the `InParallelError`
+ If one or more functions raises an exception, then the :exc:`InParallelError`
exception will be raised.
- :Parameters:
- - `fun_arg_list`: A list of ``(fun, args)`` tuples.
+ :param fun_arg_list: A list of ``(fun, args)`` tuples.
- :Return:
- Returns a list of return values from the functions.
+ :returns: A list of return values from the functions.
- :Exceptions:
- - `InParallelError`: One or more of the functions raised an exception.
+ :raises InParallelError: One or more of the functions raised an exception.
"""
# InParallelError, [(SUCCESS, result0), (FAILURE, exc_info1), ...]
@@ -257,14 +147,11 @@ def tick_updater():
def waitpid (pid):
"""Wait for a process to exit.
- :Parameters:
- - `pid`: The process ID to wait for.
+ :param pid: The process ID to wait for.
- :Return:
- Returns a tuple ``(pid, status)`` of the process.
+ :returns: A tuple ``(pid, status)`` of the process.
- :Exceptions:
- - `SimultaneousError`: Something is already waiting for this process
+ :raises SimultaneousError: Something is already waiting for this process
ID.
"""
if UNAME == "Linux":
@@ -290,26 +177,21 @@ def waitpid (pid):
def get_thread_by_id (thread_id):
"""Get a coro thread by ID.
- :Parameters:
- - `thread_id`: The thread ID.
+ :param thread_id: The thread ID.
- :Return:
- Returns the coroutine object.
+ :returns: The coroutine object.
- :Exceptions:
- - `KeyError`: The coroutine does not exist.
+ :raises KeyError: The coroutine does not exist.
"""
return all_threads[thread_id]
def where (co):
"""Return a string indicating where the given coroutine thread is currently
running.
- :Parameters:
- - `co`: The coroutine object.
+ :param co: The coroutine object.
- :Return:
- Returns a string displaying where the coro thread is currently
+ :returns: A string displaying where the coro thread is currently
executing.
"""
f = co.get_frame()
@@ -318,8 +200,7 @@ def where (co):
def where_all():
"""Get a dictionary of where all coroutines are currently executing.
- :Return:
- Returns a dictionary mapping the coroutine ID to a tuple of ``(name,
+ :returns: A dictionary mapping the coroutine ID to a tuple of ``(name,
coro, where)`` where ``where`` is a string representing where the
coroutine is currently running.
"""
@@ -339,13 +220,11 @@ def spawn (fun, *args, **kwargs):
Additional arguments and keyword arguments will be passed to the given function.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
- - `thread_name`: The name of the thread. Defaults to the name of the
+ :param fun: The function to call when the coroutine starts.
+ :param thread_name: The name of the thread. Defaults to the name of the
function.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
@@ -364,13 +243,11 @@ def new (fun, *args, **kwargs):
This will not start the coroutine. Call the ``start`` method on the
coroutine to schedule it to run.
- :Parameters:
- - `fun`: The function to call when the coroutine starts.
- - `thread_name`: The name of the thread. Defaults to the name of the
+ :param fun: The function to call when the coroutine starts.
+ :param thread_name: The name of the thread. Defaults to the name of the
function.
- :Return:
- Returns the new coroutine object.
+ :returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
@@ -457,8 +334,7 @@ def install_thread_emulation():
def coro_is_running():
"""Determine if the coro event loop is running.
- :Return:
- Returns True if the event loop is running, otherwise False.
+ :returns: True if the event loop is running, otherwise False.
"""
return event_loop_is_running
@@ -468,8 +344,7 @@ def sigterm_handler (*_unused_args):
def event_loop (timeout=30):
"""Start the event loop.
- :Parameters:
- - `timeout`: The amount of time to wait for kevent to return
+ :param timeout: The amount of time to wait for kevent to return
events. You should probably *not* set this value.
"""
global event_loop_is_running, with_timeout, sleep_relative
Oops, something went wrong.

0 comments on commit e921e61

Please sign in to comment.