Permalink
Browse files

Initial otrace revision for git

  • Loading branch information...
0 parents commit 58ec2d7c7dcb96f68137feb42935400b2a7fca5a @mitotic committed Jun 4, 2012
Showing with 5,530 additions and 0 deletions.
  1. +3 −0 .gitignore
  2. +281 −0 GettingStarted.rst
  3. +203 −0 README.rst
  4. +108 −0 hello_trace.py
  5. +4,923 −0 otrace.py
  6. +12 −0 setup.py
3 .gitignore
@@ -0,0 +1,3 @@
+.DS_Store
+*~
+*.pyc
281 GettingStarted.rst
@@ -0,0 +1,281 @@
+Getting started with *otrace* and *oshell*
+*********************************************************
+.. sectnum::
+.. contents::
+
+This tutorial introduces the main features of *otrace* by explaining
+how to "debug" the demo program, ``hello_trace.py``,
+included in the distribution.
+
+Installation
+==============================
+
+Download the latest version of the *otrace* zip archive. The unzipped
+archive should contain the following files (and perhaps more):
+
+ ``hello_trace.py otrace.py README.rst setup.py``
+
+All the code for the *otrace* module is contained in a single file,
+``otrace.py``. To use it without installing it, just ensure that it is
+present in the module load path. If you wish to install *otrace*, type:
+
+ ``python setup.py``
+
+help, cd, ls, and view commands
+====================================================
+
+``hello_trace.py`` is a simple multi-threaded web server using the
+``BaseHTTPServer`` module. It listens on port 8888 and displays a simple
+form where the user inputs a number and the server displays the
+reciprocal of the number. Inputting a zero value will raise an exception,
+which will be used to illustrate the capabilities of *otrace*.
+
+Run ``hello_trace.py`` from the command line (user input appears after
+the prompt ">"). The ``help`` command displays all the available *oshell* commands::
+
+ otrace$ ./hello_trace.py
+ Listening on port 8888
+ ***otrace object shell (v0.30)*** (type 'help' for info)
+ > help
+ Commands:
+ alias cd cdls del dn edit exec help ls popd
+ pr pushd pwd quit read repeat resume rm save set
+ swapd tag trace tracing unpatch untag untrace up view
+
+ If you omit the command, "pr" is assumed.
+ Use TAB key for command completion.
+ Type "help <command>" or "help *" for more info
+
+ See http://info.mindmeldr.com/code/otrace for documentation
+ globals> help pwd
+ pwd [-a] # Print current working "directory"
+
+ -a Print all paths in stack; top first
+
+The ``pwd`` command displays the current working directory.
+The ``ls`` command displays directory names and content. The special directory name **~~**
+refers to the most recent *trace context*, and is initially
+``/osh/globals``. There are also other special directory names of the
+form ~~<letter>::
+
+ > pwd
+ /osh/globals
+ > ls ~~
+ /osh/globals
+ > ls ~~g
+ /osh/globals
+ > ls ~~w
+ /Users/rsarava/app4/repo/meldr-hg/otrace
+
+The ``ls`` options *-c*, *-f*, *-m*, *-v* can be used to selectively display
+only the *classes*, *functions/methods*, *modules*, and *variables* in
+a directory. Initially we examine the ``globals`` directory, which
+contains three classes, ``GetHandler, MultiThreadedServer, and OShell``::
+
+ > ls
+ BaseHTTPServer GetHandler MultiThreadedServer OShell
+ Page_template Request_stats SocketServer http_addr
+ http_port http_server logging sys
+ test_fun trace_shell traceassert urlparse
+ > ls -c
+ GetHandler MultiThreadedServer OShell
+
+The ``cd`` command is used to change directories. Switching to the
+directory of class ``GetHandler``, we note that it supports several methods, many of which
+are inherited::
+
+ > cd GetHandler
+ osh..GetHandler> pwd
+ /osh/globals/GetHandler
+ osh..GetHandler> ls -f
+ address_string date_time_string do_GET
+ end_headers finish handle
+ handle_one_request log_date_time_string log_error
+ log_message log_request parse_request
+ respond send_error send_header
+ send_response setup version_string
+
+The ``ls`` options *-..* can be used to exclude inherited attributes
+of a class, and we note that ``GetHandler`` has two methods of its own::
+
+ osh..GetHandler> ls -f -..
+ do_GET respond
+
+We can examine the source code for the ``respond`` method using the
+``view`` command with the *-i* (inline-display) option::
+
+ osh..GetHandler> view -i respond
+ def respond(self, number):
+ # Respond to request by processing user input
+ number = float(number)
+
+ # Diagnostic print (initially commented out)
+ ##if number <= 0.001:
+ ## print "Client address", self.client_address
+
+ # Trace assertion (initially commented out)
+ ##traceassert(number > 0.001, label="num_check")
+
+ # Compute reciprocal of number
+ response = "The reciprocal of %s is %s" % (number, 1.0/number)
+ return response
+
+trace command
+===============================================
+
+The ``trace`` command is used to trace functions and methods. Without
+any options, it simply traces exceptions. The ``-c <condition>``
+option, where ``<condition>`` may be
+``call``, ``return``, or ``all``, may be used to trace function/method
+calls, returns, or both. ``<condition>`` may also be
+``argname1.comp1==value1,argname2!=value2,...`` to trace on argument
+value matching (values with commas/spaces must be quoted; the special
+argument name ``return`` may also be used).
+Without any arguments, the ``trace`` command displays currently traced names.
+Next, we initiate tracing on the ``respond`` method using the
+``trace`` command::
+
+ osh..GetHandler> trace respond
+ Tracing GetHandler.respond
+ osh..GetHandler> trace
+ GetHandler.respond
+
+Now we load the URL *http://localhost:8888* in the browser, and enter
+the number 3 followed by the number zero in the input form. A log message
+is generated for each value, and the zero value triggers a
+``ZeroDivisionError`` exception in the ``respond`` method.
+In the exception backtrace shown below, note the additional methods ``wrapped``
+and ``trace_function_call`` between ``do_GET`` and ``respond``. These
+are inserted by ``otrace`` for tracing::
+
+ rootW path=/?number=3
+ rootW path=/?number=0
+ GetHandler.respond:ex-ZeroDivisionError:23-01-33
+ ----------------------------------------
+ Exception happened during processing of request from ('127.0.0.1', 59872)
+ Traceback (most recent call last):
+ ...
+ File "./hello_trace.py", line 61, in do_GET
+ self.wfile.write(Page_template % self.respond(number))
+ File "/Users/rsarava/app4/repo/meldr-hg/otrace/otrace.py", line 4535, in wrapped
+ return cls.trace_function_call(info, *args, **kwargs)
+ File "/Users/rsarava/app4/repo/meldr-hg/otrace/otrace.py", line 4289, in trace_function_call
+ return_value = info.fn(*args, **kwargs)
+ File "./hello_trace.py", line 71, in respond
+ response = "The reciprocal of %s is %s" % (number, 1.0/number)
+ ZeroDivisionError: float division by zero
+ ----------------------------------------
+
+When a trace condition occurs, like an exception in a traced function or method, a trace id
+``GetHandler.respond:ex-ZeroDivisionError:23-01-33`` is generated and displayed,
+as shown above. Also, the default action of the ``trace`` command is
+to create a new virtual directory
+``/osh/recent/exceptions/GetHandler.respond/ex-ZeroDivisionError/23-01-33``
+to hold the *trace context* for the event. The shorthand notation
+**~~** can be used to display the most recent *trace context*::
+
+ > ls ~~
+ /osh/recent/exceptions/GetHandler.respond/ex-ZeroDivisionError/23-01-33
+ > cd ~~
+ GetHandler..01-33> pwd
+ /osh/recent/exceptions/GetHandler.respond/ex-ZeroDivisionError/23-01-33
+
+The trace context contains information about the function like
+argument values and the call stack.::
+
+ GetHandler..01-33> ls
+ __down __trc number self
+ GetHandler..01-33> ls -l
+ __down = {path_comps, __trc, __up, __down, number, self, query_args}
+ __trc = {exc_context, thread, framestack, frame, related, funcname, context, exc_stack, where, id, argvalues}
+ number = 0.0
+ self = <__main__.GetHandler instance at 0x108a34d88>
+ GetHandler..01-33> cd __trc
+ osh..__trc> ls
+ argvalues context exc_context exc_stack frame framestack
+ funcname id related thread where
+ osh..__trc> ls -l where
+ where = '__bootstrap-->__bootstrap_inner-->run-->process_request_thread-->
+ finish_request-->__init__-->handle-->handle_one_request-->do_GET-->respond'
+ osh..__trc>
+
+
+edit and traceassert
+=========================================================
+
+The ``edit`` command is perhaps the most useful command in *otrace*. It
+allows you to modify (`monkey patch <http://en.wikipedia.org/wiki/Monkey_patch>`_) any function or method in the
+running program. In particular, it makes it easy to use the "oldest"
+debugging technique, viz., inserting ``print`` statements in the code,
+without having to modify the actual source code files.
+
+Now that we know the there is an exception occurring in the method
+``respond``, we pretend that we don't know the exact cause, and will
+use the ``traceassert`` function to determine the cause. The ``traceassert``
+functions has the signature ``traceassert(condition, label="", action="")``.
+As long as ``condition`` is true, ``traceassert`` simply returns. If
+``condition`` is false, the call is logged and a *trace context*
+virtual directory is created.
+
+We suspect that the exception is caused because the user entered a
+number that was too small. First, we switch off *safe mode*, which
+disallows code editing. We then use the ``edit`` command to modify
+the ``respond`` method in the running program to insert a
+call to ``traceasset``. (Actually ``hello_trace.py`` already has a
+``traceassert`` call that is commented out. We simply uncomment it,
+as well as the diagnostic ``print`` statement, via the ``edit`` command.)::
+
+ osh..__trc> cd ~~g
+ globals> set safe_mode false
+ safe_mode = False
+ globals> edit GetHandler.respond
+ Patched GetHandler.respond:
+
+Now the call ``traceassert(number > 0.001, label="num_check")`` has been
+inserted into ``GetHandler.respond``. In the browser, enter the number
+2 and then the number 0.0005. The latter value triggers a false
+condition on the ``traceassert``. We switch to the assert trace
+context directory ``/osh/recent/asserts/GetHandler.respond/as-num_check/23-40-13``,
+which allows us to examine the local variables when the assertion failed::
+
+ rootW path=/?number=2
+ rootW path=/?number=0.0005
+ Client address ('127.0.0.1', 64211)
+ GetHandler.respond:as-num_check:23-40-13
+
+ > ls ~~
+ /osh/recent/asserts/GetHandler.respond/as-num_check/23-40-13
+ > cd ~~
+ GetHandler..40-13> ls
+ __down __trc number self
+ GetHandler..40-13> self.headers["User-Agent"]
+ Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 Safari/534.55.3
+ GetHandler..40-13> self.client_address
+ ('127.0.0.1', 64211)
+
+The default action when the traceassert condition is false is to
+create the trace context directory. The ``action`` argument to
+``traceassert`` can be used set a breakpoint when the assertion fails.
+
+unpatch and untrace
+=========================================================
+
+After debugging is complete, the ``unpatch`` command can be used to
+restore the original code for ``GetHandler.respond``.
+The ``untrace`` command can be used to switch off tracing::
+
+ globals> cd /osh/patches
+ patches> ls
+ GetHandler.respond
+ patches> unpatch GetHandler.respond
+ Unpatching GetHandler.respond
+ patches> cd ~~g
+ patches> trace
+ GetHandler.respond
+ globals> untrace GetHandler.respond
+ untraced GetHandler.respond
+ globals>
+
+.. |date| date::
+
+*Last modified:* |date|
203 README.rst
@@ -0,0 +1,203 @@
+otrace: An object-oriented debugger for nonlinear tracing
+*********************************************************************************
+.. sectnum::
+.. contents::
+
+This README file provides a brief introduction to *otrace*.
+Additional documentation and other information can be found on the
+project home page,
+`mindmeldr.com/code/otrace <http://info.mindmeldr.com/code/otrace>`_.
+
+Installation
+==============================
+
+Download the latest version of the *otrace* zip archive. The unzipped
+archive should contain the following files (and perhaps more):
+
+ ``hello_trace.py otrace.py README.rst setup.py``
+
+All the code for the *otrace* module is contained in a single file,
+``otrace.py``. To use it without installing it, just ensure that it is
+present in the module load path. If you wish to install *otrace*, type:
+
+ ``python setup.py``
+
+Usage
+=================================
+
+*otrace* may be used as:
+ - a tracing tool for debugging web servers and interactive programs
+ - a console or dashboard for monitoring production servers
+ - a teaching tool for exploring the innards of a program
+ - a code patching tool for unit testing
+
+*otrace* does not consume any resources until some tracing action is
+initiated. So it can be included in production code without any
+performance penalty.
+*otrace* works well with detached server processes (*daemons*)
+via the GNU `screen <http://www.gnu.org/software/screen>`_
+utility that emulates a terminal.
+
+*otrace* is meant to be used in conjunction with an *event loop*, which
+is usually present in programs that interact with users such as web
+servers or GUI applications. *otrace* takes control of the terminal,
+and would not work very well with programs that read user input
+directly from the terminal (or standard input).
+
+To use *otrace*, simply ``import otrace`` and instantiate the class ``otrace.OShell``,
+which provides a unix-like shell interface to interact with a running
+program via the terminal.
+
+Here is a simple server example::
+
+ import BaseHTTPServer
+ from SimpleHTTPServer import SimpleHTTPRequestHandler
+ from otrace import OShell, traceassert
+
+ http_server = BaseHTTPServer.HTTPServer(("", 8888), SimpleHTTPRequestHandler)
+ oshell = OShell(locals_dict=locals(), globals_dict=globals(),
+ new_thread=True, allow_unsafe=True, init_file="server.trc")
+ try:
+ oshell.loop()
+ http_server.serve_forever() # Main event loop
+ except KeyboardInterrupt:
+ oshell.shutdown()
+
+*Usage notes:*
+
+ - If you run in *oshell* in its own daemon thread as shown above, use
+ the ^C sequence to abort the main thread, and call ``OShell.shutdown``
+ from main thread to cleanup terminal I/O etc.
+
+ - If you run *oshell* in the main thread and the event loop in a
+ separate thread, ^C will abort and cleanup *oshell*. You may need to
+ shutdown the event loop cleanly after that.
+
+ - Install the python ``readline`` module to enable *TAB* command completion.
+
+ - To start a detached server (daemon) process, use the command:
+ ``screen -d -m -S <screen_name> <executable> <argument1> ...``
+ To attach a terminal to this process, use:
+ ``screen -r <screen_name>``
+
+ - By default, *otrace* logs to the ``logging`` module. Subclass
+ ``TraceCallback``, overriding the methods ``callback`` and ``returnback``
+ to implement your own logging (see ``DefaultCallback`` for a simple example)
+
+Synopsis
+==========================================
+
+*otrace* uses a *Virtual Directory Shell Interface* which maps all the
+objects in a a running python program to a virtual filesystem mounted in
+the directory ``/osh`` (sort of like the unix ``/proc`` filesystem, if you are
+familiar with it). Each module, class, method, function, and variable in the global namespace
+is mapped to a virtual file within this directory.
+For example, a class ``TestClass`` in the ``globals()`` dictionary can be accessed as::
+
+ /osh/globals/TestClass
+
+and a method ``test_method`` can be accessed as::
+
+ /osh/globals/TestClass/test_method
+
+and so on.
+
+*otrace* provides a unix shell-like interface, *oshell*, with commands
+such as ``cd``, ``ls``, ``view``, and ``edit`` that can be used navigate, view,
+and edit the virtual files. Editing a function or method
+"`monkey patches <http://en.wikipedia.org/wiki/Monkey_patch>`_" it,
+allowing the insertion of ``print`` statements etc. in the running program.
+
+The ``trace`` command allows dynamic tracing of function or method invocations,
+return values, and exceptions. This is accomplished by
+dynamically *decorating* (or *wrapping*) the function to be traced.
+When a trace condition is satisfied, the function-wrapper saves *context information*, such as
+arguments and return values, in a newly created virtual directory in::
+
+ /osh/recent/*
+
+These *trace context* directories can be navigated just like
+``/osh/globals/*``. (If there are too many trace contexts, the oldest
+ones are deleted, unless they have been explicitly *saved*.)
+
+*oshell* allows standard unix shell commands to be interspersed with
+*oshell*-specific commands. The path of the "current working directory"
+determines which of the these two types of commands will be executed.
+If the current working directory is not in ``/osh/*``, the command is
+treated as a standard unix shell command (except for ``cd``, which is
+always handled by *oshell*.)
+
+Commands
+=================
+*oshell* supports the following commands ([..] denotes optional
+parameters; | denotes alternatives)::
+
+ alias name cmd <arg\*> <arg\1>... # Define alias for command
+ cd [pathname] # change directory to "pathname", which may be omitted, "..", or "/" or a path
+ cdls [pathname] # cd to "pathname" and list "files" (cd+ls)
+ del [trace_id1..] # Delete trace context
+ dn # Command alias to move down stack frames in a trace context
+ edit [-f] (filename|class[.method]) [< readfile] # Edit/patch file/method/function
+ exec python_code # Execute python code (also !<python_code>)
+ help [command|*] # Display help information
+ ls [-a] [-l] [-t] [-v] [-(.|..|.baseclass)] [pathname1|*] # List pathname values (or all pathnames in current "directory")
+ pr python_expression # Print value of expression (DEFAULT COMMAND)
+ pwd # Print current working "directory"
+ quit # Quit shell
+ read filename # Read input lines from file
+ repeat command # Repeat command till new input is received
+ resume [trace_id1..] # Resume from breakpoint
+ rm [-r] [pathname1..] # Delete entities corresponding to pathnames (if supported)
+ save [trace_id1..] # Save current or specified trace context
+ set [parameter [value]] # Set (or display) parameter
+ tag [(object|.) [tag_str]] # Tag object for tracing
+ trace [-a (break|debug|hold|tag)] -c [call|return|all|tag|comma_sep_arg_match_conditions] [-n +/-count] ([class.][method]|key_path|:<label>|log:<str>) # Enable tracing for class/method/key/label/log on matching condition
+ tracing [on|off] # Enable/disable tracing
+ unpatch class[.method]|* [> savefile] # Unpatch method (and save patch to file)
+ untag [object|.] # untag object
+ untrace [class.][method] # Disable tracing for class/method
+ up # Command alias to move up stack frames in a trace context
+ view [-d] [-i] [class/method/file] # Display source/doc for objects/traces/files
+
+The default command is ``pr``, which evaluates an expression. So you
+can simply type a python variable to print out its value. You can also
+insert ``otrace.traceassert(<condition>,label=..,action=..)`` to trace
+assertions.
+
+
+Caveats
+===============================
+
+ - *Reliability:* This software has not been subject to extensive testing. Use at your own risk.
+
+ - *Thread safety:* In principle, *otrace* should thread-safe, but more testing is needed to confirm this in practice.
+
+ - *Memory leaks:* The trace contexts saved by *otrace* could potentially lead to increased memory usage. Again, only experience will tell.
+
+ - *Current limitations:*
+ * Decorated methods cannot be patched.
+ * TAB command completion is a work in progress.
+ * Spaces and other special characters in command arguments need to be handled better.
+
+
+Credits
+===============================
+
+*otrace* was developed as part of the `Mindmeldr
+<http://mindmeldr.com>`_ project, which is aimed at improving classroom interaction.
+
+*otrace* was inspired by the following:
+ - the tracing module `echo.py <http://wordaligned.org/articles/echo>`_ written by Thomas Guest <tag@wordaligned.org>. This nifty little program uses decorators to trace function calls.
+
+ - the python ``dir()`` function, which treats objects as directories. If objects are directories, then shouldn't we be able to inspect them using the familiar ``cd`` and ``ls`` unix shell commands?
+
+ - the unix `proc <http://en.wikipedia.org/wiki/Procfs>`_ filesystem, which cleverly maps non-file data to a filesystem interface mounted at ``/proc``
+
+ - the movie `Being John Malkovich <http://en.wikipedia.org/wiki/Being_John_Malkovich>`_ (think of ``/osh`` as the portal to the "mind" of a running program)
+
+
+License
+=====================
+
+*otrace* is distributed as open source under the `BSD-license <http://www.opensource.org/licenses/bsd-license.php>`_.
+
108 hello_trace.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+
+# hello_trace.py: demo application for otrace to compute reciprocal of a number
+
+import BaseHTTPServer
+import logging
+import SocketServer
+import sys
+import urlparse
+
+from otrace import OShell, traceassert
+
+Page_template = """<html>
+<head>
+ <title>Hello Trace</title>
+</head>
+<body>
+ <h2>Hello Trace</h2>
+ <form method="get" action="/">
+ Find reciprocal of:
+ <input id="number" name="number" type="text" autocomplete="off" autofocus="autofocus"></input>
+ <input type="submit" value="Submit" />
+ </form>
+ <p>
+ <span>%s</span>
+</body>
+</html>
+"""
+
+# Request statistics
+Request_stats = {"count":0, "path":""}
+
+class GetHandler(BaseHTTPServer.BaseHTTPRequestHandler):
+ def do_GET(self):
+ # Process GET request
+ path_comps = urlparse.urlparse(self.path)
+ query_args = urlparse.parse_qs(path_comps.query)
+
+ logging.warning("path=%s", self.path)
+
+ # Update request statistics
+ Request_stats["count"] += 1
+ Request_stats["path"] = self.path
+
+ if path_comps.path == "/favicon.ico":
+ self.send_error(404)
+ return
+
+ # Retrieve user input
+ number = query_args.get("number", [None])[0]
+
+ self.send_response(200)
+ self.send_header("Content-Type", "text/html")
+ self.end_headers()
+
+ if number is None:
+ # No user input; display input form
+ self.wfile.write(Page_template % "")
+ else:
+ # Process user input and display response
+ self.wfile.write(Page_template % self.respond(number))
+
+ def respond(self, number):
+ # Respond to request by processing user input
+ number = float(number)
+
+ # Diagnostic print (initially commented out)
+ ##if number <= 0.001:
+ ## print "Client address", self.client_address
+
+ # Trace assertion (initially commented out)
+ ##traceassert(number > 0.001, label="num_check")
+
+ # Compute reciprocal of number
+ response = "The reciprocal of %s is %s" % (number, 1.0/number)
+ return response
+
+ def log_message(self, format, *args):
+ # Suppress server logging
+ return
+
+class MultiThreadedServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer):
+ pass
+
+if __name__ == "__main__":
+ http_addr = ""
+ http_port = 8888
+
+ # HTTP server
+ http_server = MultiThreadedServer((http_addr, http_port), GetHandler)
+ print >> sys.stderr, "Listening on port %d" % http_port
+
+ # Test function that raises an exception
+ def test_fun():
+ raise Exception("TEST EXCEPTION")
+
+ # Initialize OShell instance (to run on separate thread)
+ trace_shell = OShell(locals_dict=locals(), globals_dict=globals(), allow_unsafe=True,
+ init_file="hello_trace.trc", new_thread=True)
+
+ try:
+ # Start oshell
+ trace_shell.loop()
+
+ # Start server
+ http_server.serve_forever()
+ except KeyboardInterrupt:
+ trace_shell.shutdown()
4,923 otrace.py
4,923 additions, 0 deletions not shown because the diff is too large. Please use a local Git client to view these changes.
12 setup.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+
+from distutils.core import setup
+
+setup(name='otrace',
+ version='0.30',
+ description='OTrace: An object-oriented shell for nonlinear tracing',
+ author='Ramalingam Saravanan',
+ author_email='sarava@sarava.net',
+ url='http://info.mindmeldr.com/code/otrace',
+ py_modules=['otrace'],
+ )

0 comments on commit 58ec2d7

Please sign in to comment.