Skip to content
This repository

Emacs to Python interface

Fetching latest commit…

Cannot retrieve the latest commit at this time

README
* README for `Pymacs'				allout -*- outline -*-

.. Presentation.

. : What is Pymacs?

    Pymacs is a powerful tool which, once started from Emacs, allows both-way
    communication between Emacs LISP and Python.  Pymacs aims at using
    Python as an extension language for Emacs.  One may load and use Python
    modules from within Emacs LISP code.  Python functions may themselves
    use Emacs services, and handle LISP objects kept in LISP space.

    The goals are to write "naturally" in both languages, debug with ease,
    fall back gracefully on errors, and allow full cross-recursivity.

    It is not expected that Pymacs users have a deep knowledge of both Emacs
    LISP and Python, yet knowledge always helps!  As Python scripting is the
    main goal for Pymacs, you'll find at the end of this documentation a few
    examples meant for Python users having a limited experience with Emacs.

. : Warnings to Pymacs users.

    This is alpha status software: specifications are not frozen, so be
    prepared to later adapt your code to specification changes.  Report the
    problems you see to François Pinard at `pinard@iro.umontreal.ca'.

. : History and references.

    Pymacs revisits previous Cedric Adjih's works about running Python as a
    process separate from Emacs.  See `http://www.crepuscule.com/pyemacs/',
    or write Cedric at `adjih-pam@crepuscule.com'.  Cedric presented
    `pyemacs' to me as a proof of concept.  I spiced the `pyemacs' concept
    with a few simplification ideas on my own, and decided to drop the `e'
    from `pyemacs' to witness that simplification :-).  Cedric told me that
    there also exist some older patches for linking Python right into XEmacs.

    Brian McErlean contacted me, as he independently and simultaneously
    wrote a very similar project.  Amusing coincidence, he even chose
    `pymacs' as a name.  As he paid good attention to complex details that
    escaped my courage, his help and collaboration have been beneficial.
    You may reach Brian at `brianmce@crosswinds.net'.

    One other reference of interest is Doug Bagley shoot out project,
    which compares the relative speed of many popular languages.  See
    `http://www.bagley.org/~doug/shootout/' for more information.

.. Installation.

. : Unpack the archive.

    You unpacked the archive already if you are reading this file! :-).

. : Edit `Makefile'.

    The `Makefile' should be modified to suit your own usages and habits.
    The following instructions might help you deciding which value you should
    give to some `Makefile' variables.  Whenever a command is given below,
    always complete each command with RET (or Enter).  We presume you are
    already familiar with the Emacs manul notation for keys and commands.

.  , `lispdir'.

     This is where `pymacs.el', the Emacs side code of Pymacs, should go:
     somewhere on your Emacs `load-path'.  To see the current value of this
     variable, start Emacs and do `C-h v load-path'.

.  , `pythondir'.

     This is where `pymacs.py', the Python side code of Pymacs, should go:
     somewhere on your Python `sys.path'.  To see the current value of this
     variable, start an interactive Python session and successively type
     `import sys' and `print sys.path'.

.  , `bindir'.

     This is where `pymacs-services', the Python process starter, should
     go: somewhere on your shell `PATH'.  To see the current value of this
     variable, type `echo $PATH' in your shell.

.  , `pymacsdir'.

     This is where you intend to keep your Python code to extend Emacs.
     You should make sure this variable is empty if you do not want to use
     a special directory for this purpose, as you might very well save your
     Python extensions in any directory from your usual Python search path.

. : Copy files.

    To copy various Pymacs files into their final destination, editing
    them as needed while doing so, just do `make install'.

. : Check installation.

    Your installation choices are consistent with your various search paths.
    To check that `pymacs.el' is properly installed, start Emacs and give
    it the command `M-x load-library RET pymacs': you should not receive
    any error.  To check that `pymacs.py' is properly installed, start
    an interactive Python session and type `import pymacs': you should
    not receive any error.  To check that `pymacs-services' is properly
    installed, type `pymacs-services </dev/null' in a shell; you should
    then receive two lines: one ending with "(pymacs-version VERSION)",
    and another saying: "Protocol error: `>' expected.".

. : Modify `.emacs'.

    The ".emacs" file is not given in the distribution, you likely have
    one already in your home directory.  You need to add these two lines:

    (autoload 'pymacs-eval "pymacs" nil t)
    (autoload 'pymacs-load "pymacs" nil t)

    If the file "$HOME/.emacs" does not exist, merely create it with the
    two above lines.  You are now all set to use Pymacs.

. : Try Pymacs!

    Start a fresh Emacs session, and within it, type `M-x pymacs-eval'.
    Emacs should prompt you for a Python expression.  Try "`2L**111`" (type
    the backquotes, but not the external double-quotes).  The minibuffer
    should display `2596148429267413814265248164610048L'.

    Here is another test.  Within an Emacs session, type `M-x pymacs-load'.
    Emacs will then prompt you for a Python module name.  Reply `os'.
    After Emacs prompts you for a prefix, merely hit Enter to accept the
    default prefix.  This should have the effect of importing the Python "os"
    module within Emacs.  Typing `M-: (os-getcwd)' should echo the current
    directory in the message buffer, after executing the Python function.

. : Caveats.

    I noticed that Emacs 20 has hash tables, yet not weak ones.  It silently
    ignores the request for creating weak hash tables.  So Pymacs does
    not crash, but the table does not fill its role.  The consequence is
    that, with older Emacses, memory will silently leak on both Emacs and
    Python sides.  This is not too serious, if users carefully refrain
    from often transmitting transient complex Pythons objects to Emacs.

.. Usage on the LISP side.

. : `pymacs-eval'.

    Function `(pymacs-eval TEXT)' gets TEXT evaluated as a Python expression,
    and returns the value of that expression.

    Python `None', as well as the empty tuple `()', are returned as `nil'
    in LISP.  Python numbers, either integer or floating, and strings, have
    a natural expression in LISP which is then used.  However, long Python
    integers, complex numbers, and Unicode strings may not be returned.
    The special `lisp.SYMBOL' or `lisp[STRING]' writings on the Python
    side (see later in this document) can be used to return LISP symbols.
    Python tuples are returned as proper LISP lists, Python lists as
    LISP vectors.  Other Python objects are returned opaquely.

. : `pymacs-load',

    Function `(pymacs-load MODULE PREFIX)' imports the Python MODULE into
    LISP space.  Each top-level function in the module produces a trampoline
    function in LISP having the same name, except that underlines in Python
    names are turned into dashes in LISP, and that PREFIX is uniformly added
    before the LISP name (as a way to avoid name clashes).  PREFIX may be
    omitted, in which case it defaults to MODULE followed by a dash.

    When later calling one of these functions, all provided arguments are
    converted to Python and transmitted, it is left to the Python side to
    check for argument consistency.  Keyword arguments are not supported.
    LISP `nil' is converted to Python `None'.  LISP numbers, either integer
    (including characters) or floating, and strings, are naturally converted
    to Python equivalents.  LISP strings are then considered non-mutable,
    as this is the most common usage, but this can be overridden (see later
    in this document).  LISP symbols are transmitted using the special
    `lisp.SYMBOL' or `lisp[STRING]' notation in Python, and all other LISP
    types are transmitted as LISP handles (see later in this document).

    The return value of these functions is converted back to LISP, as
    explained with `pymacs-eval' above.  Also, returning any LISP handle
    does in fact return the original LISP object behind that handle.

    Note that none of the imported Python function is marked interactive
    on the LISP side, so in particular, these cannot directly be bound
    to keys.  Emacs functions have the concept of user interaction for
    completing the specification of their arguments while being called.
    I do not see how to naturally retrofit that facility on the Python side.
    You need to define your own trampoline functions in LISP if you want
    them interactive.  See the examples provided elsewhere in this document.

    Calling `lisp.interactive(...)' in Python is not going to work.
    The requirement that "(interactive ...)" be first in a `defun' let me
    think that there is some magic in the Emacs LISP interpreter itself,
    which looks for that call _before_ the function is actually entered.
    One might supply the `interactive` declaration in Python doc-strings:
    John Aycock received many protests when he used doc-strings for SPARK.
    For one, I think Jonh did the most right thing for the problem.  Yet,
    I understand that users might prefer keeping doc-strings for themselves.

. : Expected usage.

    We do not expect that `pymacs-eval' will be much used, if ever.
    In practice, the LISP side of a Pymacs application might call
    `pymacs-load' a few times for linking into the Python modules,
    with the indirect effect of defining trampoline functions for these
    modules on the LISP side, than can be called like usual LISP functions.

    These imported functions are really those which are of interest for
    the user, and the preferred way to call Python services with Pymacs.

. : Special LISP variables.

    Users could alter the inner working of Pymacs through a few variables,
    which are documented here.  Except for `pymacs-load-path', which should
    be set before the first call to `pymacs-eval' or `python-load', the
    value of these variables can be changed at any time.

.  , pymacs-load-path

     Users might want to use special directories for holding their Python
     modules, when these modules are meant to be used from Emacs.  Best is
     to preset `pymacs-load-path, `nil' by default, to a list of these
     directory names.  (Tilde expansions and such occur automatically.)

.  , pymacs-trace-transit

     The `*Pymacs*' buffer, within Emacs, holds a trace of transactions
     between Emacs and Python.  When `pymacs-trace-transit' is `nil',
     and this is the default setting, the buffer only holds the last
     bi-directional transaction (a request and a reply).  If that variable
     is not `nil', all transactions are kept.  This could be useful for
     debugging, but the drawback is that this buffer could grow big over
     time, to the point of diminishing Emacs performance.

.  , pymacs-forget-mutability

     The default behaviour of Pymacs is to transmit LISP objects
     to Python in such a way thay they are fully modifiable from the
     Python side, would it mean triggering LISP functions to act on them.
     When `pymacs-forget-mutability' is not `nil', the behaviour is changed,
     and the flexibility is lost.  Pymacs then tries to expand proper lists
     and vectors as full copies when transmitting them on the Python side.
     This variable, seen as a user setting, is best left to `nil'.  It may
     be temporarily overriden within some functions, when deemed useful.

     There is no corresponding variable from objects transmitted to Emacs
     from Python.  Pymacs automatically expands what gets transmitted.
     Mutability is preserved only as a side-effect of not having a natural
     LISP representation for the Python object.  This assymetry is on
     purpose, yet debatable.  Maybe Pymacs could have a variable telling
     that mutability _is_ important for Python objects?  That would give
     Pymacs users the capability of restoring the symmetry somewhat, if
     they have a strong appetite for it.  But I'm not sure it would be
     worth the effort: I merely tried to guess what's most useful.

.  , pymacs-mutable-strings

     Strictly speaking, Emacs LISP strings are mutable. Yet, it does not
     come naturally to a Python programmer to modify a string "in-place",
     as Python strings are never mutable.  When `pymacs-mutable-strings' is
     `nil', and this is the default setting, LISP strings are transmitted
     to Python as Python strings, and so, loose their mutability.  If that
     variable is not `nil', LISP strings are rather passed as LISP handles.
     This variable is ignored whenever `pymacs-forget-mutability' is set.

.. Usage on the Python side.

. : Python setup.

    Pymacs requires little or no setup in the Python modules which are
    meant to be used from Emacs, for the simple situations where these
    modules receive nothing but numbers or strings, or return nothing but
    numbers or strings.  LISP `nil' corresponds to Python `None'.

    For other cases, Python modules should have `from pymacs import lisp'
    near their beginning.  Some users might `import pymacs', and then always
    refer to `pymacs.lisp', I find this cumbersome.  `from pymacs import *'
    would uselessly pollute the namespace of the module.  As things stand,
    the Python programmer does not need much beyond the `lisp' object from
    the Pymacs module.  For full flexibility however, one may also write:

        import pymacs
        lisp = pymacs.lisp

. : Response mode.

    When Python receives a request from Emacs in the context of Pymacs,
    and until it returns the reply, Emacs keeps listening to serve
    Python requests.  Emacs is not listening otherwise.  Consequently,
    Python should never attempt calling for Emacs services at other times.
    Other Python threads may not call Emacs without careful synchronisation.

. : LISP handles.

    When a Python function is called from LISP, the function arguments
    have already been converted to Python types from LISP types (see the
    description of `pymacs-load' above) and the function result is going to
    be converted back to LISP (see the description of `pymacs-eval' above).

    Several LISP objects do not have Python equivalents, like for Emacs
    windows, buffers, markers, overlays, etc.  It is nevertheless useful to
    pass them to Python functions, hoping that these Python functions will
    "operate" on these LISP objects.  Of course, the Python side may not
    itself modify such objects, it has to call for Emacs services to do so.
    LISP handles are a mean to ease this communication.

    Whenever a LISP object may not be converted to a Python object, an
    LISP handle is created and used instead.  Whenever that LISP handle is
    returned into LISP from a Python function, or is used as an argument
    to a LISP function from Python, the original LISP object behind the
    LISP handle is automatically retrieved.

    LISP handles are either instances of the `pymacs.Lisp' class, or
    of one of its subclasses.  If `object' is a LISP handle, and if the
    underlying LISP object is a LISP sequence, then whether `object[index]',
    `object[index] = value' and `len(object)' are meaningful, these may be
    used to fetch or alter an element of the sequence directly in LISP space.
    Also, if `object' corresponds to a LISP function,

    `object(ARGUMENTS)' may be used to apply the LISP function over the
    given arguments.  Since arguments have been evaluated the Python way on
    the Python side, it would be conceptual overkill evaluating them again
    the LISP way on the LISP side, so Pymacs manage to quotes arguments for
    defeating LISP evaluation.  The same logic applies the other way around.

    LISP handles have a `value()' method, which merely returns self.
    They also have a `copy()' method, which tries to "open the box"
    if possible.  LISP proper lists are turned into Python tuples, LISP
    vectors are turned into Python lists.  Then, modifying the structure
    of the copy on the Python side has no effect on the LISP side.

. : LISP symbols.

    `lisp' is a special object which has useful built-in magic.
    Its attributes do nothing but represent LISP symbols, created on the
    fly as needed (symbols also have their built-in magic).  Whenever
    `lisp.SYMBOL' gets transmitted to the Emacs side, the communication
    protocol takes care that it produces the proper corresponding LISP
    symbol.  The correspondance is established by replacing all underline
    characters in SYMBOL on the Python side, by dashes on the LISP side.
    This convention is welcome, as LISP programmers commonly prefer using
    dashes, where Python programmers use underlines.

    A few LISP symbols use special characters that would not be allowed
    in the name of a Python attribute, or else, using underlines, would
    give rise to ambiguous interpretation.  The notation `lisp[STRING]',
    where STRING ought to be a lexically valid LISP symbol name, refers to
    a LISP symbol having exactly STRING as a symbol name.  In that notation,
    there is no automatic underline to dash conversion.

    Except for `lisp.nil' or `lisp["nil"]', which are the same as `None',
    both `lisp.SYMBOL' and `lisp[STRING]' yield objects of `pymacs.Symbol'
    type.  These are genuine Python objects, that could be referred to
    by simple Python variables.  One can write `quote = lisp.quote',
    for example, and use `quote' afterwards to mean that LISP symbol.
    Here are other examples.  If a Python function received a LISP symbol
    as an argument, it can check with `==' if that argument is `lisp.never'
    or `lisp.ask'.  And a Python function may choose to return `lisp.t'.

    In Python, writing `lisp.SYMBOL = VALUE' or `lisp[STRING] = VALUE' does
    assign VALUE to the corresponding symbol in LISP space.  Beware that
    in such cases, the `lisp.' prefix may not be spared.  One cannot write
    `result = lisp.result' and hope that a later `result = 3' will have any
    effect in the LISP space: this would merely change the Python variable
    `result', which was a reference to a `pymacs.Symbol' instance, so it
    is now a reference to the number 3.

    The `pymacs.Symbol' class has `value()' and `copy()' methods.  One can
    use either `lisp.SYMBOL.value()' or `lisp.SYMBOL.copy()' to access
    the LISP value of a symbol, after conversion to some Python object,
    of course.  However, if `value()' would have given a LISP handle,
    `lisp.SYMBOL.copy()' has the same effect of `lisp.SYMBOL.value().copy()',
    that is, it returns the value of the symbol as opened as possible.

    A symbol may also be as if it was a Python function, in which case it
    really names a LISP function that should be applied over the following
    function arguments.  The result of the LISP function becomes the value
    of the call, with all due conversions of course.

. : Raw LISP expressions.

    Pymacs offers a device for evaluating a raw LISP expression expressed
    as a string.  One merely uses `lisp' as a function, like this:

    lisp("""
    ...
    POSSIBLY-LONG-LISP-EXPRESSION
    ...
    """)

    The LISP value of the expression becomes the value of the `lisp' call,
    after conversion back to Python.

. : Keybindings.

    To translate bindings like "C-x w", say, one might have to know a
    bit more how LISP processes string escapes like "\C-x" or "\M-\C-x"
    in LISP, and emulate it within Python strings, since Python does not
    have such escapes.  "\C-L", where L is an upper case letter, produces a
    character which ordinal is the result of subtracting 0x40 from ordinal
    of `L'.  "\M-" has the ordinal one gets by adding 0x80 to the ordinal
    of following described character.  So people can use self-inserting
    non-ASCII characters, "\M-" is given another representation, which is
    to replace the addition of 0x80 by prefixing with `ESC', that is 0x1b.

    So "\C-x" in Emacs is '\x18' in Python.  This is easily found, using an
    interactive Python session, by givin it: chr(ord('X') - ord('A') + 1).
    An easier way would be using the `kbd' function on the LISP side,
    like with lisp.kbd('C-x w') or lisp.kbd('M-<f2>').

    To bind the F1 key to the `helper' function in some `module':

        lisp.global_set_key([lisp.f1], lisp.module_helper)

    [item] is a Python list yielding a LISP vector.  `lisp.f1' translates
    to the LISP symbol `f1'.  So, Python `[lisp.f1]' is LISP `[f1]'.
    Keys like `[M-f2]' might require some more ingenuity, one may write
    either [lisp['M-f2']] or [lisp.M_f2] on the Python side.

.. Debugging.

. : The `*Pymacs*' buffer.

    The main debugging tool is the communication buffer between Emacs and
    Python, which is named `*Pymacs*'.  To make good use of it, first set
    `pymacs-trace-transit' to `t', so all exchanges are accumulated in
    that buffer.  It helps understanding the communication protocol,
    so it is shortly explained here.  Consider:

    ---------------------------------------------------------------------->
    (pymacs-eval "lisp('(pymacs-eval \"`2L**111`\")')")
    "2596148429267413814265248164610048L"
    ----------------------------------------------------------------------<

    Here, Emacs asks Python to ask Emacs to ask Python for a simple bignum
    computation.  Note that Emacs does not natively know how to handle
    big integers, nor has an internal representation for them.  This is
    why I use backticks, so Python returns a string representation of the
    result, instead of the result itself.  Here is a trace for this example.
    The `<' character flags a message going from Python to Emacs and is
    followed by an expression written in LISP.  The '>' character flags
    a message going from Emacs to Python and is followed by a expression
    written in Python.  The number gives the length of the message.

    ---------------------------------------------------------------------->
    <22   (pymacs-version "0.3")
    >49   eval("lisp('(pymacs-eval \"`2L**111`\")')")
    <25   (pymacs-eval "`2L**111`")
    >18   eval("`2L**111`")
    <47   (pymacs-reply "2596148429267413814265248164610048L")
    >45   reply("2596148429267413814265248164610048L")
    <47   (pymacs-reply "2596148429267413814265248164610048L")
    ----------------------------------------------------------------------<

    Python evaluation is done in the context of the `pymacs' module,
    so a mere `reply' really means `pymacs.reply'.  On the LISP side,
    there is no concept of module namespaces, so we use the `pymacs-'
    prefix as an attempt to stay clean.  Of course, users should ideally
    refrain from naming their LISP objects with a `pymacs-' prefix.

    `pymacs.reply' and `pymacs-reply' are special functions meant to indicate
    that an expected result is finally transmitted.  `pymacs.error' and
    `pymacs-error' are special functions that introduce a string which
    explains an exception which recently occurred.  `pymacs-expand' is
    a special function implementing the `copy()' methods of LISP handles
    or symbols.  In all other cases, the expression is a request for the
    other side, that request stacks until a corresponding reply is received.

. : Emacs usual debugging.

    The Python backtrace gets displayed in the mini-buffer (which in recent
    Emacses, is resized for the circumstance), and the LISP backtrace is
    simultaneously shown in the `*Backtrace*' window.  I made no kind of
    effort to transmit the LISP backtrace on the Python side, as I do not
    see a purpose for it: all debugging is done within Emacs windows anyway.

    If cross-calls between LISP and Python nest deeply, an error will raise
    successive exceptions alternatively on both size as requests unstack,
    and the diagnostic gets transmitted back and forth, slightly growing
    as we go.

. : Auto-reloading on save.

    I found useful to automatically `pymacs-load' some Python files whenever
    they get saved from Emacs.  Here is how I do it.  The code below assumes
    that Python files meant for Pymacs are kept in `~/share/emacs/python'.

    (defun fp-maybe-pymacs-reload ()
      (let ((pymacsdir (expand-file-name "~/share/emacs/python/")))
        (when (and (string-equal (file-name-directory buffer-file-name)
                                 pymacsdir)
                   (string-match "\\.py\\'" buffer-file-name))
          (pymacs-load (substring buffer-file-name 0 -3)))))
    (add-hook 'after-save-hook 'fp-maybe-pymacs-reload 'append)

.. Exemples.

. : Paul Winkler's

.  , The problem

     Let's say I have a a module, let's call it manglers.py, containing
     this simple python function:

     def break_on_whitespace(some_string):
         words = some_string.split()
         return '\n'.join(words)

     The goal is telling Emacs about this function so that I can call it on
     a region of text and replace the region with the result of the call.
     And bind this action to a key, of course, let's say `[f7]'.

     Note that the Emacs buffer should be handled in some way.  If this is
     not on the lisp side, it has to be on the Python side, but we cannot
     escape handling the buffer.  So, there is an equilibrium in the work
     to do for the user, that could be displaced towards LISP or towards
     Python.  For one, I would probably manage so the LISP side transmit
     the region as passed as arguments to the Python function, but let's
     not even do that, and rather discover the region from Python code.

.  , Python side

     Here is a first draft for the Python side of the problem:

     from pymacs import lisp

     def break_on_whitespace():
         start = lisp.point()
         end = lisp.mark()
         if start > end:
             start, end = end, start
         text = lisp.buffer_substring(start, end)
         words = text.split()
         replacement = '\n'.join(words)
         lisp.delete_region(start, end)
         lisp.insert(replacement)

     For various stylistic reasons, this could be rewritten into:

     from pymacs import lisp
     from string import join

     def break_on_whitespace():
         start, end = lisp.point(), lisp.mark()
         words = lisp.buffer_substring(start, end).split()
         lisp.delete_region(start, end)
         lisp.insert(join(words, '\n'))

     relying on the fact that for those LISP functions used here, `start'
     and `end' may be given in any order.

.  , Emacs side

     On the Emacs side, one would do:

     (pymacs-load "manglers")

     (defun break-on-whitespace ()
       (interactive)
       (manglers-break-on-whitespace))

     (global-set-key [f7] 'break-on-whitespace)
Something went wrong with that request. Please try again.