Support for rst-mode?

[njsmith]

Hi!

poly-markdown-mode is pretty sweet! I'm trying to figure out now if something similar can be accomplished for restructured text. The problem is that ReST's delimiter syntax is not quite as simple as most of the other examples I see... docs are here: http://www.sphinx-doc.org/en/stable/markup/code.html

But basically the main case one would like to handle looks like:

.. code-block:: python
   :linenos:

   def f():
       print("ReST is a bit complicated")

As the example above shows, ...

So:

• The literal .. code-block::, which can appear at any indentation
• Followed by the language name
• Followed by 0 or more lines that are indented to line up with the c in code-block
• Followed by a blank line
• and then an arbitrarily long block of source code that's indented to line up with the c in code-block, ending when you hit a dedent. (dedent = a line that has at least one non-whitespace character, and where the first whitespace character has less indentation than the code block)

I'd actually be really happy to see an example of how to make this work even if the only language supported was python, just to get started... I started fiddling with pm-create-indented-block-matchers but couldn't figure out quite what I was doing.

"would be nice" features:

• support code and sourcecode as substitutes for code-block (they're all identical, just different spellings)
• support for arbitrary languages using a multi-auto mode -- not sure how feasible this is. I note that the slim mode seems to hard-code the full list of possible modes, suggesting that there might be something tricky here once you've gone off the deep end trying to handle weird indentation-based delimiters?
• support the alternative syntax raw literal version, that just looks like

.. warning::

   You have to watch out for code like::

     def f():
         print("hello world")

   because ...

The key notes here are: (1) started by a line that doesn't start with .., but that does end with ::, (2) then a blank line, (3) then a block of code that's more-indented than the starting line.

The downside is that this syntax doesn't give any way to specify the embedded language, so the interpretation is up to context. E.g. github's .rst renderer treats this as a raw literal, but if you're writing docs in sphinx then it treats it as python... unless you have configured it to treat it as some other language in the sphinx config file. So in a perfect world we'd have a poly mode that detected this and was configurable about what language it used for it... but I'm also perfectly happy to stick with the more-verbose .. code-block:: syntax if I have to :-).

Any suggestions on how this could be accomplished?

[vspinu]

What you described should be pretty easy. There is already a function that creates a paired indentation matchers - pm-create-indented-block-matchers. You will likely need a modification of that but it should be rather straightforward. I should probably into it myself asap as well as there seem to be a bug in the function matchers (#107).

[Xaldew]

Hello!

I started looking into this based on the question on Emacs Stack Exchange. Since I'm not very familiar with ReST I created the naïve solution:

Restructued Text Testing
================

This is a document that uses various programming languages in order to
test the functionality of *mmm-mode* and *poly-mode* in *Emacs*.


Generic Code Block
------------------

This is a generic code-block. It should not receive special highlights.

::

    import sys

    class python_test(object):
        pass

    def main():
        pass

    if __name__ == '__main__':
        sys.exit(main())


This is a generic code-block without language specifiers.

.. code-block::

    import sys


    class python_test(object):
        pass


    def main():
        pass


    if __name__ == '__main__':
        sys.exit(main())


Python
------

.. sourcecode:: python

    import sys


    class python_test(object):
        pass


    def main(ab):
        """This does wierd things.

        .. Keyword Arguments:
        :param ab: Test.

        .. Returns:
        :return: The number 42.

        """
        return 42


    if __name__ == '__main__':
        sys.exit(main())

.. warning::

   The following code works on Python 2, but not Python 3:

   .. code-block:: python
      :caption: Python Warning Test

      print "Hello world"
      assert 5 / 2 == 2

   In Python 3, you need to add parentheses around ``"Hello world"``, and the division call
   will return 2.5 instead of 2.


C/C++
-----

.. highlight:: c++
   :caption: C/C++

    #include <iostream>

    using namespace std;

    class test { public: test() {} ~test() {} }

    int main(char argc, char *argv[])
    {
        (void) argc;
        (void) argv;
        cout << "Hello, World!" << endl;
        return 0;
    }


.. code:: c

    #include <stdio.h>

    struct test
    {
        int dummy;
    }

    int main(char argc, char *argv[])
    {
        (void) argc;
        (void) argv;
        printf("Hello, world!\n");
        return 0;
    }


Rust
----

How does it handle relatively new languages?

.. code-block:: rust
   :caption: Rust

    use std::fmt;

    fn main()
    {
        println!("Hello, world!");
    }


LaTeX
-----

.. code-block:: LaTeX
   :caption: Latex

    \documentclass{beamer}
    \usepackage{etex}
    \usepackage[latin1]{inputenc}
    \usepackage{mathtools}

    \title[]{Video Modeling}
    \subtitle{}
    \author{Gustaf Waldemarson}
    \institute{ARM Sweden AB}
    \date{\today}
    \subject{Computer Science}

    \begin{document}

    \begin{frame}
      \titlepage
      \begin{columns}
        \begin{column}{0.4\textwidth}
          \begin{figure}
            % \includegraphics[width=0.5\textwidth]{lund_university_seal}
          \end{figure}
        \end{column}
        \begin{column}{0.3\textwidth}
          % Push them apart a little more.
        \end{column}
        \begin{column}{0.4\textwidth}
          \begin{figure}
            \includegraphics[width=0.5\textwidth]{arm_logo}
          \end{figure}
        \end{column}
      \end{columns}
    \end{frame}

    \bgroup
    \setbeamercolor{background canvas}{bg=black}
    \begin{frame}[t,plain]{}{}
      \begin{center}
        {\tiny \textcolor{white}{The End}}
      \end{center}
    \end{frame}
    \egroup

    \end{document}


Make
----

.. code:: makefile

    CPPFLAGS= -D_POSIX_C_SOURCE=200809L
    CFLAGS=-g3 -std=c99
    LDLIBS=-lxcb
    PROGS= xcb_query_keymap xcb_events xcb_modmap xcb_keyboard_grab

    .PHONY: all
    all: $(PROGS)

    .PHONY: clean
    clean:
        $(RM) $(PROGS)


Shell
-----

.. code:: shell-script

    add_subtitles()
    {
        vid=${1:?"No video clip set."}
        sub=${2:?"No subtitle file set."}
        enc=${3:-UTF-8}
        lang=${4:-eng}
        scodec=${5:-srt}
        tmp=$(mktemp XXXXXXXX.mkv)
        ffmpeg \
            -y \
            -i ${vid} \
            -sub_charenc ${enc} \
            -i ${sub} \
            -map 0 \
            -map 1 \
            -c copy \
            -scodec ${scodec} \
            -metadata:s:s:0 language=${lang} \
            ${tmp}
        # Overwrite if successful.
        if [ $? -eq 0 ]; then
            mv -f ${tmp} ${vid}
        fi
    }


Lisp
----

.. code:: lisp

    (defparameter *small* 1)
    (defparameter *big* 100)

    (defun guess-my-number ()
         (ash (+ *small* *big*) -1))

    (defun smaller ()
         (setf *big* (1- (guess-my-number)))
         (guess-my-number))

    (defun bigger ()
         (setf *small* (1+ (guess-my-number)))
         (guess-my-number))

    (defun start-over ()
       (defparameter *small* 1)
       (defparameter *big* 100)
       (guess-my-number))


Emacs Lisp
----------

If it handled lisp okay, how does it handle the Emacs Lisp dialect? Or
other dialects for that matter?

.. code:: elisp

    (defun srt-renumber-file ()
    "Re-number all lines in the current subrip subtitle file."
      (interactive)
      (save-excursion
        (goto-char (point-min))
        (let ((cnt 0))
          (while (search-forward-regexp "^[0-9]+$")
            (replace-match (number-to-string (cl-incf cnt)))))))

Does spelling affect it?

.. code:: emacs-lisp

    (defun srt-renumber-file ()
    "Re-number all lines in the current subrip subtitle file."
      (interactive)
      (save-excursion
        (goto-char (point-min))
        (let ((cnt 0))
          (while (search-forward-regexp "^[0-9]+$")
            (replace-match (number-to-string (cl-incf cnt)))))))

[njsmith]

@Xaldew: awesome :-). To your test case I'd add an example of a code-block nested inside some other indented construct, like this:

.. warning::

   The following code works on Python 2, but not Python 3:

   .. code-block:: python

      print "Hello world"
      assert 5 / 2 == 2

   In Python 3, you need to add parentheses around ``"Hello world"``, and the division call
   will return 2.5 instead of 2.

[Xaldew]

@njsmith Ah, that's a good point. I'll edit the above post to contain that nesting test.

So far, I've created this:

;;; poly-rest-mode.el -- Polymode for ReST. -*- lexical-binding: t -*-
;;
;;; Commentary:
;; A mode for interacting with multiple kinds of code inside a ReST host-mode.
;;
;;; Code:

(require 'polymode)


(defcustom pm-host/ReST
  (pm-bchunkmode "ReST"
                 :mode 'rst-mode
                 :init-functions '(poly-rest-fixes))
  "Restructued Text host chunkmode."
  :group 'hostmodes
  :type 'object)

(defvar pm-ReST-code-regexp
  (regexp-opt '("code" "code-block" "sourcecode" "highlight"))
  "Regular expression to match all possible ReST code blocks.")

(defvar pm-ReST-head-regexp
  (concat "^[^ ]*\\( *\.\. +" pm-ReST-code-regexp "::.*\\)$")
  "Regular expression to match the `head' of a ReST code block.")

(defvar pm-ReST-retriever-regexp
  (concat "^[^ ]*\\(?: *\.\. +" pm-ReST-code-regexp ":: \\(.+\\)\\)$")
  "Regular expression to retrieve the mode of the code block.")

(pm-create-indented-block-matchers "ReST" pm-ReST-head-regexp)

(defcustom pm-inner/ReST-code
  (pm-hbtchunkmode-auto "ReST"
                        :head-reg #'pm-ReST-head-matcher
                        :tail-reg #'pm-ReST-tail-matcher
                        :head-mode 'host
                        :tail-mode 'host
                        :retriever-regexp pm-ReST-retriever-regexp
                        :font-lock-narrow t)
  "Restructured Text inner code block mode."
  :group 'innermodes
  :type 'object)


(defcustom pm-poly/ReST
  (pm-polymode-multi-auto "ReST"
                          :hostmode 'pm-host/ReST
                          :auto-innermode 'pm-inner/ReST-code)
  "Restructured Text typical `polymode' configuration."
  :group 'polymodes
  :type 'object)


;;;###autoload (autoload #'poly-rest-mode "poly-rest-mode")
(define-polymode poly-rest-mode pm-poly/ReST)


(defun poly-rest-fixes ()
  "Fix various minor issues that can occur in the poly-ReST-mode."
  (remove-hook 'prog-mode-hook   #'whitespace-mode)
  (remove-hook 'python-mode-hook #'mmm-mode))

(provide 'poly-rest-mode)

;;; poly-rest-mode.el ends here

As far as I can tell, this solves the indentation problem. The remaining issue is that ReSToption blocks become part of the chunk-buffer rather than the host-buffer. I believe that can be solved with a multi-line regexp but then I think the #107 issue will appear. I'll keep looking into it.

[Xaldew]

Changing the pm-ReST-head-regexp to a multi-line regexp does seem to work:

(defvar pm-ReST-head-regexp
  (concat "^[^ ]*\\( *\.\. +" pm-ReST-code-regexp "::.*\n"
          "\\(?: +:.+\n\\)*\\)")      ;; Match 0 or more option lines.
  "Regular expression to match the `head' of a ReST code block.")

Editing the chunk-header is still a bit finicky however as it may jump between
host and chunk mode. I'm not sure I can fix that. I should note however that it
will correct itself after a while.

Here's an example of it in action:

@vspinu Is this something you'd like to distribute with polymode? Or should we simply close this issue?

[vspinu]

Awesome indeed!

@vspinu Is this something you'd like to distribute with polymode? Or should we simply close this issue?

I would like to split polymode core functionality out and create an organization with each polymode as a separate project. This change (among others) has been pending for a while as part of next branch. Let's keep this issue open. I will ping you once I have merged the next branch in and created a separate project for each mode.

[Xaldew]

@vspinu Alright! Just give me a good poke when you're ready :)

@njsmith Fancy taking the code for a spin before I update the Stack Exchange entry?

[spluque]

I'm writing documentation for a Python package using Sphinx, and it looks like Pweave, which uses noweb chunk syntax is the way to go. Before adapting some of the code that @Xaldew posted, I wanted to check if there's any further updates/suggestions on this front.

The following poly-rst+python-mode seems to provide proper font-locking and navigation, but I'm still lacking the weaving and export. Any help is appreciated @vspinu

(require 'polymode)

(defcustom pm-host/ReST
  (pm-bchunkmode "ReST"
                 :mode 'rst-mode)
  "Restructured Text host chunkmode"
  :group 'hostmodes
  :type 'object)

(defcustom pm-inner/noweb+python
  (clone pm-inner/noweb :mode 'python-mode)
  "Noweb for Python"
  :group 'innermodes
  :type 'object)

(defcustom pm-poly/rst+python
  (clone pm-poly/noweb
	 :hostmode 'pm-host/ReST
	 :innermode 'pm-inner/noweb+python)
  "Noweb for Python configuration"
  :group 'polymodes
  :type 'object)

(define-polymode poly-rst+python-mode pm-poly/rst+python
  :lighter " PM-Pnw")


(provide 'poly-rst)

[Xaldew]

Hello @spluque, I don't really have any updates to this code, but the full (and latest) version of it available here:
http://emacs.stackexchange.com/questions/27679/ergonomic-way-to-edit-source-code-inside-rest-or-markdown-code-blocks/27695#27695

As far as I can tell from the pweave format, each section begins with the noweb <<>>= header and ends with a @, so I don't think you'll have any issue with the code above in regard to navigation and font-locking at least.

[spluque]

Thanks @Xaldew, I just tried that, and it works well for the ReST code, but the noweb chunks are not properly identified.

[Xaldew]

Ah, I think I misunderstood your intentions @spluque. If I understand things correctly, you want to edit things in a rst-mode with both automatically detected code-chunks and noweb style chunks containing Python code. Does that sound about right?

I ran a number of tests, and to get the above working, you can re-use a lot of the code I wrote, but you must define the polymode with pm-polymode-multi-auto, similar to how I did with the pm-poly/ReST in the snippets above.

The full code I used is:

;;; poly-rest-pweave.el -- Polymode for pweave-rest. -*- lexical-binding: t -*-
;;
;;; Commentary:
;; A mode for interacting with multiple kinds of code inside a ReST host-mode
;; with additional `noweb' chunks containing `python' code.
;;
;;; Code:

(require 'polymode)
(require 'poly-noweb)
(require 'poly-rest-mode)


(defcustom pm-inner/noweb+python
  (clone pm-inner/noweb :mode 'python-mode)
  "Noweb for Python."
  :group 'innermodes
  :type 'object)

(defcustom pm-poly/rst+python
  (pm-polymode-multi-auto
   "Pweave"
   :hostmode 'pm-host/ReST

Note that this code assumes you can `require` the 
   :innermodes '(pm-inner/noweb+python)
   :auto-innermode 'pm-inner/ReST-code)
  "Polymode using pweave with ReST and Python."
  :group 'polymodes
  :type 'object)

(define-polymode poly-rst+python-mode pm-poly/rst+python
  :lighter " PM-Pnw")


(provide 'poly-rest-pweave)

;;; poly-rest-pweave.el ends here

[spluque]

Thanks @Xaldew, I think I've distilled it to this:

(require 'poly-noweb)

(defcustom pm-host/ReST
  (pm-bchunkmode "ReST"
                 :mode 'rst-mode)
  "Restructured Text host chunkmode"
  :group 'hostmodes
  :type 'object)

(defcustom pm-inner/noweb+python
  (clone pm-inner/noweb :mode 'python-mode)
  "Noweb for Python"
  :group 'innermodes
  :type 'object)

(defcustom pm-poly/rst+python
  (clone pm-poly/noweb
	 :hostmode 'pm-host/ReST
	 :innermode 'pm-inner/noweb+python)
  "Noweb for Python configuration"
  :group 'polymodes
  :type 'object)

(define-polymode poly-rst+python-mode pm-poly/rst+python
  :lighter " PM-Pnw")

;; Pweave
(defcustom pm-weaver/Pweave
  (pm-shell-weaver "pweave"
                   :from-to
                   '(("rst" "\\.rstw\\'" "rst"
		      "ReStructuredText" "pweave -o %o %i")))
  "Shell Pweave weaver."
  :group 'polymode-weave
  :type 'object)

(polymode-register-weaver pm-weaver/Pweave t
                          pm-poly/rst+python)


(provide 'poly-rst)

which got me up to weaving. todo: get it to export by running sphinx-build.

[Evidlo]

One year later. What's the status on getting this merged into polymode proper?

Also, @spluque, do you think you could add support for the .. math:: block (latex-mode) and possibly the :math: role?

[vspinu]

I has been a while but I have finally found time to rewrite polymode. There are quite some changes and and fixes. This old code won't work but I wonder if anyone here is still interested in rst mode.

Particularly @Xaldew, I would be more than happy to port your code to new system and place it in a dedicated repo inside polymode organization. Would you be interested to step-in as a co-maintainer for that mode?

[oscarvarto]

Hi @vspinu I am interested in rst mode!
I am trying to get syntax highlighting for block codes and I am using Spacemacs.

I am an Emacs newbie, so I would love to get some help!

[Evidlo]

I still want this.

[Xaldew]

Hello @vspinu, I would indeed be willing to step in as co-maintainer for that.

[vspinu]

Great! I will give it a first try based on your code over the weekend. How should the repo be called?
poly-rst or poly-python and dedicate it to multiple python polymodes? I am favoring the latter because it's easier to maintain all in one place.

Regarding the noweb discussion from above. Does ReST specification defines noweb chunks or it should be a stand-alone poly-noweb+python-mode derived from poly-noweb-mode?

[Xaldew]

Honestly, I don't have any strong opinions either way. If I were to choose, I'd
probably choose poly-python and let poly-rst be part of it, since ReST is most
prevalent in Python contexts.

Regarding noweb, I honestly don't know much about it and after skimming through the specification, I couldn't find any mention of noweb, so it's probably an unofficial extension, so I think that it makes sense to let it be stand-alone, at least for now.

[vspinu]

@Xaldew, one question regarding your rst snippet from above.

The :caption is not indented the same way as the code and it could go both ways. Sometime caption is indented more and sometimes less. Is this intended or just a typo?

I see a number of rst-indent-xyz variables in rst.el. Which shall I use for the indentation offset of the code? More specifically, to indent code with respect to the beginning of the header starting at ...

[vspinu]

The poly-rst is there. Works fine in my tests except for the indent-region which is not yet provisioned for in the polymode core. Would really appreciate if people could try it out.

I have decided for a separate poly-rst because the scope of ReST is much wider than python, as this link suggests. poly-python could add additional tooling and derived modes, but a stand-alone rst mode seems warranted.

[Xaldew]

@vspinu, I think the caption placement is mostly a typo on my part. It's been a while so I'm afraid I don't remember. The official ReST specification seems to suggest that the indentation should match the code. (See e.g., directives).

And yeah, after brief glance at that link I agree with your suggestion to separate poly-rst from Python tooling.