Support for SageTeX

[antimike]

Feature request ("what")

I'd like to suggest that support for the SageTeX package be "officially" added to vimtex.
This has been suggested before, but as the issue was closed because of apparent lack of interest, I'm opening a new one.

Disclaimer

Although I've been using vim a lot recently, I'm not an expert by any means, so it's possible that there's an obvious solution to the problems I list below which doesn't involve any extra features. In which case, I apologize.

Background ("why")

The Good

First of all, I'm a huge fan of vimtex---I use it nearly every day for note-taking, homework, and scratch work (I'm a grad student in physics).

I'm also a huge fan of using Sage for computations, and it is incredibly convenient to be able to embed simple calculations directly into the markup of a mathematical document via SageTeX. Recently I've become comfortable enough with Sage that almost all my LaTeX documents include some embedded Sage code---for instance, I routinely use Sage (which is really just Python with extra stuff) to construct tedious bits of markup like tables. (See below for an example of this kind of document structure.)

The Bad

Unfortunately, editing Sage code, e.g. embedded in a \begin{sagesilent} ... \end{sagesilent} environment, is not very functional at the moment. This is because

• Python is whitespace-delimited, which makes automatic application of TeX indentation rules very inconvenient;
• Python textobjects defined by other plugins aren't available in such a context;
• Python linting (via Jedi, e.g.) is either not working or working unpredictably;
• Various other filetype-based odds and ends (e.g., code-commenting keybindings from NerdCommenter) aren't working optimally, or at all. (Another one I use frequently is vimcmdline, which makes running quick tests extremely convenient. It would be nice to be able to use this plugin, or something similar, to send Sage / Python code to a REPL from within a TeX document.)

The Ugly

My current workaround is via the syntax-range plugin, and looks like this:

autocmd BufReadPost *.tex call SyntaxRange#Include('"""begin=sage"""', '"""end=sage"""', 'sage', 'NonText')
autocmd BufReadPost *.tex call SyntaxRange#Include('%%%begin=sage%%%', '%%%end=sage%%%', 'sage', 'NonText')

This works...somewhat OK, in that the local indentation rules seem to respect the Sage filetype. In other words, when I open a new line in a Sage codeblock, it's opened at a sensibly-indented position with respect to the previous line.

However, periodically---on writes, and when adding certain kinds of textobjects (single- and double-quotes, e.g.)---the line I'm editing will be auto-maliciously re-indented to match the line above it, which of course renders the resulting code invalid as Python. And of course, typing gg=G, which I'm in the habit of doing semi-regularly, just destroys the fragile Python indentation in any fenced blocks...so I periodically have to manually re-indent significant chunks of code, which gets annoying after awhile.

Examples

Below is a fairly minimal example showing my most common use-cases: the sagesilent environment and \sagestr{} command. It won't compile with SageTeX but should compile with LaTeX; if necessary I can modify it further so that it compiles with both.

\documentclass{article}
\usepackage{autoref}
\usepackage{sagetex}
\begin{document}

Random LaTeX stuff

  \begin{sagesilent}
"""begin=sage"""
"""Note the indentation: the SageTeX preprocessor doesn't like it when the embedded Python code is indented, even if it's indented consistently."""
import pickle
import mytexmodule as tex

with open('modes.p', 'rb') as file:
  data = pickle.load(file)

def random_function(arg):
  return 3

ans = random_function(data)
plt = plot([], (0, 7), axes_labels=[r"$\omega/\omega_\lambda$", r"$\beta_\lambda$"], ticks=[[0, 1, sqrt(3), *optima], None], tick_formatter=[[r"$0$", r"$1$", r"$\sqrt{3}$", *s.serialize(*optima, data=True)], None], detect_poles='show') # No actual data plotted
xi_eta_table = tex.Table(title=r"Values of $\xi$ and $\eta$ for the first ten cylindrical modes")
"""end=sage"""
  \end{sagesilent}

  As shown in~autoref{p1:b:tab:dimensionless-mode-constants}, the answer is \sagestr{tex.texify(ans)}.

\begin{table}[ht]
  \centering
  \sagestr{xi_eta_table.latex}
  \caption{Numerical values of the dimensionless constants $\displaystyle \xi$ and $\displaystyle \eta$ for the first ten cylindrical modes, as calculated by Sage.  The values in the last column are optimal relative frequencies, i.e., values of $\displaystyle \omega/\omega_\lambda$ such that the attenuation is minimal.}
  \label{p1:b:tab:dimensionless-mode-constants}
\end{table}

\begin{figure}[ht]
  \centering
  \sageplot[]{plt}
  \caption{Plot of the attenuation constant $\displaystyle \beta$ vs.\ the relative frequency $\displaystyle \omega/\omega_\lambda$ for the lowest two modes of the cylindrical waveguide.  Points of minimum attenuation are highlighted for each mode, as is the point of intersection.  Note that the ``intersection point'' is measured relative to the respecitve cutoff frequencies of both modes.}
  \label{p1:b:plot:lowest-two-modes}
\end{figure}

\end{document}

[user202729]

Workaround

Add this to .vim/after/syntax/tex.vim: (it has the disadvantage of changing the color of the environment name however, I haven't figured out how to undo that)

let s:TeX_indentexpr=&indentexpr  " backup the normal values
let s:TeX_indentkeys=&indentkeys

augroup TeXInlineBlocks
	auto!

	if !exists("s:Lua_indentkeys")
		unlet b:did_indent
		source $VIMRUNTIME/indent/lua.vim
		let s:Lua_indentkeys=&indentkeys
		let &indentexpr=s:TeX_indentexpr
	endif
	call SyntaxRange#Include('^\s*\\begin{luacode\*}', '\\end{luacode\*}', 'lua', 'Statement')
	call SyntaxRange#Include('^\s*\\directlua\s*{', '}', 'lua', 'Statement')
	call OnSyntaxChange#Install("LuaEnvironmentInTeX", "^synIncludeLua$", 1, "a")
	auto User SyntaxLuaEnvironmentInTeXEnterA set indentexpr=GetLuaIndent() | let &indentkeys=s:Lua_indentkeys
	auto User SyntaxLuaEnvironmentInTeXLeaveA let &indentexpr=s:TeX_indentexpr | let &indentkeys=s:TeX_indentkeys

	if !exists("s:Python_indentkeys")
		unlet b:did_indent
		source $VIMRUNTIME/indent/python.vim  " to define that function ^^ (assuming the internal indentation function is named GetPythonIndent)
		let s:Python_indentkeys=&indentkeys
		let &indentexpr=s:TeX_indentexpr  " revert the indent
	endif
	call SyntaxRange#Include('\\begin{python}', '\\end{python}', 'python', 'Statement')
	call SyntaxRange#Include('\v\\(py|pyc|pyq|pycq)\s*\{', '}', 'python', 'Statement')
	call OnSyntaxChange#Install("PyEnvironmentInTeX", "^synIncludePython$", 1, "a")
	auto User SyntaxPyEnvironmentInTeXEnterA set indentexpr=GetPythonIndent(v:lnum) | let &indentkeys=s:Python_indentkeys
	auto User SyntaxPyEnvironmentInTeXLeaveA let &indentexpr=s:TeX_indentexpr | let &indentkeys=s:TeX_indentkeys
augroup END

For this one indentation while editing is correct; however gg=G is still incorrect.

To partially make behavior of gg=G correct (it will leave the indentation in blocks verbatim), patch

vimtex/indent/tex.vim

Line 102 in 60bec44

let s:verbatim_envs = ['lstlisting', 'verbatim', 'minted', 'markdown']

to add the environment name. (or use some <SNR> trick to patch the variable content from vimrc)

Implementing the normal way (without changing indentexpr) is possible, but depends on the internal implementation details of Python indentation plugin -- see for example how html.vim implement indentation of inline JavaScript, or how vim.vim implement python << EOF.

[user202729]

Thinking about it, the user may use several packages which define new verbatim-like environment (VerbatimOut in fancyvrb, scontents in scontents, luacode* in luacode, python in pyluatex/PythonTeX, etc.), and I don't think it's possible to parse them, so perhaps making the list of verbatim environments vimrc-configurable is reasonable. @lervag

[lervag]

@antimike

I'd like to suggest that support for the SageTeX package be "officially" added to vimtex. This has been suggested before, but as the issue was closed because of apparent lack of interest, I'm opening a new one.

Thanks! Sorry for taking so long before addressing this, but life's been very busy lately!

Unfortunately, editing Sage code, e.g. embedded in a \begin{sagesilent} ... \end{sagesilent} environment, is not very functional at the moment. This is because

• Python is whitespace-delimited, which makes automatic application of TeX indentation rules very inconvenient;

This really is quite a hard problem, and I'm not sure I'm interested in addressing it myself. I may consider pull requests.

• Python textobjects defined by other plugins aren't available in such a context;

Same as above.

• Python linting (via Jedi, e.g.) is either not working or working unpredictably;

Same as above.

• Various other filetype-based odds and ends (e.g., code-commenting keybindings from NerdCommenter) aren't working optimally, or at all. (Another one I use frequently is vimcmdline, which makes running quick tests extremely convenient. It would be nice to be able to use this plugin, or something similar, to send Sage / Python code to a REPL from within a TeX document.)

Same.

The thing that makes all of the above hard is that Vim is not really designed to support multiple file types in a single file. For Neovim, there is Tree-sitter, which I believe in the future will actually allow relatively simple solutions to perhaps all of the above. But it is still not so mature, and Tree-sitter syntax highlighting is currently not as good as the syntax highlighting provided by VimTeX. That may change, perhaps even relatively soon.

However, I do not currently plan to build VimTeX around Tree-sitter. So features in this direction will probably be built by other enthusiasts. Sorry.

My current workaround is via the syntax-range plugin, and looks like this: ...

I think such workarounds are interesting, and it may be relevant to document them in the VimTeX docs.

Below is a fairly minimal example showing my most common use-cases: the sagesilent environment and \sagestr{} command. It won't compile with SageTeX but should compile with LaTeX; if necessary I can modify it further so that it compiles with both.

I'm not really "acquainted" with sagetex, so whether you should provide the more advanced example is more or less up to you. But if we agree on desired changes to VimTeX, then I might ask for specific examples. In any case, thanks for providing the example.

@user202729

Thanks for providing another idea for a workaround.

Thinking about it, the user may use several packages which define new verbatim-like environment (VerbatimOut in fancyvrb, scontents in scontents, luacode* in luacode, python in pyluatex/PythonTeX, etc.), and I don't think it's possible to parse them, so perhaps making the list of verbatim environments vimrc-configurable is reasonable. @lervag

Well, yes and no. I believe it is better to add specific support to various packages, as that allows us to handle things in a better manner. Let's focus on SageTeX now. If you think the above still holds after you see what I mean, then feel free to open a new issue and repeat the suggestions.

[lervag]

So, the question is then: what do we mean with support for SageTeX? My proposal is to add support for the various SageTeX environments and comments within which we should assume nested Python syntax. This would mean that e.g. within a sagesilent environment, VimTeX would provide standard Python syntax highlighting with the syntax group texSagetexZone. I'll do this immediately with sagesilent, but you will need to help me with references and examples to address other relevant commands and environments.

[lervag]

I've now added initial syntax support to sagesilent, sagestr and sageplot. The groups texSagetexZone, texSagetexOpt and texSagetexArg have been introduced here.