Improper syntax highlight for dollar sign $

[milanglacier]

Describe the bug

The highlight for the dollar sign $ is improper

Steps to reproduce

1. open the following example org file test.org

#+title: Test

example 1: the verbatim should be respected
=$= the text should not be highlighted =$=

example 2: text surrounded by dolloar signs that are distanced by over 2 lines should not be highlighted
$2 billion
hello
oh not good
$3 billion

Expected behavior

1. when $ is closed by two =, i.e =$= then verbatim should be respected. i.e there should be no syntax highlight
2. when two $ are distanced by more two lines, text between them should not be highlighted.

Emacs functionality

see description here
https://orgmode.org/manual/LaTeX-fragments.html

Text within the usual LaTeX math delimiters. To avoid conflicts with currency specifications, single ‘$’ characters are only recognized as math delimiters if the enclosed text contains at most two line breaks, is directly attached to the ‘$’ characters with no whitespace in between, and if the closing ‘$’ is followed by whitespace, punctuation or a dash. For the other delimiters, there is no such restriction, so when in doubt, use ‘(...)’ as inline math delimiters.

Minimal init.lua

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvim/site]]

local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'

local function load_plugins()
    require('packer').startup {
        {
            'wbthomason/packer.nvim',
            { 'nvim-treesitter/nvim-treesitter' },
            { 'kristijanhusak/orgmode.nvim', branch = 'master' },
        },
        config = {
            package_root = package_root,
            compile_path = install_path .. '/plugin/packer_compiled.lua',
        },
    }
end

_G.load_config = function()
    require('orgmode').setup_ts_grammar()
    require('nvim-treesitter.configs').setup {
        ensure_installed = { 'latex' },
        highlight = {
            enable = true,
            additional_vim_regex_highlighting = { 'org' },
        },
    }

    vim.cmd [[packadd nvim-treesitter]]
    vim.cmd [[runtime plugin/nvim-treesitter.lua]]
    vim.cmd [[TSUpdateSync org]]

    -- Close packer after install
    if vim.bo.filetype == 'packer' then
        vim.api.nvim_win_close(0, true)
    end

    require('orgmode').setup {
        org_highlight_latex_and_related = 'native',
    }

    -- Reload current file if it's org file to reload tree-sitter
    if vim.bo.filetype == 'org' then
        vim.cmd [[edit!]]
    end
end

if vim.fn.isdirectory(install_path) == 0 then
    vim.fn.system { 'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path }
    load_plugins()
    require('packer').sync()
    vim.cmd [[autocmd User PackerCompileDone ++once lua load_config()]]
else
    load_plugins()
    load_config()
end

Screenshots and recordings

the first is neovim

the second is emacs

OS / Distro

macOS 12.6

Neovim version/commit

0.8.1

Additional context

No response

[jgollenz]

I cannot reproduce this. The highlighting for me looks as you in the Emacs screenshot you posted. @kristijanhusak is a latex parser needed here?

[milanglacier]

I cannot reproduce this. The highlighting for me looks as you in the Emacs screenshot you posted. @kristijanhusak is a latex parser needed here?

Sorry, I missed the option org_highlight_latex_and_related = 'native', and ensure_installed = 'latex'

I have updated the minimal config to make it reproducible.

Thanks for reminding me!

[kristijanhusak]

When you set org_highlight_latex_and_related = 'native' it actually loads the whole syntax file from vim's tex filetype. Support for latex is generally lacking and not stable. Maybe you could disable the native and just rely on what's there with treesitter/markup highlighter. Recently some highlights were added (#428), but not all of them.

[milanglacier]

When you set org_highlight_latex_and_related = 'native' it actually loads the whole syntax file from vim's tex filetype. Support for latex is generally lacking and not stable. Maybe you could disable the native and just rely on what's there with treesitter/markup highlighter. Recently some highlights were added (#428), but not all of them.

Thanks for your advice. I decide to go with org_highlight_latex_and_related = 'entity' right by now

[seflue]

Still reproduces on tree-sitter-org 2.0.4. Root cause is upstream in the grammar, not this repo.

Here is what I found out with the help of Claude:

There is no grammar node for verbatim or code: =...= and ~...~ are recognised only as highlight patterns in queries/org/markup.scm ((expr "=" @verbatim)). To the parser, = is just a symbol on the generic expr node, so the math scanner has no concept of "inside verbatim".

Inline math $...$ is decided by the external scanner (src/scanner.c#L334-L380, tokens LATEX_MATH_SINGLE_DOLLAR / TEXT_DOLLAR). Two heuristics decide whether a $ opens math:

1. The previous character of an opening $ is never inspected (scanner.c#L346); only $$ is rejected as a special case (L348-L350, delegated to the grammar).
2. The candidate closing $ is accepted unless the next char is [0-9A-Za-z] (scanner.c#L356-L358). Anything else, including {, =, ~, punctuation, opens math.

So any two $ on the same line with a non-alphanumeric char after the second one open a math block, and verbatim spans in between get swallowed. The example in this issue (=$= text =$=) hits exactly that.

Suggested fix

Promote verbatim and code to first-class grammar nodes so their contents are not parsed as expr. The inline_math_block work by @NickHu (tree-sitter-org#5, tree-sitter-org#6) is a direct precedent for the same pattern; no scanner change needed.

This matches what Emacs does. In org-element.el, verbatim and code are leaf objects (in org-element-all-objects, not in org-element-object-restrictions); their :value is never re-lexed for objects like latex-fragment. So the $-rules in the Org manual only apply once the parser has confirmed the position is not inside verbatim/code. Structural protection, not a regex heuristic.

Downstream impact looks moderate: queries/org/markup.scm patterns for = and ~ rewrite to match the new nodes; emphasis.lua and init.lua in colors/highlighter/markup/ need adapting, with the pairing logic getting simpler once =/~ are atomic nodes; extmark-based tests should be refactor-tolerant.

A scanner-only heuristic (tighter pre/post-char check on $ in scanner.c) would close the common cases without a grammar change, but it cannot reproduce the structural guarantee Emacs gives. More of a hack than a real fix. Would be a fallback option if we want to avoid touching the grammar.

@kristijanhusak: does this direction look right? Happy to open the two PRs (grammar + orgmode), or leave them to you.

[kristijanhusak]

@seflue do you already have something in place or you plan to? I have this commit on another branch in grammar that adds bunch of things, including the markup. Created it with AI though, so a bit skeptic around it.

[seflue]

@kristijanhusak I didn't start yet, but I have to admit, that I was able to produce pretty good results regarding treesitter grammars with Opus 4.7. So I'm quite confident (with a bit of back and force with the model and a solid test corpus) to produce a solid solution.
What I currently have are the analysis results of the problem, written down in a big org-file (which is typically a good starting point for unleashing the agents to implement something useful).
What I can offer is to do some analysis of your commit to raise your confidence. Or implement something on my own and compare it.

[kristijanhusak]

I did my with copilot and gpt 5.4. Opus should be better.
If you want, check the whole branch https://github.com/nvim-orgmode/tree-sitter-org/tree/feat/markup. It has inline latex (\thing), footnotes (already on next branch), and markup.
You can try and create your own, just make sure that everything is covered with tests (AI should take care of that). After that you can compare and we will see. WDYT?

[seflue]

I never let the AI implement anything without writing the tests first. 😉 Costs more tokens in the first place, but strict TDD keeps the agent in check, so it doesn't slop away too much.
I will put something together and let you know.