treesitter overrides not working after nvim breaking changes

[petobens]

Consider the following minimal.lua file:

local fn = vim.fn

-- Ignore default config and plugins and define new dirs
local test_dir = '/tmp/nvim-minimal'
vim.opt.runtimepath:remove(fn.expand('~/.config/nvim'))
vim.opt.packpath:remove(fn.expand('~/.local/share/nvim/site'))
vim.opt.runtimepath:append(fn.expand(test_dir))
vim.opt.packpath:append(fn.expand(test_dir))

-- Install packer
local install_path = test_dir .. '/pack/packer/start/packer.nvim'
if fn.empty(fn.glob(install_path)) > 0 then
    packer_bootstrap = fn.system({
        'git',
        'clone',
        '--depth',
        '1',
        'https://github.com/wbthomason/packer.nvim',
        install_path,
    })
    vim.cmd([[packadd packer.nvim]])
end

-- Setup packer
local packer = require('packer')
packer.init({
    package_root = test_dir .. '/pack',
    compile_path = test_dir .. '/plugin/packer_compiled.lua',
})
packer.startup(function(use)
    -- Packer can manage itself
    use('wbthomason/packer.nvim')

    -- Plugins
    use({
        'nvim-treesitter/nvim-treesitter',
        run = ':TSUpdate',
        requires = { 'nvim-treesitter/playground' },
    })
    use({ 'olimorris/onedarkpro.nvim', branch = 'develop' })

    -- Auto install plugins
    if packer_bootstrap then
        packer.sync()
    end
end)

-- Plugin setup
local ok, treesitter = pcall(require, 'nvim-treesitter.configs')
if ok then
    treesitter.setup({
        highlight = {
            enable = true,
        },
        ensure_installed = { 'bash', 'markdown' },
    })
end

local ok, onedarkpro = pcall(require, 'onedarkpro')
if ok then
    local p = {
        fg = '#abb2bf',
        dark_red = '#be5046',
    }
    onedarkpro.setup({
        theme = 'onedark',
        colors = p,
        ft_highlights = {
            markdown = {
                markdownTSPunctSpecial = { fg = p.dark_red, style = 'bold' },
            },
        },
        ft_highlights_force = true,
    })
    onedarkpro.load()
end

Now, as in the GIF, do

1. Open a markdown file such as ~/Desktop/foo.md
2. Note that # symbol is not highlighted in red
3. Run TSHighlightCapturesUnderCursor (the reported highlight group is no longer markdownTSPunctuationSpecial)

I guess that after neovim/neovim#19931 there are some breaking changes to treesitter highlights (as stated in neovim/neovim#14090 (comment))

Also relevant: nvim-treesitter/playground#90

[olimorris]

So I've just tested this with the following:

# Ruby file
def this_is_a_test
  message = "Testing Treesitter"
  puts message
end

and with TSHighlightCapturesUnderCursor:

and when I edit my onedarkpro config to:

highlights = {
  TSVariable = { bg = "${green}" },
  variable = { bg = "${red}"},
  rubyvariable = { bg = "${purple}"},
}

I see:

Which suggests that TSVariable is still working.

This is the challenge with nightly, there's always so many moving parts and things break all the time. Whilst I can't see the same as you it may be the difference in when I update nightly.

[petobens]

Can you try reproducing the following? Start with this minimal.lua file:

local fn = vim.fn

-- Ignore default config and plugins and define new dirs
local test_dir = '/tmp/nvim-minimal'
vim.opt.runtimepath:remove(fn.expand('~/.config/nvim'))
vim.opt.packpath:remove(fn.expand('~/.local/share/nvim/site'))
vim.opt.runtimepath:append(fn.expand(test_dir))
vim.opt.packpath:append(fn.expand(test_dir))

-- Install packer
local install_path = test_dir .. '/pack/packer/start/packer.nvim'
if fn.empty(fn.glob(install_path)) > 0 then
    packer_bootstrap = fn.system({
        'git',
        'clone',
        '--depth',
        '1',
        'https://github.com/wbthomason/packer.nvim',
        install_path,
    })
    vim.cmd([[packadd packer.nvim]])
end

-- Setup packer
local packer = require('packer')
packer.init({
    package_root = test_dir .. '/pack',
    compile_path = test_dir .. '/plugin/packer_compiled.lua',
})
packer.startup(function(use)
    -- Packer can manage itself
    use('wbthomason/packer.nvim')

    -- Plugins
    use({
        'nvim-treesitter/nvim-treesitter',
        run = ':TSUpdate',
        requires = { 'nvim-treesitter/playground' },
    })
    use({ 'olimorris/onedarkpro.nvim' })

    -- Auto install plugins
    if packer_bootstrap then
        packer.sync()
    end
end)

-- Plugin setup
local ok, treesitter = pcall(require, 'nvim-treesitter.configs')
if ok then
    treesitter.setup({
        highlight = {
            enable = true,
        },
        ensure_installed = { 'python' },
    })
end

local ok, onedarkpro = pcall(require, 'onedarkpro')
if ok then
    local p = {
        dark_red = '#be5046',
    }
    onedarkpro.setup({
        theme = 'onedark',
        colors = p,
        ft_highlights = {
            python = {
                pythonTSConstant = { fg = p.dark_red }, -- Doesn't work (used to)
                -- TSConstant = { fg = p.dark_red }, -- works but is not filetype specific
            },
        },
    })
    onedarkpro.load()
end

Then, as in the GIF,

1. Open a python file (such as foo.py) with the following content:

A_CONSTANT = 'foo'

2. See that the pythonTSConstant override to dark_red is not applied
3. Comment that line in the minimal.lua file and uncomment the TS_Constant line
4. Reopen the foo.py now the ft_highlight is correctly applied (but is not filetype specific).

So my point is that under neovim master I believe the <ft><tree_sitter> highlight groups (such as pythonTSConstant) no longer work.

[olimorris]

I think we're aligned on this. As shown with the rubyVariable example above; it doesn't work. But as there are no <ft><treesitter> highlights in the colorscheme by default then there shouldn't be any issue caused by this plugin.

Do you not just need to redefine how you're applying your own custom highlights as a result of the changes in Neovim?

Reference: Treesitter highlight groups

Edit: And apologies if I am grossly misunderstanding this problem

[petobens]

Edit: And apologies if I am grossly misunderstanding this problem

No need for an apology and thanks for the patience.

Do you not just need to redefine how you're applying your own custom highlights as a result of the changes in Neovim?

That's a fair question (I don't have a concrete anwser). From what I can surmise of this neovim/neovim#14090 (comment) I believe onedarkpro is lacking two things:

1. If I define a pythonconstant highlight group that should take precedence over TSConstant (currently using pythonconstant does nothing)
2. I have no idea now how to override something like a markdownpunctuation.special highlight group which I could easily override with markdownTSPunctSpecial before (maybe the answer is simply to set TSPunctSpecial).

[petobens]

One extra comment. If I run something like:

:lua vim.api.nvim_set_hl(0, "@constant.python", {fg="Green"})

Then I can effectively access the pythonconstant highlight group. So I guess my question is how to do this from within Onedark config

[olimorris]

One extra comment. If I run something like:

:lua vim.api.nvim_set_hl(0, "@constant.python", {fg="Green"})

Then I can effectively access the pythonconstant highlight group. So I guess my question is how to do this from within Onedark config

Okay very interesting. And I guess you can't set that in the highlights config?

I will definitely look into this as it's a potentially awesome way to replace filetype highlights.

[petobens]

And I guess you can't set that in the highlights config?

Exactly. For instance while something like the following works:

lua vim.api.nvim_set_hl(0, "@punctuation.special.markdown", {fg="Green"})

it cannot be added to the higlights config (since we have @ and .).

I will definitely look into this as it's a potentially awesome way to replace filetype highlights.

Awesome! Thanks once again and sorry for all the back and forth.

[petobens]

I will definitely look into this as it's a potentially awesome way to replace filetype highlights.

Indeed something like petobens/dotfiles@e460ac8 fixes both this issue and #67

[olimorris]

@petobens - I've just tried:

highlights = {
      ["@variable.ruby"] = { bg = "${purple}" },
}

and I get:

[petobens]

Oh neat. I guess that's the way to go. Feel free to close this issue and the other #67 if you deem fit.

[olimorris]

I'll close this but leave #67 for now. After you've discovered this then I'm not sure what will happen with filetype highlights as this is a game changer.