nvim_get_hl_by_name("Group", true) doesn't always return colors

[adrian5]

Neovim version (nvim -v)

NVIM v0.7.0-dev+1392-gdaa8ac051

Installation

GitHub nightly builds

How to reproduce the issue

nvim --clean

1. :lua =vim.api.nvim_get_hl_by_name("Normal", true) prints:

{                                                                                                             
  [true] = 6
}

2. Set highlight via lua vim.api.nvim_set_hl(0, "Normal", { fg = "#ff0000", bg = "#202020" }). 1) prints:

{                                                                                                             
  [true] = 6
}

3. Set highlight via :hi Normal guifg=#ff0000 guibg=#202020. 1) prints:

{
  background = 2105376,
  foreground = 16711680
}

Note: This behavior might be specific to the Normal group.

Expected behavior

nvim_get_hl_by_name("Normal", true) returns the group's colors.

Actual behavior

It only does so when the highlights are set the vimscript way.

[dundargoc]

Using nvim --clean -S minimal.vim, where minimal.vim is:

:lua vim.api.nvim_get_hl_by_name("Normal", true)

doesn't print anything for me. Am I missing something?

[adrian5]

My bad, that should've been :lua =vim.api.nvim_get_hl_by_name("Normal", true) to print the table.

Here as a file:

:echo "Clean:"
:lua =vim.api.nvim_get_hl_by_name("Normal", true)

:echo "\nnvim_set_hl(Normal):"
:lua vim.api.nvim_set_hl(0, "Normal", { fg = "#ff0000", bg = "#202020"})
:lua =vim.api.nvim_get_hl_by_name("Normal", true)

:echo "\nhighlight Normal:"
:hi Normal guifg=#ff0000 guibg=#202020
:lua =vim.api.nvim_get_hl_by_name("Normal", true)

[rktjmp]

I think it's correct to say this is specific to Normal.

Was checking due to above linked issue, so the check involves a Lush theme, to get a list of valid group names but that's the only Lush involvement

local rb = require("rosebones")
local api_get = vim.api.nvim_get_hl_by_name
local c = 0
for name, _ in pairs(rb) do
  c = (c + 1)
  local val = api_get(name, true)
  if val[true] then
    print(name, "had 'true' key: ", vim.inspect(val))
  else
  end
end
return print("checked", c, "groups")

Normal had 'true' key:  {
  [true] = 6
}
checked 205 groups

[rktjmp]

sg_attr is zero for Normal which results in an empty dict being returned, non-zero for other groups.

// vim.c
Dictionary nvim_get_hl_by_id(Integer hl_id, Boolean rgb, Error *err)
  FUNC_API_SINCE(3)
{
  //// listing 1.1
  //// at this point hl_id is some +int, valid.
  Dictionary dic = ARRAY_DICT_INIT;
  if (syn_get_final_id((int)hl_id) == 0) {
    api_set_error(err, kErrorTypeException,
                  "Invalid highlight id: %" PRId64, hl_id);
    return dic;
  }

  //// kicks down to syn_id2attr, see listing 2.1
  //// listing 1.2
  int attrcode = syn_id2attr((int)hl_id);
  //// attrcode will be zero for Normal (because sg_attr was zero)
  //// see listing 3.1
  return hl_get_attr_by_id(attrcode, rgb, err);
}

// highlight_group.c
int syn_id2attr(int hl_id)
{
  //// listing 2.1
  hl_id = syn_get_final_id(hl_id);
  HlGroup *sgp = &HL_TABLE()[hl_id - 1];  // index is ID minus one

  //// sgp is mostly good, contains good data but for one field:
  //// (gdb) print *sgp                                                                          │
  //// $19 = {sg_name = 0x55eb36ab7350 "Normal", sg_name_u = 0x55eb36ab7370 "NORMAL",            │
  ////   sg_cleared = false, sg_attr = 0, sg_link = 0, sg_deflink = 0, sg_set = 0,               │
  ////   sg_deflink_sctx = {sc_sid = 0, sc_seq = 0, sc_lnum = 0}, sg_script_ctx = {              │
  ////     sc_sid = 115, sc_seq = 119, sc_lnum = 8}, sg_cterm = 0, sg_cterm_fg = 0,              │
  ////   sg_cterm_bg = 0, sg_cterm_bold = false, sg_gui = 0, sg_rgb_fg = 14800084,               │
  ////   sg_rgb_bg = 1710117, sg_rgb_sp = -1, sg_rgb_fg_name = 0x55eb36ae3b90 "#E1D4D4",         │
  ////   sg_rgb_bg_name = 0x55eb36adf570 "#1A1825", sg_rgb_sp_name = 0x0, sg_blend = -1}

  //// attr will be -1, so we dont return here
  int attr = ns_get_hl(-1, hl_id, false, sgp->sg_set);
  if (attr >= 0) {
    return attr;
  }

  /// here we return 0 back to listing 1.2
  return sgp->sg_attr;
}

// highlight.c
Dictionary hl_get_attr_by_id(Integer attr_id, Boolean rgb, Error *err)
{
  //// listing 3.1
  //// attr_id is 0
  Dictionary dic = ARRAY_DICT_INIT;

  if (attr_id == 0) {
    //// returns what looks like {[true] = 6} to lua
    return dic;
  }

For reference here is a working sgp value:

/// $23 = {sg_name = 0x55eb36ae1370 "Constant", sg_name_u = 0x55eb36ae1390 "CONSTANT",
///   sg_cleared = false, sg_attr = 218, sg_link = 0, sg_deflink = 0, sg_set = 0,
///   sg_deflink_sctx = {sc_sid = 0, sc_seq = 0, sc_lnum = 0}, sg_script_ctx = {
///     sc_sid = 115, sc_seq = 119, sc_lnum = 8}, sg_cterm = 4, sg_cterm_fg = 0,
///   sg_cterm_bg = 0, sg_cterm_bold = false, sg_gui = 4, sg_rgb_fg = 12358803,
///   sg_rgb_bg = -1, sg_rgb_sp = -1, sg_rgb_fg_name = 0x55eb36e39bd0 "#BC9493",
///   sg_rgb_bg_name = 0x0, sg_rgb_sp_name = 0x0, sg_blend = -1}

And the definition of sg_attr says @see ATTR_ENTRY but this is the only reference to ATTR_ENTRY in the codebase.

typedef struct {
  char_u *sg_name;         ///< highlight group name
  char *sg_name_u;       ///< uppercase of sg_name
  bool sg_cleared;              ///< "hi clear" was used
  int sg_attr;                  ///< Screen attr @see ATTR_ENTRY
  int sg_link;                  ///< link to this highlight group ID
  int sg_deflink;               ///< default link; restored in highlight_clear()

I am assuming this is the bug:

// highlight_group.c
void set_hl_group(int id, HlAttrs attrs, Dict(highlight) *dict, int link_id)
/// snip

  if (STRCMP(g->sg_name_u, "NORMAL") == 0) {
    cterm_normal_fg_color = g->sg_cterm_fg;
    cterm_normal_bg_color = g->sg_cterm_bg;
    normal_fg = g->sg_rgb_fg;
    normal_bg = g->sg_rgb_bg;
    normal_sp = g->sg_rgb_sp;
    ui_default_colors_set();
    //// sg_attr never set!
  } else {
    g->sg_attr = hl_get_syn_attr(0, id, attrs);

Setting sg_attr via the same means certainly "fixes" the listed issue, but I cant speak to the repercussions, whether it should be if; then vs if&else.

[Iron-E]

Seems like this issue still happens when running nvim_get_hl_by_name('Normal', false):

nvim --clean -c 'lua vim.pretty_print(vim.api.nvim_get_hl_by_name("Normal", false))'

It outputs {[true] = 6} just like before 9aba204.

[akinsho]

This behaviour is the present for the MsgArea and CurSearch highlight groups

[rktjmp]

Do your groups have any data besides colors on those groups?

nvim_get_hl_by_name({name}, {rgb})                     *nvim_get_hl_by_name()*
                Gets a highlight definition by name.

                Parameters: ~
                    {name}  Highlight group name
                 => {rgb}   Export RGB colors <=

                Return: ~
                    Highlight definition map

                See also: ~
                    nvim_get_hl_by_id

-- edit: sorry, code pulled from the test file but should be pretty understandable

-- set group with colours and gui attrs
set_hl(0, 'Normal', {fg='#000083', bg='#0000F3', bold = true})
-- request group data *without* RGB color export, returns gui attrs
nvim("get_hl_by_name", 'Normal', false)) -- => {bold = true}

-- set group with colours and NO gui attrs
set_hl(0, 'Normal', {fg='#000083', bg='#0000F3'})
-- request group data *without* RGB color export, returns "empty" dict
nvim("get_hl_by_name", 'Normal', false)) -- => {[true] = 6}

Empty dict value looks "not empty" once it gets back to nvim, Dictionary dic = ARRAY_DICT_INIT; translates out to {[true] = 6}, perhaps this value can be fiddled before its finally sent back to the user or lua state? To be explicit, this would effect much more than just this function.

The normal group has no colours by default:

nvim --clean -c 'lua vim.pretty_print(vim.api.nvim_get_hl_by_name("Normal", false))'

:hi Normal
Normal         xxx cleared

If you try something else you will get a the cterm values, which is why this looks ok even without gui attrs.

nvim --clean -c 'lua vim.pretty_print(vim.api.nvim_get_hl_by_name("Comment", false))'

{
  foreground = 14
}

[Iron-E]

I expect it to not return anything if Normal has no cterm definition, like in this example:

-- set group with colours and gui attrs
set_hl(0, 'Normal', {fg='#000083', bg='#0000F3', bold = true})
-- request group data without color export, returns gui attrs
nvim("get_hl_by_name", 'Normal', false)) -- => {bold = true}

-- set group with colours and NO gui attrs
set_hl(0, 'Normal', {fg='#000083', bg='#0000F3'})
-- request group data without color export, returns "empty" dict
nvim("get_hl_by_name", 'Normal', false)) -- => {[true] = 6}

However when using e.g. the builtin "shine" colorscheme with notermguicolors, Normal does have a definition for cterm:

:hi Normal
Normal         xxx ctermfg=0 ctermbg=15 guifg=Black
                   guibg=White

However running nvim_get_hl_by_name('Normal', false) still just returns {[true] = 6}.

Edit: So I guess the true MVCE is this

nvim --clean -c 'colorscheme shine' -c 'lua vim.pretty_print(vim.api.nvim_get_hl_by_name("Normal", false))'

[rktjmp]

A small test gives me this:

-- cterm value tests

-- Set Normal via lua api, get value
vim.api.nvim_set_hl(0, "Normal", {ctermfg = 16, ctermbg = 231})
vim.pretty_print("nvim_set_hl normal", vim.api.nvim_get_hl_by_name("Normal", false))
-- FAIL => {true = 6}

-- Normal group has some special handling sometimes, so check another group
-- but set the values = Normal
vim.api.nvim_set_hl(0, "NotNormal", {ctermfg = 16, ctermbg = 231})
vim.pretty_print("nvim_set_hl not-normal = normal", vim.api.nvim_get_hl_by_name("NotNormal", false))
-- FAIL => {true = 6}

-- Check another group but set values to NOT normal
vim.api.nvim_set_hl(0, "NotNormal", {ctermfg = 61, ctermbg = 132})
vim.pretty_print("nvim_set_hl not-normal", vim.api.nvim_get_hl_by_name("NotNormal", false))
-- OK => {background = 132, foreground = 61}

-- set via classic method
vim.cmd("hi Normal ctermfg=16 ctermbg=231")
vim.pretty_print("hi normal", vim.api.nvim_get_hl_by_name("Normal", false))
-- FAIL => {true = 6}
--
vim.cmd("hi NotNormal ctermfg=16 ctermbg=231")
vim.pretty_print("hi not-normal = normal", vim.api.nvim_get_hl_by_name("NotNormal", false))
-- FAIL => {true = 6}

vim.cmd("hi NotNormal ctermfg=17 ctermbg=18")
vim.pretty_print("hi not-normal", vim.api.nvim_get_hl_by_name("NotNormal", false))
-- OK => {background = 18, foreground = 17}

vim.pretty_print("default comment", vim.api.nvim_get_hl_by_name("Comment", false))
-- OK => {foreground = 14}

I think I have a fix for the Normal group, which propagate the issue to other groups if they share the same highlight data.

[nyngwang]

I just experienced that same thing. I don't think this is fixed.