help syntax don't support help tags with unicode characters

[mildred]

Here is defined the syntax highlighting pattern for help tags:

vim/runtime/syntax/help.vim

Lines 27 to 29 in 3ec574f

	syn match helpHyperTextJump "\\\@<!|[#-)!+-~]\+|" contains=helpBar
	syn match helpHyperTextEntry "\*[#-)!+-~]\+\*\s"he=e-1 contains=helpStar
	syn match helpHyperTextEntry "\*[#-)!+-~]\+\*$" contains=helpStar

This pattern doesn't match utf-8 unicode characters. More specifically, the [...+-~] matches the pure ASCII alphabet but not any alphabetic character beyond that. Specifying negative character class using [^ "*\t] instead of [#-)!+-~] would match all unicode charactes (and control characters, but those are not supposed to be present in text files except for tabs)

A ASCII table would show that out of the #-) range, and out of +-~ with the ! character excluded there are only control characters below 32, space , double quote", and the star *

Note that :helptags already supports unicode tags, it's just syntax hifhlighting that is lagging behind. I currently use this in my .vimrc:

augroup vimrc_filetype_help
  au FileType help syn match helpHyperTextJump	"\\\@<!|[^ \þ!*]\+|" contains=helpBar
  au FileType help syn match helpHyperTextEntry	"\*[^ \t!*]\+\*\s"he=e-1 contains=helpStar
  au FileType help syn match helpHyperTextEntry	"\*[^ \t!*]\+\*$" contains=helpStar
augroup END

Also reported with neovim in neovim/neovim#7402

[k-takata]

Is \þ in helpHyperTextJump a mistake of \t?

[mildred]

yes, it is, sorry

augroup vimrc_filetype_help
  au FileType help syn match helpHyperTextJump	"\\\@<!|[^ \t!*]\+|" contains=helpBar
  au FileType help syn match helpHyperTextEntry	"\*[^ \t!*]\+\*\s"he=e-1 contains=helpStar
  au FileType help syn match helpHyperTextEntry	"\*[^ \t!*]\+\*$" contains=helpStar
augroup END

[brammool]

I prefer to keep the help tags ASCII, so that they still work when 'encoding' is some code page. Some people still use that. And there is not much point in non-ASCII characters in help tags.

[k-takata]

But isn't it useful for translated help?
At near the end of :help help-translated:

Hints for translators:
- Do not translate the tags.  This makes it possible to use 'helplang' to
  specify the preferred language. You may add new tags in your language.

[vim-ml]

Most helpfiles are in ASCII anyway (which is compatible both with UTF-8 multibyte and with ISO-8859 8bit). A few, like arabic.txt, are in UTF-8 but won't be needed when compiled with -multi_byte. options.txt is in UTF-8 because of the example for 'langmap' which includes Greek letters, and is clearly marked as being in UTF-8. (By default, +langmap is compiled into Big and Huge builds only, while +multi_byte is already in Normal builds of Vim.) For translated help, the _helpfiles_ may be in UTF-8 for non-ASCII characters in the languages concerned, but IMHO the _tags_ themselves ought to remain identical with those in the English-language help, i.e., untranslated and in ASCII, so they will be found regardless of which help language is in use, and even if one translated helpfile is missing (in which case Vim falls back on the English help). Words in the non-English 'helplang' language can be found by means of :helpgrep Best regards, Tony.

[chrisbra]

closing then.