Refine ctags with language specific kinds

[liuchengxu]

Fixes #4

This PR introduces more detailed kinds for various language(thanks to tagbar) and refactor the ctags parser to work with new options. However, these extra information has not been utilized nicely, which will be improved later.

[Avi-D-coder]

@liuchengxu
vimrc

let g:vista_ctags_cmd = {
      \ 'haskell': 'hasktags -o - -c',
      \ }

Extended ctags with hasktags -o - -x also produces errors.

:Vista ctags

Error detected while processing function <SNR>147_on_exit[4]..<SNR>147
_ExtractLinewise:
line    3:
E121: Undefined variable: s:TagParser

!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_PROGRAM_NAME      hasktags
Lib     src/Lib.hs      /^module Lib$/;"        m       line:1  language:Haskell
someFunc        src/Lib.hs      /^someFunc :: IO ()$/;" ft      signature:(IO ())       line:5  language:Haskell

Extended also produces hasktags: <stdout>: hPutStr: illegal operation (handle is closed) on stderr. So vista should only read stdout.

Lib     src/Lib.hs      1
someFunc        src/Lib.hs      5

[liuchengxu]

First of all, I recommend you compile the ctags with --output-format=json support, which is more reliable than parsing by vista.vim itself. But yes, vista.vim should fix these errors. I'll see what's wrong later.

[liuchengxu]

I have pushed some commits to fix your issue. You should use -x for now, that means extended ctags.

let g:vista_ctags_cmd = {
      \ 'haskell': 'hasktags -o - -x',
      \ }

Do you know how to distinguish the extended and non-extended options rationally?

[Avi-D-coder]

hasktags does not have a json option.

To determine format look at:
!_TAG_FILE_FORMAT 2 /extended format; --format=1 will not append ;" to lines/

It works with extended now, but not normal ctags.

Thanks

[smhc]

Seems to work ok, although I need to pass specific ctags options taken from the code:

let g:vista_ctags_cmd = {
\   'xyz': 'ctags --options=/my/options/ctag --format=2 --excmd=pattern --fields=nksSaf --extras= --file-
scope=yes --sort=no --append=no -f -'
\}

[liuchengxu]

I think !_TAG_FILE_FORMAT 2 /extended format; --format=1 will not append ;" to lines/ only exists for non-json output and you won't know it until you got some some output. The ideal way is to know which parser to use when passing the ctags command line options, so vista.vim can invoke the correct parser. I wish vista.vim could support both extended and plain ctags, just need a way to know which parser should be used.

[Avi-D-coder]

The ctag/extended format is pretty well defined why use json output?

[liuchengxu]

You need to extract the key/value on your own if using the ctag/extended format.

function! s:ParseTagfield(tagfields) abort
  let fields = {}

  if stridx(a:tagfields[0], ':') > -1
    let colon = stridx(a:tagfields[0], ':')
    let value = tagfield[colon+1:]
    let fields.kind = value
  else
    let fields.kind = s:ShortToLong(a:tagfields[0])
  endif

  if len(a:tagfields) > 1
    for tagfield in a:tagfields[1:]
      let colon = stridx(tagfield, ':')
      let name = tagfield[0:colon-1]
      let value = tagfield[colon+1:]
      let fields[name] = value
    endfor
  endif

  return fields
endfunction

function! vista#parser#ctags#FromRaw(line, container) abort
  if a:line =~ '^!_TAG'
    return
  endif
  let items = split(a:line, '\t')

  let line = {}

  let line.name = items[0]
  let line.tagfile = items[1]
  let line.tagaddress = items[2]

  let tagfields = s:ParseTagfield(items[3:])

  call extend(line, tagfields)

  let kind = line.kind

  let picked = {'lnum': line.line, 'text': line.name}

  if kind =~# '^f' || kind =~# '^m'
    if has_key(line, 'signature')
      let picked.signature = line.signature
    endif
    call add(t:vista.functions, picked)
  endif

  call s:Insert(a:container, kind, picked)

endfunction

With json ouptut, json_decode is what all I need:

function! vista#parser#ctags#FromJSON(line, container) abort
  let line = json_decode(a:line)

  let kind = line.kind

  let picked = {'lnum': line.line, 'text': line.name }

  if kind =~# '^f' || kind =~# '^m'
    if has_key(line, 'signature')
      let picked.signature = line.signature
    endif
    call add(t:vista.functions, picked)
  endif

  call s:Insert(a:container, kind, picked)
endfunction

The json output is definitely more reliable IMO.

[smhc]

My 2c. I think JSON format support is still fairly uncommon. I don't have the libjansson library available for my platform without going out of my way.

I have no doubt JSON is easier to code for, but if you choose to support non-json then the non-json parser would still need to be written anyway. You may as well only write one parser rather than two. Maintenance-wise it is much better to just have the one.

As far as modernisation/reliability is concerned, LSP should be the next step forward for people with those concerns.

[Avi-D-coder]

@liuchengxu I can see how json is simpler, just as long as ctag format is also supported.
Is determining if output is extended needed for json?
I can't figure out a simple a way of determining if json output is extended.
The obvious way is to watch for extended features, but that is more complex.

I think the default for custom commands should remain non json, so that the parser can rely on the header to determine format.

The following should be detected as original ctags by the parser.

let g:vista_ctags_cmd = {
      \ 'haskell': 'hasktags -o - -c',
      \ }

[liuchengxu]

I'm actually not against supporting the both since they have been done mostly already, just need to coordinate them right.

I think there is no need to check whether the output is extended for json format.

Currently I just check if the --output-format=json present in the options, if not, parse the output by vista.vim.

[liuchengxu]

@Avi-D-coder Does hasktags -o - -c work in tagbar? From the README of hasktags, it also needs -x option, see https://github.com/MarcWeber/hasktags/blob/assets/hasktags.vim#L3. And I have tried with tagbar, it won't work as expected without -x option.

[Avi-D-coder]

@liuchengxu It appears not. Tagbar is integrated with lushtags so I never used hasktags with it.

[liuchengxu]

So, in this PR, we add the support of parsing extended ctags, which fixes #4.

Although there still some remaining problems:

• how to take care of the custom ctags cmd.
• transform the short kind to the long, e.g., the long kind missing in hasktags.
• how to handle multiple filetypes share one type, e.g., cpp and cuda.
• the new ctags options may be unneccessary for the finder functionality, which can be replaced with the previous simpler one.

These problems will be resolved later, I think this PR is good to merge if the extended ctags parser works well. @Avi-D-coder @smhc What do you think?

[Avi-D-coder]

@liuchengxu Thanks for all your work