2x faster syntax highlighting using Python Interface to Vim :-)

Author: xolox

README.md

@@ -6,7 +6,7 @@ When you're familiar with integrated development environments you may recognize
 
 There's just one problem: You have to manually keep your tags files up-to-date and this turns out to be a royal pain in the ass! So I set out to write a Vim plug-in that would do this boring work for me. When I finished the plug-in's basic functionality (one automatic command and a call to [system()] [system] later) I became interested in dynamic syntax highlighting, so I added that as well to see if it would work -- surprisingly well I'm happy to report!
 
-## Install & usage
+## Installation
 
 Unzip the most recent [ZIP archive](http://peterodding.com/code/vim/downloads/easytags.zip) file inside your Vim profile directory (usually this is `~/.vim` on UNIX and `%USERPROFILE%\vimfiles` on Windows), restart Vim and execute the command `:helptags ~/.vim/doc` (use `:helptags ~\vimfiles\doc` instead on Windows). Now try it out: Edit any file type supported by Exuberant Ctags and within ten seconds the plug-in should create/update your tags file (`~/.vimtags` on UNIX, `~/_vimtags` on Windows) with the tags defined in the file you just edited! This means that whatever file you're editing in Vim (as long as it's on the local file system), tags will always be available by the time you need them!
 
@@ -18,10 +18,12 @@ Note that if the plug-in warns you `ctags` isn't installed you'll have to downlo
 
     $ sudo apt-get install exuberant-ctags
 
-### If you're using Windows
+### A note about Windows
 
 On Windows the [system()] [system] function used by `easytags.vim` causes a command prompt window to pop up while Exuberant Ctags is executing. If this bothers you then you can install my [shell.vim](http://peterodding.com/code/vim/shell/) plug-in which includes a [DLL](http://en.wikipedia.org/wiki/Dynamic-link_library) that works around this issue. Once you've installed both plug-ins it should work out of the box! Please let me know if this doesn't work for you.
 
+## Commands
+
 ### The `:UpdateTags` command
 
 This command executes [Exuberant Ctags] [exuberant_ctags] from inside Vim to update the global tags file defined by `g:easytags_file`. When no arguments are given the tags for the current file are updated, otherwise the arguments are passed on to `ctags`. For example when you execute the Vim command `:UpdateTags -R ~/.vim` (or `:UpdateTags -R ~\vimfiles` on Windows) the plug-in will execute `ctags -R ~/.vim` for you (with some additional arguments, see the troubleshooting section "`:HighlightTags` only works for the tags file created by `:UpdateTags`" for more information).
@@ -36,9 +38,11 @@ When you execute this command while editing one of the supported file types (see
 
 Note that this command will be executed automatically every once in a while, assuming you haven't changed `g:easytags_on_cursorhold`.
 
+## Options
+
 ### The `g:easytags_cmd` option
 
-The plug-in will try to determine the location where Exuberant Ctags is installed on its own but this might not always work because any given executable named `ctags` in your `$PATH` might not in fact be Exuberant Ctags but some older, more primitive `ctags` implementation which doesn't support the same command line options and thus breaks the `easytags.vim` plug-in. If this is the case you can set the global variable `g:easytags_cmd` to the location where you've installed Exuberant Ctags, e.g.:
+The plug-in will try to determine the location where Exuberant Ctags is installed on its own but this might not always work because any given executable named `ctags` in your `$PATH` might not in fact be Exuberant Ctags but some older, more primitive `ctags` implementation which doesn't support the same command line options and thus breaks the easytags plug-in. If this is the case you can set the global variable `g:easytags_cmd` to the location where you've installed Exuberant Ctags, e.g.:
 
     :let g:easytags_cmd = '/usr/local/bin/ctags'
 
@@ -124,7 +128,19 @@ If this is set and not false, it will suppress the warning on startup if ctags i
 
     :let g:easytags_suppress_ctags_warning = 1
 
-### How to customize the highlighting colors?
+## Faster syntax highlighting using Python
+
+The Vim script implementation of dynamic syntax highlighting is quite slow on large tags files. When the Python Interface to Vim is enabled the easytags plug-in will therefor automatically use a Python script that performs dynamic syntax highlighting about twice as fast as the Vim script implementation. The following options are available to change the default configuration.
+
+### The `g:easytags_python_enabled` option
+
+To disable the Python implementation of dynamic syntax highlighting you can set this option to true (1).
+
+### The `g:easytags_python_script` option
+
+This option defines the pathname of the script that contains the Python implementation of dynamic syntax highlighting.
+
+## How to customize the highlighting colors?
 
 The easytags plug-in defines new highlighting groups for dynamically highlighted tags. These groups are linked to Vim's default groups so that they're colored out of the box, but if you want you can change the styles. To do so use a `highlight` command such as the ones given a few paragraphs back. Of course you'll need to change the group name. Here are the group names used by the easytags plug-in:
 

autoload/xolox/easytags.vim

@@ -240,41 +240,59 @@ function! xolox#easytags#highlight() " {{{2
     let tagkinds = get(s:tagkinds, filetype, [])
     if exists('g:syntax_on') && !empty(tagkinds) && !exists('b:easytags_nohl')
       let starttime = xolox#misc#timer#start()
-      if !has_key(s:aliases, filetype)
-        let ctags_filetype = xolox#easytags#to_ctags_ft(filetype)
-        let taglist = filter(taglist('.'), "get(v:val, 'language', '') ==? ctags_filetype")
-      else
-        let aliases = s:aliases[&ft]
-        let taglist = filter(taglist('.'), "has_key(aliases, tolower(get(v:val, 'language', '')))")
-      endif
+      let used_python = 0
       for tagkind in tagkinds
         let hlgroup_tagged = tagkind.hlgroup . 'Tag'
-        if hlexists(hlgroup_tagged)
-          execute 'syntax clear' hlgroup_tagged
-        else
+        " Define style on first run, clear highlighting on later runs.
+        if !hlexists(hlgroup_tagged)
           execute 'highlight def link' hlgroup_tagged tagkind.hlgroup
+        else
+          execute 'syntax clear' hlgroup_tagged
         endif
-        let matches = filter(copy(taglist), tagkind.filter)
-        if matches != []
-          call map(matches, 'xolox#misc#escape#pattern(get(v:val, "name"))')
-          let pattern = tagkind.pattern_prefix . '\%(' . join(xolox#misc#list#unique(matches), '\|') . '\)' . tagkind.pattern_suffix
-          let template = 'syntax match %s /%s/ containedin=ALLBUT,.*String.*,.*Comment.*,cIncluded'
-          let command = printf(template, hlgroup_tagged, escape(pattern, '/'))
-          try
-            execute command
-          catch /^Vim\%((\a\+)\)\=:E339/
-            let msg = "easytags.vim: Failed to highlight %i %s tags because pattern is too big! (%i KB)"
-            call xolox#misc#msg#warn(printf(msg, len(matches), tagkind.hlgroup, len(pattern) / 1024))
-          endtry
+        " Try to perform the highlighting using the fast Python script.
+        if s:highlight_with_python(hlgroup_tagged, tagkind)
+          let used_python = 1
+        else
+          " Fall back to the slow and naive Vim script implementation.
+          if !exists('taglist')
+            " Get the list of tags when we need it and remember the results.
+            if !has_key(s:aliases, filetype)
+              let ctags_filetype = xolox#easytags#to_ctags_ft(filetype)
+              let taglist = filter(taglist('.'), "get(v:val, 'language', '') ==? ctags_filetype")
+            else
+              let aliases = s:aliases[&ft]
+              let taglist = filter(taglist('.'), "has_key(aliases, tolower(get(v:val, 'language', '')))")
+            endif
+          endif
+          " Filter a copy of the list of tags to the relevant kinds.
+          if has_key(tagkind, 'tagkinds')
+            let filter = 'v:val.kind =~ tagkind.tagkinds'
+          else
+            let filter = tagkind.vim_filter
+          endif
+          let matches = filter(copy(taglist), filter)
+          if matches != []
+            " Convert matched tags to :syntax command and execute it.
+            call map(matches, 'xolox#misc#escape#pattern(get(v:val, "name"))')
+            let pattern = tagkind.pattern_prefix . '\%(' . join(xolox#misc#list#unique(matches), '\|') . '\)' . tagkind.pattern_suffix
+            let template = 'syntax match %s /%s/ containedin=ALLBUT,.*String.*,.*Comment.*,cIncluded'
+            let command = printf(template, hlgroup_tagged, escape(pattern, '/'))
+            try
+              execute command
+            catch /^Vim\%((\a\+)\)\=:E339/
+              let msg = "easytags.vim: Failed to highlight %i %s tags because pattern is too big! (%i KB)"
+              call xolox#misc#msg#warn(printf(msg, len(matches), tagkind.hlgroup, len(pattern) / 1024))
+            endtry
+          endif
         endif
       endfor
       redraw
       let bufname = expand('%:p:~')
       if bufname == ''
         let bufname = 'unnamed buffer #' . bufnr('%')
       endif
-      let msg = "%s: Highlighted tags in %s in %s."
-      call xolox#misc#timer#stop(msg, s:script, bufname, starttime)
+      let msg = "%s: Highlighted tags in %s in %s%s."
+      call xolox#misc#timer#stop(msg, s:script, bufname, starttime, used_python ? " (using Python)" : "")
       return 1
     endif
   catch
@@ -561,6 +579,49 @@ endfunction
 
 let s:cached_filenames = {}
 
+function! s:python_available() " {{{2
+  if !exists('s:is_python_available')
+    try
+      execute 'pyfile' fnameescape(g:easytags_python_script)
+      redir => output
+        silent python easytags_ping()
+      redir END
+      let s:is_python_available = (output =~ 'it works!')
+    catch
+      let s:is_python_available = 0
+    endtry
+  endif
+  return s:is_python_available
+endfunction
+
+function! s:highlight_with_python(syntax_group, tagkind) " {{{2
+  if g:easytags_python_enabled && s:python_available()
+    " Gather arguments for Python function.
+    let context = {}
+    let context['tagsfiles'] = tagfiles()
+    let context['syntaxgroup'] = a:syntax_group
+    let context['filetype'] = xolox#easytags#to_ctags_ft(&ft)
+    let context['tagkinds'] = get(a:tagkind, 'tagkinds', '')
+    let context['prefix'] = get(a:tagkind, 'pattern_prefix', '')
+    let context['suffix'] = get(a:tagkind, 'pattern_suffix', '')
+    let context['filters'] = get(a:tagkind, 'python_filter', {})
+    " Call the Python function and intercept the output.
+    try
+      redir => commands
+      python import vim
+      silent python print easytags_gensyncmd(**vim.eval('context'))
+      redir END
+      execute commands
+      return 1
+    catch
+      redir END
+      " If the Python script raised an error, don't run it again.
+      let g:easytags_python_enabled = 0
+    endtry
+  endif
+  return 0
+endfunction
+
 " Built-in file type & tag kind definitions. {{{1
 
 " Don't bother redefining everything below when this script is sourced again.
@@ -591,29 +652,29 @@ set cpo&vim
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'lua',
       \ 'hlgroup': 'luaFunc',
-      \ 'filter': 'get(v:val, "kind") ==# "f"'})
+      \ 'tagkinds': 'f'})
 
 " C. {{{2
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'c',
       \ 'hlgroup': 'cType',
-      \ 'filter': 'get(v:val, "kind") =~# "[cgstu]"'})
+      \ 'tagkinds': '[cgstu]'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'c',
       \ 'hlgroup': 'cEnum',
-      \ 'filter': 'get(v:val, "kind") ==# "e"'})
+      \ 'tagkinds': 'e'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'c',
       \ 'hlgroup': 'cPreProc',
-      \ 'filter': 'get(v:val, "kind") ==# "d"'})
+      \ 'tagkinds': 'd'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'c',
       \ 'hlgroup': 'cFunction',
-      \ 'filter': 'get(v:val, "kind") =~# "[fp]"'})
+      \ 'tagkinds': '[fp]'})
 
 highlight def link cEnum Identifier
 highlight def link cFunction Function
@@ -622,7 +683,7 @@ if g:easytags_include_members
   call xolox#easytags#define_tagkind({
         \ 'filetype': 'c',
         \ 'hlgroup': 'cMember',
-        \ 'filter': 'get(v:val, "kind") ==# "m"'})
+        \ 'tagkinds': 'm'})
  highlight def link cMember Identifier
 endif
 
@@ -631,27 +692,27 @@ endif
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'php',
       \ 'hlgroup': 'phpFunctions',
-      \ 'filter': 'get(v:val, "kind") ==# "f"',
+      \ 'tagkinds': 'f',
       \ 'pattern_suffix': '(\@='})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'php',
       \ 'hlgroup': 'phpClasses',
-      \ 'filter': 'get(v:val, "kind") ==# "c"'})
+      \ 'tagkinds': 'c'})
 
 " Vim script. {{{2
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'vim',
       \ 'hlgroup': 'vimAutoGroup',
-      \ 'filter': 'get(v:val, "kind") ==# "a"'})
+      \ 'tagkinds': 'a'})
 
 highlight def link vimAutoGroup vimAutoEvent
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'vim',
       \ 'hlgroup': 'vimCommand',
-      \ 'filter': 'get(v:val, "kind") ==# "c"',
+      \ 'tagkinds': 'c',
       \ 'pattern_prefix': '\(\(^\|\s\):\?\)\@<=',
       \ 'pattern_suffix': '\(!\?\(\s\|$\)\)\@='})
 
@@ -663,13 +724,15 @@ call xolox#easytags#define_tagkind({
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'vim',
       \ 'hlgroup': 'vimFuncName',
-      \ 'filter': 'get(v:val, "kind") ==# "f" && get(v:val, "cmd") !~? ''<sid>\w\|\<s:\w''',
+      \ 'vim_filter': 'v:val.kind ==# "f" && get(v:val, "cmd", "") !~? ''<sid>\w\|\<s:\w''',
+      \ 'python_filter': { 'kind': 'f', 'nomatch': '(?i)(<sid>\w|\bs:\w)' },
       \ 'pattern_prefix': '\C\%(\<s:\|<[sS][iI][dD]>\)\@<!\<'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'vim',
       \ 'hlgroup': 'vimScriptFuncName',
-      \ 'filter': 'get(v:val, "kind") ==# "f" && get(v:val, "cmd") =~? ''<sid>\w\|\<s:\w''',
+      \ 'vim_filter': 'v:val.kind ==# "f" && get(v:val, "cmd", "") =~? ''<sid>\w\|\<s:\w''',
+      \ 'python_filter': { 'kind': 'f', 'match': '(?i)(<sid>\w|\bs:\w)' },
       \ 'pattern_prefix': '\C\%(\<s:\|<[sS][iI][dD]>\)'})
 
 highlight def link vimScriptFuncName vimFuncName
@@ -679,19 +742,19 @@ highlight def link vimScriptFuncName vimFuncName
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'python',
       \ 'hlgroup': 'pythonFunction',
-      \ 'filter': 'get(v:val, "kind") ==# "f"',
+      \ 'tagkinds': 'f',
       \ 'pattern_prefix': '\%(\<def\s\+\)\@<!\<'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'python',
       \ 'hlgroup': 'pythonMethod',
-      \ 'filter': 'get(v:val, "kind") ==# "m"',
+      \ 'tagkinds': 'm',
       \ 'pattern_prefix': '\.\@<='})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'python',
       \ 'hlgroup': 'pythonClass',
-      \ 'filter': 'get(v:val, "kind") ==# "c"'})
+      \ 'tagkinds': 'c'})
 
 highlight def link pythonMethodTag pythonFunction
 highlight def link pythonClassTag pythonFunction
@@ -701,12 +764,12 @@ highlight def link pythonClassTag pythonFunction
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'java',
       \ 'hlgroup': 'javaClass',
-      \ 'filter': 'get(v:val, "kind") ==# "c"'})
+      \ 'tagkinds': 'c'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'java',
       \ 'hlgroup': 'javaMethod',
-      \ 'filter': 'get(v:val, "kind") ==# "m"'})
+      \ 'tagkinds': 'm'})
 
 highlight def link javaClass Identifier
 highlight def link javaMethod Function
@@ -721,12 +784,12 @@ highlight def link javaMethod Function
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'cs',
       \ 'hlgroup': 'csClassOrStruct',
-      \ 'filter': 'get(v:val, "kind") ==# "c"'})
+      \ 'tagkinds': 'c'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'cs',
       \ 'hlgroup': 'csMethod',
-      \ 'filter': 'get(v:val, "kind") =~# "[ms]"'})
+      \ 'tagkinds': '[ms]'})
 
 highlight def link csClassOrStruct Identifier
 highlight def link csMethod Function
@@ -736,17 +799,17 @@ highlight def link csMethod Function
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'ruby',
       \ 'hlgroup': 'rubyModuleName',
-      \ 'filter': 'get(v:val, "kind") ==# "m"'})
+      \ 'tagkinds': 'm'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'ruby',
       \ 'hlgroup': 'rubyClassName',
-      \ 'filter': 'get(v:val, "kind") ==# "c"'})
+      \ 'tagkinds': 'c'})
 
 call xolox#easytags#define_tagkind({
       \ 'filetype': 'ruby',
       \ 'hlgroup': 'rubyMethodName',
-      \ 'filter': 'get(v:val, "kind") =~# "[fF]"'})
+      \ 'tagkinds': '[fF]'})
 
 highlight def link rubyModuleName Type
 highlight def link rubyClassName Type