Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omnicompletion with eclim / ScalaIDE is broken #2096

Closed
11 tasks done
matt-gardner opened this issue Apr 6, 2016 · 6 comments · Fixed by ycm-core/ycmd#455
Closed
11 tasks done

Omnicompletion with eclim / ScalaIDE is broken #2096

matt-gardner opened this issue Apr 6, 2016 · 6 comments · Fixed by ycm-core/ycmd#455

Comments

@matt-gardner
Copy link
Contributor

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside
the brackets) before filing your issue:

  • I have read and understood YCM's CONTRIBUTING document.
  • I have read and understood YCM's CODE_OF_CONDUCT document.
  • I have read and understood YCM's README, especially the
    Frequently Asked Questions section.
  • I have searched YCM's issue tracker to find issues similar to the one I'm
    about to report and couldn't find an answer to my problem. (Example Google
    search.
    )
  • If filing a bug report, I have included the output of vim --version.
  • If filing a bug report, I have included the output of :YcmDebugInfo.
  • If filing a bug report, I have included the output of
    :YcmToggleLogs stderr.
  • If filing a bug report, I have included which OS (including specific OS
    version) I am using.
  • If filing a bug report, I have included a minimal test case that reproduces
    my issue. (as well as I could...)
  • I understand this is an open-source project staffed by volunteers and
    that any help I receive is a selfless, heartfelt gift of their free time. I
    know I am not entitled to anything and will be polite and courteous.
  • I understand my issue may be closed if it becomes obvious I didn't
    actually perform all of these steps.

Thank you for adhering to this process! It ensures your issue is resolved
quickly and that neither your nor our time is needlessly wasted.

Issue Details

I have used YCM and eclim together successfully for several years (i.e., I know how to set this up properly, and I know what it's supposed to do). Recently, however, I have noticed a problem. When getting completions in scala files, YCM will correctly display the semantic completions returned by eclim, but it will not update them with its fuzzy matching when I continue typing. For example, if I type System.out.[pause], it will give me a correct list of completions. If I then type p (so the line looks like System.out.p[pause]), I get the same list of completions, not a filtered list that's been passed through YCM's fuzzy matching. The completion dropdown redraws, so it's clearly doing something, but it just gives the same list. When completing identifiers (not semantic completions that come from eclim), or when doing semantic completions in java files, the fuzzy matching works as expected. Also, when I hit backspace when in a semantic completion context, it updates the completion list with correct fuzzy matching. If I start typing again, however, the full (unfiltered) list reappears.

I can confirm that this is something introduced by an update to YCM. If I checkout commit 7fda494 (January 3, 2016), completion works as expected (that's just a randomly chosen commit because I knew this worked a few months ago). If I instead checkout commit f44435b (March 10, 2016), I see this problem. Somewhere in between those two commits, something broke. I'm happy to provide whatever else is needed to help debug the issue here.

Steps to reproduce:

I did all of these steps today, with fresh copies of everything.

  1. Compile vim from source (just to be sure it's not a vim version issue).
  2. Get a fresh install of eclim and eclipse (following, e.g., the instructions here: http://eclim.org/install.html#install-headless, being sure to install the sbt installer dependency)
  3. Install the latest YCM version using your favorite method.
  4. Edit a file in an eclim project with a scala nature
  5. Try to get semantic completions on some scala object. You will see the behavior outlined above.

Then, to get working semantic completion:

  1. Nuke the YCM directory (because I wasn't sure that the git submodules would downgrade properly; this step probably isn't necessary)
  2. In ~/.vim/bundle/: git clone https://github.com/Valloric/YouCompleteMe.git
  3. cd YouCompleteMe
  4. git checkout 7fda494026b081d1f4e7b
  5. git submodule update --init --recursive
  6. ./install.py
  7. Repeat steps 4 and 5 from above. Semantic completions now work as expected.

Requested Debug output

vim --version output:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Apr  6 2016 21:06:16)
Included patches: 1-1715
Compiled by mattg@mattg.dev.ai2
Huge version with GTK2 GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +tag_binary
+arabic          +file_in_path    +mouse_sgr       +tag_old_static
+autocmd         +find_in_path    -mouse_sysmouse  -tag_any_white
+balloon_eval    +float           +mouse_urxvt     -tcl
+browse          +folding         +mouse_xterm     +terminfo
++builtin_terms  -footer          +multi_byte      +termresponse
+byte_offset     +fork()          +multi_lang      +textobjects
+channel         +gettext         -mzscheme        +timers
+cindent         -hangul_input    +netbeans_intg   +title
+clientserver    +iconv           +packages        +toolbar
+clipboard       +insert_expand   +path_extra      +user_commands
+cmdline_compl   +job             -perl            +vertsplit
+cmdline_hist    +jumplist        +persistent_undo +virtualedit
+cmdline_info    +keymap          +postscript      +visual
+comments        +langmap         +printer         +visualextra
+conceal         +libcall         +profile         +viminfo
+cryptv          +linebreak       +python          +vreplace
+cscope          +lispindent      -python3         +wildignore
+cursorbind      +listcmds        +quickfix        +wildmenu
+cursorshape     +localmap        +reltime         +windows
+dialog_con_gui  -lua             +rightleft       +writebackup
+diff            +menu            +ruby            +X11
+digraphs        +mksession       +scrollbind      -xfontset
+dnd             +modify_fname    +signs           +xim
-ebcdic          +mouse           +smartindent     +xsmp_interact
+emacs_tags      +mouseshape      +startuptime     +xterm_clipboard
+eval            +mouse_dec       +statusline      -xterm_save
+ex_extra        -mouse_gpm       -sun_workshop    +xpm
+extra_search    -mouse_jsbterm   +syntax          
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/usr/share/vim"
 f-b for $VIMRUNTIME: "/usr/share/vim/vim74"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK  -pthread -I/usr/include/gtk-2.0 -I/usr/lib/x86_64-linux-gnu/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/harfbuzz     -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -L/build/buildd/ruby1.9.1-1.9.3.484/debian/lib -rdynamic -Wl,-export-dynamic  -L/usr/local/lib -Wl,--as-needed -o vim   -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo -lpango-1.0 -lfontconfig -lgobject-2.0 -lglib-2.0 -lfreetype   -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE  -lm -ltinfo -lnsl  -lselinux  -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions   -lruby-1.9.1 -lpthread -lrt -ldl -lcrypt -lm  -L/usr/lib

:YcmDebugInfo:

Printing YouCompleteMe debug information...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
-- Server has Clang support compiled in: False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
-- Server running at: http://127.0.0.1:53816                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
-- Server process ID: 12202                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
-- Server logfiles:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
--   /tmp/ycm_temp/server_53816_stdout.log                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
--   /tmp/ycm_temp/server_53816_stderr.log

:YcmToggleLogs stderr:

2016-04-06 22:14:16,190 - INFO - Received health request                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
2016-04-06 22:14:16,195 - INFO - Received debug info request  

OS: Both OS X (Yosemite) and Linux (Ubuntu 14.04, on AWS)

@puremourning
Copy link
Member

I suspect this is because one of the completion options contains non ASCII characters.We are working on fixes for these issues a which came with the port to Python 3. I have a branch which fixes most of them which I am working on. If you would like to try it out to see if it solves your issue there are some instructions here

If not please can you enable debug logging level and re-run your test and post the output of YcmToggleLogs stderr.

@matt-gardner
Copy link
Contributor Author

First, updated error information:

I enabled the debug logging level (the issue template should be modified to suggest this, by the way), and found a couple of exceptions. The first is this: ValueError: No semantic completer exists for filetypes: [u'scala']. This is found all over the place, but I'm not sure it's really meaningful, as I get completions just fine through eclim (and I see it for filetype java too). The second, more informative one is this:

2016-04-11 11:24:46,371 - INFO - Received filter & sort request                                                                                                           
Traceback (most recent call last):                                                                                                                                        
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/third_party/bottle/bottle.py", line 861, in _handle                                  
    return route.call(**args)                                                                                                                                             
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/third_party/bottle/bottle.py", line 1734, in wrapper                                 
    rv = callback(*a, **ka)                                                                                                                                               
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/watchdog_plugin.py", line 106, in wrapper                               
    return callback( *args, **kwargs )                                                                                                                                    
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/hmac_plugin.py", line 72, in wrapper                                    
    body = callback( *args, **kwargs )                                                                                                                                    
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/handlers.py", line 147, in FilterAndSortCandidates                      
    request_data[ 'query' ] ) )                                                                                                                                           
  File "/Users/mattg/clone/config/dotfiles/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/completers/completer_utils.py", line 160, in FilterAndSortCandidatesWrap
    ToCppStringCompatible( query ) )                                                                                                                                      
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 0: ordinal not in range(128)                                                               

So, it does indeed look like this is a unicode error, as you suggested.

I tried to follow your instructions to see if they fix this; hopefully I did it correctly. Specifically, what I did was:

  1. I added your fork as a remote to my YCM repository, and checked out your unicode branch.
  2. I ran git submodule update --init --recursive
  3. I added your fork as a remote to my ycmd repository, and checked out your unicode-investigation branch
  4. I ran ./install.py from the base YCM directory.

If I missed a step there, let me know.

After doing those steps, the unicode error no longer appears.

I do see some other weird behavior, though (perhaps I messed something up in checking out branches?). On the initial call to the semantic completion engine (by typing a . after an object), I get completions that are JSON-formatted (e.g., {'info': 'equals(Object obj) : boolean - Object', 'kind': 'f', ...}). Typing after this just makes the list of suggestions go away. If instead I hit ctr-space, the formatting changes to what I am used to seeing (e.g., equals( f equals(Object obj) : boolean - Object). But further typing in this state just does substring matching, not the fuzzy matching typical of YCM. The log file didn't contain anything that looked informative about what was going on here.

@vheon
Copy link
Contributor

vheon commented Apr 11, 2016

The first is this: ValueError: No semantic completer exists for filetypes: [u'scala']. This is found all over the place, but I'm not sure it's really meaningful, as I get completions just fine through eclim (and I see it for filetype java too).

That is normal: every event notification that we send to YCM we send it to the Identifier completer and the semantic completer, if it exists. In your case we don't have a native semantic completer for scala.

On the initial call to the semantic completion engine (by typing a . after an object), I get completions that are JSON-formatted (e.g., {'info': 'equals(Object obj) : boolean - Object', 'kind': 'f', ...}).

That is somewhat strange 😕 could be a bug in the way we call the omnifunc? I doubt that but we never know...

If instead I hit ctr-space, the formatting changes to what I am used to seeing (e.g., equals( f equals(Object obj) : boolean - Object). But further typing in this state just does substring matching, not the fuzzy matching typical of YCM.

This is the best behaviour that we can get with vim (at least the last time I checked): When you do <C-Space> we just do <C-X><C-O><C-P> which is "call the omnifunc and de-select the first choice"; after that we usually do fuzzy matching on CursorMovedI which unfortunately is not fired during omnifunc completion. When you type a semantic trigger we use another strategy and call the omnifunc ourself and we can provide fuzzy search even for omnifunc. The last time I looked into this it wasn't an easy/feasible task to do it even for <C-Space> but I have to admit that it was a couple of years ago...

@matt-gardner
Copy link
Contributor Author

I see; I didn't realize that the <C-Space> behavior was different from what I'm used to using. Trying this out with a working YCM commit confirms what you say.

So the weird behavior is just a bug with the semantic completion, and it's quite possibly just something wrong with how I checked out those particular branches of the code. If there's something specific you want me to do to try to figure that out, let me know. But at this point it's a separate issue, and one with the puremourning/unicode branch. We've confirmed that my original issue was a unicode problem, and one that goes away with @puremourning's fix.

@puremourning
Copy link
Member

Thanks for the additional info, I really appreciate it!

I get completions that are JSON-formatted (e.g., {'info': 'equals(Object obj) : boolean - Object', 'kind': 'f', ...})

Oops. I think I broke omnifunc completion when it returns an object rather than a list of strings. I have a feeling it might be the deepcopy I added last night, but TBH this is not an area we have too many tests for.

I'll try and debug this soon. It should be relatively easy to fix.

@puremourning
Copy link
Member

Thanks. Can confirm I can repro this with eclim, so we'll get it fixed :)

homu added a commit to ycm-core/ycmd that referenced this issue Apr 24, 2016
[READY] Fix issues with multi-byte characters

## Summary

This change introduces more general support for non-ASCII characters in buffers handled by YCMD.

In ycmd's public API, all offsets are byte offsets into the UTF-8 encoded buffers. We also assume (because, we have no other choice) that files stored on disk are also UTF-8 encoded. Internally, almost all of ycmd's functionality operates on unicode strings (python 2 `unicode()` and python 3 `str()` objects, transparently via `future`). Many of the downstream completion engines expect unicode code points as the offsets in their APIs. One special case is the `ycm_core` library (identifier completer and clang completer), which requires instances of the _native_ `str` type. All strings used within the c++ using `boost::python` require passing through `ToCppStringCompatible`

Previously, we were largely just assuming that `code point == byte offset` - i.e. all buffers contained only ASCII characters. This worked up to a point, but more by luck than judgement in a number of places.

## References

In combination with a YCM change and PR #453, I hope this:

- fixes #109
- fixes ycm-core/YouCompleteMe#2096
- fixes ycm-core/YouCompleteMe#2088
- fixes ycm-core/YouCompleteMe#2069
- fixes ycm-core/YouCompleteMe#2066
- fixes ycm-core/YouCompleteMe#1378

## Overview of changes

The changes fall into the following areas:

- Providing access to and conversion to/from code points and byte offsets (`request_wrap.py`)
- Changing certain algorithms/features to work entirely in codepoint space when they are trying to operate on logical 'characters' within the buffer (see known issues for why this isn't perfect, but probably most of the way there)
- Changing the completers to convert between the external (on both sides) and internal representations by using the shortcuts provided in `request_wrap.py`
- Adding tests for each of the completers for both completions and subcommands

## Completer-specific notes

Pretty much all of the completers I tested required some changes:
- clang uses utf-8 and byte offsets, but had some bugs with the `GetDoc` parsing stuff
- OmniSharp speaks codepoint offsets
- Tern speaks codepoint offsets
- JediHTTP speaks codepoint offsets
- tsserver speaks codepoint offsets
- gocode speaks byte offsets
- racer i did not test

## Further work / Known issues

- we act blissfully ignorant of the case where a unicode character consumes multiple code points (such as where there is a modifier after the code point)
- when typing a unicode character, we still get an exception from `bitset` (see #453 for that fix)
- the filtering and sorting system is 100% designed for ASCII only, and it is not in the scope of this PR to change that. Currently after any filtering operation, words containing non-ASCII characters are excluded.
- I did not get round to testing rust using racer
- there are further changes required to YouCompleteMe client (a further PR is coming for that)

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/ycmd/455)
<!-- Reviewable:end -->
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants