support files with tabs in them

[unhammer]

because you can't always get what you want

current-column gives the “visual” column, we want the stringual

[abingham]

There's a related change in 74cc5f3 on the column-in-bytes branch, though focused on ycmd itself rather than company. Can you look and see if these can be combined?

I used position-bytes because I think that's going to give ycmd what it needs. I may be wrong.

[unhammer]

Using your column function I can still complete with the tabby file, so that seems right :)

And using bytes does seem like the right thing for the protocol to do (rather than hoping clients are able to get decoded string length correctly). However, I don't think it's working 100 % yet. The error I got with the old version, when typing

    std::vec

was stuff like

deferred error : (args-out-of-range #("vec" 0 3 (fontified t)) 0 -4)

This went away when using (- (point) (line-beginning-position)).

However, I still can't do e.g.

/* «đŋtŧzčš» */  std::vec

– this gives

REQUEST [error] Error (error) while connecting to http://127.0.0.1:50589/completions.
Exception while fetching candidates: string index out of range

with ycmd-server showing

2015-04-10 16:20:16,500 - INFO - Received health request
2015-04-10 16:20:45,832 - INFO - Received completion request
2015-04-10 16:20:45,834 - DEBUG - Using filetype completion: False
Traceback (most recent call last):
  File "/home/kiwibird/src/ycmd/third_party/bottle/bottle.py", line 861, in _handle
    return route.call(**args)
  File "/home/kiwibird/src/ycmd/third_party/bottle/bottle.py", line 1734, in wrapper
    rv = callback(*a, **ka)
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/watchdog_plugin.py", line 100, in wrapper
    return callback( *args, **kwargs )
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/hmac_plugin.py", line 54, in wrapper
    body = callback( *args, **kwargs )
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/handlers.py", line 100, in GetCompletions
    completer.ComputeCandidates( request_data ),
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/completers/general/general_completer_store.py", line 78, in ComputeCandidates
    if not self.ShouldUseNow( request_data ):
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/completers/general/general_completer_store.py", line 61, in ShouldUseNow
    if self._filename_completer.ShouldUseNow( request_data ):
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/completers/completer.py", line 116, in ShouldUseNow
    if not self.ShouldUseNowInner( request_data ):
  File "/home/kiwibird/src/ycmd/ycmd/../ycmd/completers/general/filename_completer.py", line 72, in ShouldUseNowInner
    return ( start_column and ( current_line[ start_column - 1 ] == '/' or
IndexError: string index out of range
2015-04-10 16:20:46,499 - INFO - Received health request

[unhammer]

This is maybe a bit outside the common use-case, but I got curious so I tried

/* aa */ std::v

and saw "vector" as the first suggestion, but then with

/* « */ std::v

I got no suggestions. With

/* « */ std::ve

I get "end" as the first suggestion.

I get the same "column_num":18 and completion_start_column. 17 in *ycmd-content-log* whether I have the string « or aa (same byte-length) in that comment, but only with aa do I get completions.

(Does this stuff work in vim?)

[abingham]

I'm seeing the same behavior you describe. I'm fairly puzzled at this point!

@Valloric We're trying to get to the bottom of how to report column offsets to ycmd. The ycmd documentation says we should use 1-based byte-oriented columns, and I think I'm doing this properly. Indeed, without multi-byte characters things work properly.

However, as @unhammer describes, if I try to get completions for e.g.:

#include <vector>
/* « */ std::v

I send a request like this (I removed the noise escape sequences):

{"file_data": {
    "/Users/sixtynorth/sandbox/ycmd/foo.cpp": {
        "contents":"#include <vector>\n/* \u00ab */ std::v\n\", 
        "filetypes ["cpp"] }
    },
    "filepath":"/Users/sixtynorth/sandbox/ycmd/foo.cpp",
    "line_num": 2,
    "column_num": 16}

and get a response like (emacs s-exp notation):

((completion_start_column . 15)
 (completions .
              []))

The HTTP request to ycmd appears to have the correct offsets, etc.. Does this all work correctly in vim? If so then I'll assume that we've got a problem in the emacs client, and any thoughts you've got on what might be going on would be greatly appreciated.

[abingham]

OK, rubber duck moment: Is the problem with the unicode escaping?

[Valloric]

@abingham I'll have to take a closer look at this, but I wouldn't be surprised if this were a bug in ycmd. Unicode support continues to be a big thorn in ycmd's side; it's just whack-a-mole with multi-byte char problems. Could you try to repro this with YCM if you have the time? If you can repro, then please open an issue on ycmd's repo.

[abingham]

This change looks good to me, so I'll go ahead and merge it.

@Valloric I'll see if I can replicate this with ycm. I should have some free time this afternoon.

[abingham]

@Valloric I built vim and tried installing YouCompleteMe, but I couldn't get ycm to work. The configuration phase of the build seems to be picking up different versions of the python executable and libraries, and I get this when I run mvim:

Fatal Python error: PyThreadState_Get: no current thread

In my experience, this is often the result of python version mismatches.

Anyhow, I don't have much time that I can devote to this, so I'm going to drop it for now. If you've got a working vim/ycm installation, though, it shouldn't take more than a few minutes to see if completion on one of the misbehaving examples above works for you.

[ptrv]

@Valloric @abingham I have setup vim and YCM in order to reproduce this issue.
I was able to reproduce the bug in YCM with @unhammer's example.

 /* aa */ std::v

gave me correct results, however with

/* « */ std::v

there were no results, and with

/* « */ std::ve

the first result was std::end().

[abingham]

I've created a new issue in ycmd for this.