Username and password required

[halbtuerke]

I have ycmd configured to work in C++ buffers and everytime I open a buffer I get asked for a username and password.

(use-package ycmd
  :defer t
  :init
  (add-hook 'c++-mode-hook 'ycmd-mode)
  :config
  (use-package company-ycmd
    :config
    (company-ycmd-setup)
    (add-to-list 'company-backends (company-mode/backend-with-yas 'company-ycmd)))

  (set-variable 'ycmd-server-command '("python" "/Users/patrick/bin/ycmd/ycmd"))
  (set-variable 'ycmd-global-config "/Users/patrick/.ycm_extra_conf.py"))

My ycmd-config file can be found at https://gist.github.com/halbtuerke/66004fbd5730d0dbcb9a

I have ycmd (20150626.109) and company-ycmd (20150514.534) from Melpa installed and I'm using GNU Emacs 25.0.50.1 built 2015-06-19 on OS X 10.10.3.

I installed ycmd the following way:

• Cloned ycmd (796519f7a01ca9b5151d77058ab3985f9f9c2809) to ~/bin/ycmd
• git submodule update --init --recursive
• ./build.py --clang-completer

After pressing C-g to cancel the username prompt this can be found in the Messages buffer:

Contacting host: 127.0.0.1:61158
error in process filter: Quit [2 times]
Contacting host: 127.0.0.1:61158
REQUEST [error] Error (error) while connecting to http://127.0.0.1:61158/healthy.

And this is the ouput of the ycmd-server buffer:

2015-06-29 13:24:29,339 - DEBUG - Global extra conf not loaded or no function YcmCorePreload
serving on http://127.0.0.1:61077
2015-06-29 13:24:30,272 - INFO - Dropping request with bad HMAC.

I found ycm-core/ycmd#115 but as described above, I used the latest versions available of everything and unfortunately it does not work.

If you need any additional information please let me know.

[abingham]

Hmmm...is it possible that you've got some old .elc files laying around? It might be that you're still running old code.

[halbtuerke]

I deleted the two packages through the package-list-packages interface and then reinstalled them. I assume this completely deletes the packages with all byte-compiled files, please correct me if I'm wrong.

[abingham]

Yep, it certainly seems to on my machine. I'll see if I can replicate what you're seeing.

[halbtuerke]

Any news?

[abingham]

I'm afraid not. I can't replicate what you're seeing. As you've pointed out, this has the exact feel of an earlier defect which had to do with server/client mismatch.

If you do M-x find-function ycmd-open does it take you to the expected file, i.e. the one provided by melpa? Essentially I'm baffled right now, so I'm trying to rule out the possibility that you're somehow using the wrong client version.

[halbtuerke]

The command opens ycmd.el in ~/.emacs.d/elpa/ycmd-20150626.109, so it seems that this is the right version.

[abingham]

And once emacs has started ycmd, can use use ps (or whatever tool is on your OS) to see if it's running the expected version/copy?

[halbtuerke]

Seems like it. ps aux | grep ycmd:

patrick         71079   0.0  0.1  2598392   8472 s010  Ss+  12:06PM   0:01.55 /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/patrick/.ycmd/ycmd --options_file=/var/folders/tz/w8j2dxpj0ys6ff8njj36cyr80000gn/T/ycmd-options22975wA --log=debug --keep_logfile --idle_suicide_seconds=10800

[r4nt]

Note that I'm also recently seeing an increased number of users reporting this problem. I can't repro myself, though... Server/client versions definitely match for those users, as we do an internal packaging / rollout.

[ptrv]

I experienced this too, but I don't have a repro. The issue just appears and if I close the server buffer or the current buffer it disappears.
Basically if you start the server, it might happen that the HMAC just doesn't get regenerated. But this is just guessing...

[r4nt]

@Valloric for ideas

[Valloric]

Zero ideas, sorry. Something in emacs is sending requests to ycmd without the correct x-ycmd-hmac header. That's the best I got. We also know it's not a ycmd issues because to the best of my knowledge, this has never happened with YCM.

[abingham]

This one is a doozy. I'm now able to reproduce the bad behavior to a degree. Essentially, there appear to be certain problematic HMAC values, and if we randomly generate one of them then YCMD and emacs seem to disagree on the HMAC value.

The bad-hmac branch has a replacement for ycmd--generate-hmac-secret which has two hard-coded return values, one of which seems fine and one which I think reproduces this problem report. You just need to un/comment the version you want.

Could someone(s) please try this out and report if the "bad" HMAC value there reproduces the problem on their machine? If it does, then I think we're on the right track. If not...I might retire and become a florist or something...

[ptrv]

@abingham I can reproduce the problem with the "bad" hmac value. I get a prompt for entering username and password

[abingham]

Notes to myself:

I was able to collect a few HMAC strings which produce the bad behavior. Each of these lists of 16 bytes are "bad" HMAC secrets:

(234 58 16 54 41 19 243 213 253 10 126 172 124 193 147 86)
(60 50 163 252 86 240 193 176 101 181 176 127 130 90 179 183)
(55 119 34 193 154 227 72 201 89 41 237 141 207 174 38 82)
(50 184 4 177 61 30 86 192 177 253 123 21 21 76 40 249)
(140 115 100 54 9 159 199 250 197 51 85 174 143 192 165 95)
(219 35 7 67 84 224 4 95 193 160 75 181 168 179 210 177)

One pattern I've noticed is that each of them contains 192 or 193. Moreover, if I change 192 to 191 or 193 to 194 in the strings, then the problem goes away. This seems like a clue, but I don't know where it's pointing.

Possibilities: encoding issue with options file. Bug in hashing on one side of transaction. Bug in construction of content to be hashed.

[halbtuerke]

@abingham I too can reproduce the problem in the bad-hmac branch, but there's still something strange:

1. Use the good value => no prompt for credentials but parsing does not stop so no autocompletion
2. Use bad value => prompt for credentials
3. Switch back to good value => prompt for credentials

When I now use the "good" value it's not working anymore.

[abingham]

@halbtuerke I'm not sure what's going on there, but I think I've got a fix to the underlying issue. Try out the hmac-fix branch and let me know if that fixes things. With this fix in place, I haven't been able to reproduce the error, even with known bad HMAC secrets.

@ptrv Could you give it a try, too?

[halbtuerke]

For certain kind of files it seems to work and for others it still prompts for credentials. Here's the log of the ycmd-server: https://gist.github.com/62653163984561f97211

I have opened 3 different C++ files and 1 worked with completion.

[abingham]

Hmmm. If you change line 1018 from:

(encode-coding-string val 'utf-8 t)

to just:

val

does the problem persist? And do you have any non-ascii characters in your source files?

[halbtuerke]

After making this change I don't get prompted for credentials but I get the following messages:

File mode specification error: (error "Attempt to change byte length of a string")
Ycmd completion unavailable while parsing is in progress. [2 times]
Contacting host: 127.0.0.1:52432
Ycmd completion unavailable while parsing is in progress.
File mode specification error: (error "Attempt to change byte length of a string")

Also the parsing never stops so I don't get completions.

do you have any non-ascii characters in your source files?

Yes, there are german umlauts in some of the comments. In the file that previously worked there should only be ASCII characters.

[abingham]

Ah, right, you're on emacs 25. We've seen this before, and it was in fact what motivated us to do the encode-coding-string stuff in the first place. I'm not entirely sure how to proceed at this point...

[abingham]

On the master branch, if you replace uses of encode-coding-string with string-as-unibyte, does that help? E.g. (encode-coding-string val 'utf-8 t) becomes (string-as-unibyte val).

As you can probably tell, I'm flailing a bit here. With multiple emacs version and some apparent bugs/idiosyncracies in functions, it's hard to keep track of what might work, etc.

[halbtuerke]

On the master branch, if you replace uses of encode-coding-string with string-as-unibyte, does that help?

Yes, this helps when the buffer consists only of ASCII characters.

I deleted all german umlauts in one of the files and now it's working with string-as-unibyte.

[abingham]

OK, thanks for checking. I haven't had any problems with umlauts or non-ascii Norwegian characters on 24, so something's definitely fishy in 25. We obviously need to work with non-ascii text, so we need to sort this out.

Unfortunately, with EuroPython coming up, I don't have any more time to work on this right now, and probably not for the next week or so. Perhaps somebody can sort it out, and I'll try to come up with more ideas in the meantime.

[halbtuerke]

Ok, thanks.

[abingham]

I couldn't put this down as easily as I'd thought. I got my hands on emacs-25 and might have sorted this out. Try out the hmac-fix-emacs25 branch.

[halbtuerke]

It seems to work now. 🎉
I have tested a couple of cpp files and I got completion on every one of them.

[abingham]

OK, that's good news. The issue now is that the change does not appear to work for emacs 24, so we'll need to come up with some way to support both versions.

@r4nt If your emacs 25 users are having problems, it should be safe to switch them over to the hmac-fix-emacs25 branch.

@ptrv You want to take a try at cooking up a cross-version solution? I may not have time to look at it for more than a week.

[r4nt]

@abingham - will that also be safe for my emacs 24 users?

[abingham]

@r4nt No, it will break 24. So unless you can support two version easily, you'll need to wait until we have that built into the package.

[ptrv]

@abingham @halbtuerke I could repro the issue by inserting german umlauts into a buffer. With the changes in hmac-fix2 the issue seems to be fixed in Emacs 24 as well as in 25 at least on my machine.
Could you please try hmac-fix2 and let me know if the issue is fixed. If the problem still shows up, please also provide the affected files or exact repro steps.

[halbtuerke]

@ptrv I tested it with the same files as before and it seems to work with Emacs 25. No prompts for credentials and auto completion with company works. 👍

[ptrv]

@halbtuerke Great to hear!

@r4nt Is it possible for you to try out the hmac-fix2 branch on Emacs 24 and let me know if it's working?

[r4nt]

Works for me.

[r4nt]

(that's on emacs 24.3.1)

[ptrv]

@r4nt thanks for testing. but there are some issues with non-ascii characters in filenames and in the code

[abingham]

This seems to be fixed :)