Response: 400 JSON parse error: Unexpected end of input

[leoliu]

The terminal running tern --verbose prints

Response: 400 JSON parse error: Unexpected end of input

after sending the following JSON document to the tern server:

{"files":[{"type":"full","name":"/Users/ok/sources/js/tern/lib/tern.js","text":"..."}]}

where ... is the JSON string of the file tern.js.

Note also tern server replies successfully if sent JSON doc:

{"files":[{"type":"full","name":"\/Users\/ok\/sources\/js\/tern\/lib\/tern.js","text":"..."}]}

or

{"files":[{"type":"full","name":"lib/tern.js","text":"..."}]}

I wonder if there is bug somewhere? I noticed this problem after observing some behaviour difference in emacs 24.5 and 25.0.91.

Thanks,
Leo

[leoliu]

In case this isn't obvious. The issue is why a absolute path encoded differently but legally (i.e. with or without \) makes a difference to the Tern server?

[marijnh]

All the existing plugins are sending regular unescaped slashes in filenames, and that works. The error comes from JSON.parse, which is part of node.js, and not something Tern implements itself.

Could you show the full JSON data that causes the problem? Does JSON.parse accept it?

[leoliu]

Hi @marijnh,

Thanks for your reply and sorry for my delay.

I have come up with a recipe to reproduce the problem. The problem only exhibits in emacs 25 so you will need that. It will also use the tern project.

1. In the terminal, enter the tern project root directory and run tern --verbose which will start the tern server and print the listening port.
2. start emacs 25 and create the following command:

(defun test ()
  (interactive)
  (let* ((url-request-method "POST")
         (url-request-extra-headers '(("Content-Type" . "application/json")))
         (url-request-data
          (json-encode `(("files" . [(:type "full"
                                            :name ,buffer-file-name
                                            :text ,(buffer-string))]))))
         (url (format "http://127.0.0.1:%d/" 53305)))
    (url-retrieve-synchronously url)))

Change 53305 to the actual port printed in step 1.
3. Open tern.js in emacs and M-x test
4. Check the terminal that is running tern server and you should see Response: 400 JSON parse error: Unexpected end of input.

The full JSON data is pasted in https://bpaste.net/show/0563125585d9 which JSON.parse seems to happily accept.

Leo

[leoliu]

Hi @marijnh,

I wonder if you have time to look at this issue. I think only the json.el from 25.x is needed to reproduce the error in case you don't have emacs 25 ready. But json.el in 25 also depends on one of its new libs map.el.

Thanks,
Leo

[marijnh]

That information seems to point towards a bug (or backwards incompatibility) in the JSON serializer that ships with Emacs 25. Have you tried adding a logging statement just before the JSON.parse call near the end of bin/tern to see what it is actually choking on?

[leoliu]

Sorry I missed your reply on this issue.

Indeed the problem seems to be with Emacs. I made a http server to catch the request sent from emacs and it stripped some chars at the end of the JSON doc leading to unbalanced parentheses. I'll speak with emacs maintainers to know what is going on.

[update] bug logged with emacs: http://debbugs.gnu.org/23750
Thanks.

[dgutov]

@leoliu Can you verify whether json.el itself performs the encoding right? I.e. whether the problem lies in it, or in the transport layer.

[leoliu]

hi @dgutov, yes the serialised json string from json-encode is perfectly fine and can be parsed back. so it seems more likely a bug in the URL package.

[dgutov]

OK, good.

I wonder if you can reduce the scenario to something easier to debug: does URL truncate the request body in these circumstances independent of its contents? Or are buffer contents and/or using JSON somehow important?

[leoliu]

url-request-data is expected to be a unibyte-string and tern.js has a utf-8 char in it; (encode-coding-string (json-encode doc) 'utf-8) fixes the error. url-http is using length to count content-length which fails when request body has non-ascii chars.

similar issue: #605.

[dgutov]

url-request-data is expected to be a unibyte-string

Hmm. Since it's easy to make a mistake, maybe URL should check whether the string it's passed is multibyte.

[leoliu]

Agreed. There is a fixme in url-http-create-request that alludes to signal an error instead.