New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use native JSON functionality in Emacs 27 #210

Open
vibhavp opened this Issue Dec 13, 2017 · 7 comments

Comments

Projects
None yet
7 participants
@vibhavp
Member

vibhavp commented Dec 13, 2017

Emacs 27.1 recently introduced a native JSON parser using the Jansson library. The entrypoints functions are json-serialize and json-parse-string, but they only work with hash tables and vectors.

Profiler data shows that maximum percentage of CPU time is spent in json-read-from-string and json-encode provided by json.el.

We can introduce a "polyfill" library that uses the native JSON implementation when available, and reverts to json.el otherwise.

@sebastiencs

This comment has been minimized.

Show comment
Hide comment
@sebastiencs

sebastiencs Dec 13, 2017

Member

I've seen those features too and I have started to working on it since yesterday.
Have you already write some code ?
As you say, the major issue is that it uses hash tables for object and vectors for arrays

Member

sebastiencs commented Dec 13, 2017

I've seen those features too and I have started to working on it since yesterday.
Have you already write some code ?
As you say, the major issue is that it uses hash tables for object and vectors for arrays

@phst

This comment has been minimized.

Show comment
Hide comment
@phst

phst Dec 13, 2017

Contributor

Note: we can't use lists for both objects and arrays because they are ambiguous (every alist is also a normal list). My plan is to allow alists for objects, but continue requiring vectors for arrays.

Contributor

phst commented Dec 13, 2017

Note: we can't use lists for both objects and arrays because they are ambiguous (every alist is also a normal list). My plan is to allow alists for objects, but continue requiring vectors for arrays.

@luxbock

This comment has been minimized.

Show comment
Hide comment
@luxbock

luxbock Apr 3, 2018

I'd like to 👍 this effort, as the LSP provided auto-complete is at times unusable for me due to input lag, and profiling seems to point at JSON parsing being the culprit.

Here is the relevant part from profiler output:

- company-post-command                                              19  45%
 - company--perform                                                 19  45%
  - company--continue                                               18  42%
   - company-calculate-candidates                                   18  42%
    - company--fetch-candidates                                     18  42%
     - company-call-backend-raw                                     18  42%
      - apply                                                       18  42%
       - company-capf                                               18  42%
        - completion-all-completions                                18  42%
         - completion--nth-completion                               18  42%
          - completion--some                                        18  42%
           - #<compiled 0xdae825>                                   18  42%
            - completion-pcm-all-completions                        13  30%
             - completion-pcm--find-all-completions                 13  30%
              - completion-pcm--all-completions                     13  30%
               - all-completions                                    13  30%
                - #<compiled 0x2867681>                             13  30%
                 - #<compiled 0x4305719>                            13  30%
                  - lsp--send-request                               13  30%
                   - lsp--make-message                              8  19%
                    - json-encode                                   8  19%
                     - json-encode-list                             8  19%
                      - json-encode-plist                           8  19%
                       - json-encode                                8  19%
                        - json-encode-list                          8  19%
                         - json-encode-plist                        8  19%
                          - json-encode                             8  19%
                           - json-encode-array                      7  16%
                            - mapconcat                             7  16%
                             - json-encode                          7  16%
                              - json-encode-list                    7  16%
                               - json-encode-plist                  7  16%
                                - json-encode                       7  16%
                                 - json-encode-list                 7  16%
                                  - json-encode-plist               7  16%
                                   - json-encode                    5  11%
                                    - json-encode-array             3  7%
                                     - mapconcat                    3  7%
                                      - json-encode                 3  7%
                                       - json-encode-list           3  7%
                                        - json-encode-plist         3  7%
                                         - json-encode              2  4%
                                          - json-encode-list        2  4%
                                           - json-encode-plist      2  4%
                                              json-encode-key       2  4%
                                         - json-encode-key          1  2%
                                          - json-read-from-string   1  2%
                                           - #<compiled 0x2792435>  1  2%
                                              kill-buffer           1  2%
                                    - json-encode-list              2  4%
                                     - json-encode-plist            2  4%
                                        json-encode-key             2  4%
                                   - json-encode-key                2  4%
                                    - json-read-from-string         2  4%
                                     + #<compiled 0xf58bc9>         1  2%
                           + json-encode-list                       1  2%
                   - lsp--send-wait                                 5  11%
                    - accept-process-output                         5  11%
                     - #<compiled 0x465b3d9>                        5  11%
                      + lsp--parser-on-message                      4  9%
                        lsp--parser-read                            1  2%
            - completion-basic-all-completions                      5  11%
             + completion-pcm--all-completions                      4  9%
             + completion-hilit-commonality                         1  2%

luxbock commented Apr 3, 2018

I'd like to 👍 this effort, as the LSP provided auto-complete is at times unusable for me due to input lag, and profiling seems to point at JSON parsing being the culprit.

Here is the relevant part from profiler output:

- company-post-command                                              19  45%
 - company--perform                                                 19  45%
  - company--continue                                               18  42%
   - company-calculate-candidates                                   18  42%
    - company--fetch-candidates                                     18  42%
     - company-call-backend-raw                                     18  42%
      - apply                                                       18  42%
       - company-capf                                               18  42%
        - completion-all-completions                                18  42%
         - completion--nth-completion                               18  42%
          - completion--some                                        18  42%
           - #<compiled 0xdae825>                                   18  42%
            - completion-pcm-all-completions                        13  30%
             - completion-pcm--find-all-completions                 13  30%
              - completion-pcm--all-completions                     13  30%
               - all-completions                                    13  30%
                - #<compiled 0x2867681>                             13  30%
                 - #<compiled 0x4305719>                            13  30%
                  - lsp--send-request                               13  30%
                   - lsp--make-message                              8  19%
                    - json-encode                                   8  19%
                     - json-encode-list                             8  19%
                      - json-encode-plist                           8  19%
                       - json-encode                                8  19%
                        - json-encode-list                          8  19%
                         - json-encode-plist                        8  19%
                          - json-encode                             8  19%
                           - json-encode-array                      7  16%
                            - mapconcat                             7  16%
                             - json-encode                          7  16%
                              - json-encode-list                    7  16%
                               - json-encode-plist                  7  16%
                                - json-encode                       7  16%
                                 - json-encode-list                 7  16%
                                  - json-encode-plist               7  16%
                                   - json-encode                    5  11%
                                    - json-encode-array             3  7%
                                     - mapconcat                    3  7%
                                      - json-encode                 3  7%
                                       - json-encode-list           3  7%
                                        - json-encode-plist         3  7%
                                         - json-encode              2  4%
                                          - json-encode-list        2  4%
                                           - json-encode-plist      2  4%
                                              json-encode-key       2  4%
                                         - json-encode-key          1  2%
                                          - json-read-from-string   1  2%
                                           - #<compiled 0x2792435>  1  2%
                                              kill-buffer           1  2%
                                    - json-encode-list              2  4%
                                     - json-encode-plist            2  4%
                                        json-encode-key             2  4%
                                   - json-encode-key                2  4%
                                    - json-read-from-string         2  4%
                                     + #<compiled 0xf58bc9>         1  2%
                           + json-encode-list                       1  2%
                   - lsp--send-wait                                 5  11%
                    - accept-process-output                         5  11%
                     - #<compiled 0x465b3d9>                        5  11%
                      + lsp--parser-on-message                      4  9%
                        lsp--parser-read                            1  2%
            - completion-basic-all-completions                      5  11%
             + completion-pcm--all-completions                      4  9%
             + completion-hilit-commonality                         1  2%
@blandest

This comment has been minimized.

Show comment
Hide comment
@blandest

blandest Aug 12, 2018

I have a tentative of a patch specifically for Emacs 27 (lightly tested on the master branch).
It replaces the usage of the 'json package with the native implementation and it also works around a number of small incompatibilities.
I don't think this was worth a pull request so I am just suggesting a patch for anyone interested.

https://pastebin.com/8CLdTday

You can apply it in the ELPA directory for lsp-mode:

[~/.emacs.d/elpa/lsp-mode-20180812.522]$ patch -p0 lsp-mode.txt

Emacs 27 must be built with support for jansson which is done automatically if libjansson-dev is installed.
$ sudo apt install libjansson-dev

blandest commented Aug 12, 2018

I have a tentative of a patch specifically for Emacs 27 (lightly tested on the master branch).
It replaces the usage of the 'json package with the native implementation and it also works around a number of small incompatibilities.
I don't think this was worth a pull request so I am just suggesting a patch for anyone interested.

https://pastebin.com/8CLdTday

You can apply it in the ELPA directory for lsp-mode:

[~/.emacs.d/elpa/lsp-mode-20180812.522]$ patch -p0 lsp-mode.txt

Emacs 27 must be built with support for jansson which is done automatically if libjansson-dev is installed.
$ sudo apt install libjansson-dev

@MaskRay

This comment has been minimized.

Show comment
Hide comment
@MaskRay

MaskRay Sep 9, 2018

Member

This feature will help a lot with completion, textDocument/references, semantic highlighting, ... which can have a large chunk of JSON data.

I just checked the payload for $ccls/publishSemanticHighlighting (implementation-specific notification, semantic highlighting in ccls)

liblmdb/mdb.c (11152 lines of code)

% strace -e write -fp $(pgrep -fn Release/ccls)
...
[pid 54042] write(1, "Content-Length: 547468\r\n\r\n{\"json"..., 547494 <unfinished ...>

clang/lib/Sema/SemaCodeComplete.cpp (8019 lines of code)

% strace -e write -fp $(pgrep -fo Release/ccls)
...
[pid 51221] write(1, "Content-Length: 556626\r\n\r\n{\"json"..., 556652 <unfinished ...>

Deserializing 500+KiB adds a significant lag and json-parse-string will assuredly help a lot.

Member

MaskRay commented Sep 9, 2018

This feature will help a lot with completion, textDocument/references, semantic highlighting, ... which can have a large chunk of JSON data.

I just checked the payload for $ccls/publishSemanticHighlighting (implementation-specific notification, semantic highlighting in ccls)

liblmdb/mdb.c (11152 lines of code)

% strace -e write -fp $(pgrep -fn Release/ccls)
...
[pid 54042] write(1, "Content-Length: 547468\r\n\r\n{\"json"..., 547494 <unfinished ...>

clang/lib/Sema/SemaCodeComplete.cpp (8019 lines of code)

% strace -e write -fp $(pgrep -fo Release/ccls)
...
[pid 51221] write(1, "Content-Length: 556626\r\n\r\n{\"json"..., 556652 <unfinished ...>

Deserializing 500+KiB adds a significant lag and json-parse-string will assuredly help a lot.

@MaskRay

This comment has been minimized.

Show comment
Hide comment
@MaskRay

MaskRay Sep 9, 2018

Member
Member

MaskRay commented Sep 9, 2018

@scturtle

This comment has been minimized.

Show comment
Hide comment
@scturtle

scturtle Sep 10, 2018

Hot fixes of :null.

diff --git a/lsp-ui-doc.el b/lsp-ui-doc.el
index 3668a82..bd1e3c1 100644
--- a/lsp-ui-doc.el
+++ b/lsp-ui-doc.el
@@ -294,6 +294,7 @@ HOVER is the doc returned by the LS.
 BOUNDS are points of the symbol that have been requested.
 BUFFER is the buffer where the request has been made."
   (if (and hover
+           (not (eq hover :null))
            (lsp--point-is-within-bounds-p (car bounds) (cdr bounds))
            (equal buffer (current-buffer)))
       (let ((doc (lsp-ui-doc--extract (gethash "contents" hover))))
diff --git a/lsp-ui-sideline.el b/lsp-ui-sideline.el
index f61e43c..7a62226 100644
--- a/lsp-ui-sideline.el
+++ b/lsp-ui-sideline.el
@@ -270,7 +270,8 @@ CURRENT is non-nil when the point is on the symbol."

 (defun lsp-ui-sideline--push-info (symbol tag bounds info)
   (when (and (= tag (lsp-ui-sideline--calculate-tag))
-             (not (lsp-ui-sideline--stop-p)))
+             (not (lsp-ui-sideline--stop-p))
+             (not (eq info :null)))
     (let* ((info (concat (thread-first (gethash "contents" info)
                            lsp-ui-sideline--extract-info
                            lsp-ui-sideline--format-info)))
diff --git a/lsp-methods.el b/lsp-methods.el
index 7c59332..7919960 100644
--- a/lsp-methods.el
+++ b/lsp-methods.el
@@ -1684,6 +1684,7 @@ RENDER-ALL if set to nil render only the first element from CONTENTS."
     (with-current-buffer buffer
       (setq lsp--cur-hover-request-id nil))
     (when (and hover
+               (not (eq hover :null))
                (lsp--point-is-within-bounds-p start end)
                (eq (current-buffer) buffer) (eldoc-display-message-p))
       (let ((contents (gethash "contents" hover)))

scturtle commented Sep 10, 2018

Hot fixes of :null.

diff --git a/lsp-ui-doc.el b/lsp-ui-doc.el
index 3668a82..bd1e3c1 100644
--- a/lsp-ui-doc.el
+++ b/lsp-ui-doc.el
@@ -294,6 +294,7 @@ HOVER is the doc returned by the LS.
 BOUNDS are points of the symbol that have been requested.
 BUFFER is the buffer where the request has been made."
   (if (and hover
+           (not (eq hover :null))
            (lsp--point-is-within-bounds-p (car bounds) (cdr bounds))
            (equal buffer (current-buffer)))
       (let ((doc (lsp-ui-doc--extract (gethash "contents" hover))))
diff --git a/lsp-ui-sideline.el b/lsp-ui-sideline.el
index f61e43c..7a62226 100644
--- a/lsp-ui-sideline.el
+++ b/lsp-ui-sideline.el
@@ -270,7 +270,8 @@ CURRENT is non-nil when the point is on the symbol."

 (defun lsp-ui-sideline--push-info (symbol tag bounds info)
   (when (and (= tag (lsp-ui-sideline--calculate-tag))
-             (not (lsp-ui-sideline--stop-p)))
+             (not (lsp-ui-sideline--stop-p))
+             (not (eq info :null)))
     (let* ((info (concat (thread-first (gethash "contents" info)
                            lsp-ui-sideline--extract-info
                            lsp-ui-sideline--format-info)))
diff --git a/lsp-methods.el b/lsp-methods.el
index 7c59332..7919960 100644
--- a/lsp-methods.el
+++ b/lsp-methods.el
@@ -1684,6 +1684,7 @@ RENDER-ALL if set to nil render only the first element from CONTENTS."
     (with-current-buffer buffer
       (setq lsp--cur-hover-request-id nil))
     (when (and hover
+               (not (eq hover :null))
                (lsp--point-is-within-bounds-p start end)
                (eq (current-buffer) buffer) (eldoc-display-message-p))
       (let ((contents (gethash "contents" hover)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment