How to change the display of newline character in candidates ?

[karta0807913]

Hello, I tried the following code, but company-mode shows characters with ? instead of ¬.

(defun company-test (command &rest args)
  (interactive (list 'interactive))
  (cl-case command
    (interactive (company-begin-backend 'company-test))
    (prefix '("T" . t))
    (candidates (list (concat "AB" (propertize "E" 'display "¬") "CD") ;; display property works.
                      (concat "AB" (propertize "\\n" 'display "¬") "CD") ;; display property works.
                      (concat "AB" (propertize "\n" 'display "¬") "CD") ;; display property not working :(
                      (concat (propertize "A\n" 'display "A¬B¬C") ))))) ;; display property not working :(

This is the story.
I tried to develop a company-mode based third party library with tabnine. Tabnine is code prediction using machine learning.
Sometimes, tabnine outputs the code block with escape character \n, and it presents in character ?.
Therefore, I want to use display property to change this situation.
However, there is no luck.
Can someone tell me how to change the display property in the escape character?

[dgutov]

First of all, there is an existing backend https://github.com/TommyX12/company-tabnine. Are you doing something different?

[karta0807913]

I created a pull request in that repository.
Trying to fix some existing problems.

[dgutov]

Second: company-mode does not support non-printable characters in completion strings, including newlines. There is no good way to show them in the popup. Hence the character ? that replaces them.

Instead of trying to use a display property (this will not work; display is used for rendering the overlay popup, the property is overridden), return modified strings which do not contain any unsupported characters.

You can add some text property on each strings which would store the original, and then use those values in post-completion to insert the full string.

[karta0807913]

Thank you for reply!
It's very helpful :).
I change my code to something like this:

 (defun company-test (command &rest args)
   (interactive (list 'interactive))
   (cl-case command
     (interactive (company-begin-backend 'company-test))
     (prefix '("" . t))
     (candidates (list "AB¬CD"
                       "ABCD"))
     (post-completion
      (print args)
      (delete-region (point) (- (point) (length (car args))))
      (insert (replace-regexp-in-string "¬" "\n" (car args))))))

Thank you a lots.

[dgutov]

Not exactly what I had in mind, but sure -- this should work.

But if you ever see the length of completion strings as a problem, you can choose some truncation algorithm and store the originals in a text property.

[karta0807913]

I think you are right.
I should consider boundary exceptions and possible character collisions.

(defun company-test (command &rest args)
  (interactive (list 'interactive))
  (cl-case command
    (interactive (company-begin-backend 'company-test))
    (prefix '("" . t))
    (candidates (list (propertize "AB¬CD"
                                  'original-string "AB\nCD")
                      (propertize "A1B¬C¬D"
                                  'original-string "AB\nC¬D")))
    (post-completion
     (delete-region (point) (- (point) (length (car args))))
     (insert (get-text-property 0 'original-string (car args))))))

Thank you!