Better handle asynchronous Eldoc sources

This is a backward compatible redesign of significant parts of the
eldoc.el library.

Previously, Eldoc clients (major/minor modes setting its documentation
gathering variables) needed to directly call eldoc-message, an
internal function, to display the docstring to the user.  When more
asynchronous sources are involved, this is hard to do or even breaks
down.

Now, an Eldoc backend may return any non-nil, non-string value and
call a callback afterwards.  This restores power to Eldoc over how
(and crucially also when) to display the docstrings to the user.

Among other things, this fixes so called "doc blinking", or the very
short-lived display of a lower priority Eldoc message.  This would
happen if a particular producer of documentation finishes shortly
before a higher priority one, like in the LSP engine Eglot as reported
by Andrii Kolomoiets <andreyk.mad@gmail.com> and Dmitry Gutov
<dgutov@yandex.ru>.

Gathering docstrings is now delegated to the variable
eldoc-documentation-strategy, which is the new name for the
now-obsolete eldoc-documentation-function, and still accepts the
so-called "old protocol".  Examples of the new strategies enabled are
codified in functions such as eldoc-documentation-enthusiast,
eldoc-documentation-compose-eagerly, along with the existing
eldoc-documentation-compose and eldoc-documentation-default.

The work of displaying and formatting docstrings is shifted almost
fully to Eldoc itself and is delegated to the internal function
eldoc--handle-docs.  Among other improvements, it handles most of
eldoc-echo-area-use-multiline-p and outputs documentation to a
temporary *eldoc* buffer.

The manual and NEWS are updated to mention the new Eldoc features.

* lisp/emacs-lisp/eldoc.el (eldoc-documentation-functions):
Overhaul docstring.
(eldoc-documentation-compose, eldoc-documentation-default): Handle
non-nil, non-string values of elements of
eldoc-documentation-functions.  Use eldoc--handle-multiline.
(eldoc-print-current-symbol-info): Honour non-nil, non-string
values returned by eldoc-documentation-callback.
(eldoc--make-callback): Now also a function.
(eldoc-documentation-default, eldoc-documentation-compose): Tweak docstring.
(eldoc-documentation-enthusiast, eldoc-documentation-compose-eagerly):
New functions.
(eldoc-echo-area-use-multiline-p): Add new semantics.
(eldoc--handle-docs): Handle some of eldoc-echo-area-use-multiline-p.
(eldoc-doc-buffer): New command.
(eldoc-prefer-doc-buffer): New defcustom.
(eldoc--enthusiasm-curbing-timer): New variable.
(eldoc-documentation-strategy): Rename from eldoc-documentation-function.
(eldoc--supported-p): Use eldoc-documentation-strategy
(eldoc-highlight-function-argument)
(eldoc-argument-case, global-eldoc-mode)
(turn-on-eldoc-mode): Mention eldoc-documentation-strategy.
(eldoc-message-function): Mention eldoc--message.
(eldoc-message): Made obsolete.
(eldoc--message): New helper.

* lisp/hexl.el (hexl-print-current-point-info): Adjust to new
eldoc-documentation-functions protocol.

* lisp/progmodes/cfengine.el (cfengine3-documentation-function):
Adjust to new eldoc-documentation-functions protocol.

* lisp/progmodes/elisp-mode.el
(elisp-eldoc-documentation-function): Adjust to new
eldoc-documentation-functions protocol.

* lisp/progmodes/octave.el (octave-eldoc-function): Adjust to new
eldoc-documentation-functions protocol.

* lisp/progmodes/python.el (python-eldoc-function): Adjust to new
eldoc-documentation-functions protocol.

(eldoc-print-current-symbol-info): Rework with cl-labels.

* doc/emacs/programs.texi (Lisp Doc): Mention
eldoc-documentation-strategy.

* doc/lispref/modes.texi (Major Mode Conventions): Mention
eldoc-documentation-functions.

* etc/NEWS: Mention eldoc-documentation-strategy.

doc/emacs/programs.texi

@@ -1273,17 +1273,19 @@ Eldoc mode, which is turned on by default, and affects buffers whose
 major mode sets the variables described below.  Use @w{@kbd{M-x
 global-eldoc-mode}} to turn it off globally.
 
-@vindex eldoc-documentation-function
+@vindex eldoc-documentation-strategy
 @vindex eldoc-documentation-functions
   These variables can be used to configure ElDoc mode:
 
 @table @code
-@item eldoc-documentation-function
+@item eldoc-documentation-strategy
 This variable holds the function which is used to retrieve
 documentation for the item at point from the functions in the hook
 @code{eldoc-documentation-functions}.  By default,
-@code{eldoc-documentation-function} returns the first documentation
-string produced by the @code{eldoc-documentation-functions} hook.
+@code{eldoc-documentation-strategy} returns the first documentation
+string produced by the @code{eldoc-documentation-functions} hook, but
+it may be customized to compose those functions' results in other
+ways.
 
 @item eldoc-documentation-functions
 This abnormal hook holds documentation functions.  It acts as a

doc/lispref/modes.texi

@@ -469,9 +469,10 @@ variable @code{imenu-generic-expression}, for the two variables
 @code{imenu-create-index-function} (@pxref{Imenu}).
 
 @item
-The mode can specify a local value for
-@code{eldoc-documentation-function} to tell ElDoc mode how to handle
-this mode.
+The mode can tell Eldoc mode how to retrieve different types of
+documentation for whatever is at point, by adding one or more
+buffer-local entries to the special hook
+@code{eldoc-documentation-functions}.
 
 @item
 The mode can specify how to complete various keywords by adding one or

etc/NEWS

@@ -252,9 +252,11 @@ doc string functions.  This makes the results of all doc string
 functions accessible to the user through the existing single function hook
 'eldoc-documentation-function'.
 
-*** 'eldoc-documentation-function' is now a user option.
-Modes should use the new hook instead of this user option to register
-their backends.
+*** New user option 'eldoc-documentation-strategy'
+The built-in choices available for this user option let users compose
+the results of 'eldoc-documentation-functions' in various ways.  The
+user option replaces 'eldoc-documentation-function', which is now
+obsolete.
 
 ** Eshell
 

lisp/hexl.el

@@ -515,7 +515,7 @@ Ask the user for confirmation."
       (message "Current address is %d/0x%08x" hexl-address hexl-address))
     hexl-address))
 
-(defun hexl-print-current-point-info ()
+(defun hexl-print-current-point-info (&rest _ignored)
   "Return current hexl-address in string.
 This function is intended to be used as eldoc callback."
   (let ((addr (hexl-current-address)))

lisp/progmodes/cfengine.el

@@ -1294,7 +1294,7 @@ Calls `cfengine-cf-promises' with \"-s json\"."
                           'symbols))
         syntax)))
 
-(defun cfengine3-documentation-function ()
+(defun cfengine3-documentation-function (&rest _ignored)
   "Document CFengine 3 functions around point.
 Intended as the value of `eldoc-documentation-function', which see.
 Use it by enabling `eldoc-mode'."

lisp/progmodes/elisp-mode.el

@@ -1403,8 +1403,10 @@ which see."
       or argument string for functions.
   2 - `function' if function args, `variable' if variable documentation.")
 
-(defun elisp-eldoc-documentation-function ()
-  "`eldoc-documentation-function' (which see) for Emacs Lisp."
+(defun elisp-eldoc-documentation-function (_ignored &rest _also-ignored)
+  "Contextual documentation function for Emacs Lisp.
+Intended to be placed in `eldoc-documentation-functions' (which
+see)."
   (let ((current-symbol (elisp--current-symbol))
 	(current-fnsym  (elisp--fnsym-in-current-sexp)))
     (cond ((null current-fnsym)

lisp/progmodes/octave.el

@@ -1639,8 +1639,8 @@ code line."
                   (nreverse result)))))
   (cdr octave-eldoc-cache))
 
-(defun octave-eldoc-function ()
-  "A function for `eldoc-documentation-function' (which see)."
+(defun octave-eldoc-function (&rest _ignored)
+  "A function for `eldoc-documentation-functions' (which see)."
   (when (inferior-octave-process-live-p)
     (let* ((ppss (syntax-ppss))
            (paren-pos (cadr ppss))

lisp/progmodes/python.el

@@ -4573,7 +4573,7 @@ returns will be used.  If not FORCE-PROCESS is passed what
   :type 'boolean
   :version "25.1")
 
-(defun python-eldoc-function ()
+(defun python-eldoc-function (&rest _ignored)
   "`eldoc-documentation-function' for Python.
 For this to work as best as possible you should call
 `python-shell-send-buffer' from time to time so context in