Info manual commands needed

[alphapapa]

Hi Daniel,

I was hoping to find some kind of consult-info command to search Info manuals, but I didn't see any, and looking at #6, I see #128, which is listed there as being rejected in favor of the existing info-goto-node command.

Having used Helm for many years, I don't see that command as being a sufficient alternative to the Helm info-related commands. For example, here is a command I use in my config, which integrates Helm sources for the Emacs, Elisp, and CL-lib manuals into a single command. It is the most efficient way I've found to find information in these manuals. It doesn't matter whether the search terms I enter are in the table of contents, node headings, or node content--wherever they're found, they show up instantly in the Helm results, and I can preview and jump to their locations instantly.

(defun ap/helm-info-emacs-elisp-cl ()
  "Helm for Emacs, Elisp, and CL-library info pages."
  (interactive)
  (helm :sources '(helm-source-info-emacs
                   helm-source-info-elisp
                   helm-source-info-cl)))

The alternative using Emacs's built-in and Consult's commands seem to require me to manually select each manual and search for the terms in turn. It's tedious by comparison, and often causes me to overlook relevant mentions of terms when they aren't in headings or the indexes.

So, IMO, what's needed is for Consult to provide a command to search any given Info manuals for given terms, to find the terms anywhere they are mentioned in the manuals, and to list the results by node heading, with a preview of the context in which the terms are found.

What do you think?

Thanks for your work.

[minad]

So you are looking for a full text search in info manuals ala consult-line or consult-grep? I think that's out of scope of Consult since it requires a lot of heavy lifting to bring the info manual in the right format and it seems a bit too specialized for Consult. Most of the Consult commands are more generic, and not specifically tied to a format.

The command proposed in #128 was essentially just a replacement for info-goto-node, much weaker than a full text search.

[minad]

Converting all the info manual to formatted text is quite expensive. One could either store the text in files such that they can be searched asynchronously by a variant of consult-grep or one keeps the entire text around in memory (I believe this is what Helm is doing?). Keeping the text in memory is too memory intensive for my taste.

Anyway both options bring complications, which I want to avoid in Consult. A better idea would be to create a separate consult-info package with full text search and preview, in the style of consult-recoll. It is better to experiment outside of Consult with this. If it turns out that everything is nicely efficient and simple one could still add it here.

Maybe @astoff has some ideas. He struggled with similar issues when trying to provide a full text search to his devdocs.el package. I am not sure what the status is there.

[astoff]

I don't know how Helm does it, but the .info files (as opposed to the .texi source) are already pretty much plain text, and info mode just narrows to a page and adds some faces and buttons.

So full-text searching a single manual should just be a matter of widening before computing the consult-line candidates, plus some logic before jumping. Of course there's the usual limitation regarding phrases split across lines.

The devdocs situation is more complicated because the manuals are in HTML format, so they require some expensive preprocessing before search. (The code that does it, FWIW, is here. I never merged it because it's very slow. But on the other hand one could argue that if you are trying a full-text search, then you are already kinda desperate so waiting a minute is acceptable.)

[minad]

@astoff Sounds good. Reading the info files, sanitizing them a bit (or even using info mode itself for formatting) and then feeding the lines into consult--read/consult-line should be doable. If anyone is interested in giving this a try, we could add it here. PR welcome. My only requirements are that the result is performant enough and that it doesn't lead to an explosion in complexity (complicated caching, preprocessing, etc.).

[alphapapa]

@minad As Augusto said, info files are barely more than plain text, so searching them should be fast. I can't imagine that any caching or preprocessing would be required. All that is required, I think, is to find matches for search tokens, then at each match, find which node the match is in for the sake of context, which shouldn't be much harder or slower than searching back to the node heading, like org-back-to-heading in an Org file. As well, when beginning a search, it would probably be necessary to bound the search to the content, i.e. excluding indexes, tables of contents, etc, but I don't think that would be difficult or slow, and there are probably existing functions in Emacs to help with that.

[alphapapa]

Here's a rough proof-of-concept. Ideally it would also make it easy to search multiple manuals, like the Helm command I showed earlier, and I don't yet understand how to integrate this into Consult's API to make use of its features, but this seems to work as a starting point.

(defun consult-info--candidates (terms manual)
  "Return candidates matching TERMS in Info MANUAL."
  (let ((buffer (get-buffer-create " *consult-info--candidates*"))
        (regexp (rx-to-string `(seq (or ,@terms))))
        (quoted-terms (mapcar #'regexp-quote terms)))
    (unwind-protect
        (with-current-buffer buffer
          (info-setup manual buffer)
          (delete-dups
           (cl-loop while (condition-case err
                              (Info-search regexp nil)
                            (user-search-failed))
                    when (cl-loop for term in quoted-terms
                                  always (save-excursion
                                           (goto-char (point-min))
                                           (re-search-forward term nil t)))
                    collect Info-current-node)))
      (kill-buffer buffer))))

(defun consult-info (manual)
  (interactive (list 
                (progn
                  (info-initialize)
                  (completing-read "Manual: "
                                   (info--manual-names current-prefix-arg)
                                   nil t))))
  (cl-labels ((collection (str _pred flag)
                          (pcase flag
                            ;; ('metadata (list 'metadata
                            ;;                  (cons 'affixation-function #'affix)
                            ;;                  (cons 'annotation-function #'annotate)))
                            (`t (unless (string-empty-p str)
                                  (consult-info--candidates (split-string str) manual)))))
              (try (string _table _pred point &optional _metadata)
                   (cons string point))
              (all (string table pred _point)
                   (all-completions string table pred)))
    (let* ((completion-styles '(consult-info))
           (completion-styles-alist (list (list 'consult-info #'try #'all "Consult Info"))))
      (info
       (format "(%s)%s" manual
               (completing-read "Search: " #'collection nil))))))

[minad]

@alphapapa Thanks. You dynamically start a regexp Info-search from a completion context (as you do in org-ql). This is probably necessary for the search to be sufficiently efficient. This approach is different from the one proposed by @astoff where the lines are collected beforehand.

[alphapapa]

@minad Yes. As well, line-based searching is inappropriate for Info manual nodes, since the lines are hard-wrapped in the files already.

Some context would probably be useful, and for that I'll add an annotation function similar to the one in org-ql-find that gets words around each matched term.

[minad]

To implement dynamic collections in Consult one can use the following code, which uses the asynchronous completion protocol from consult--read synchronously.

(defun consult--dynamic-collection (fun)
  (let (input cache)
    (lambda (action)
      (pcase action
        ('setup (consult--split-setup #'consult--split-nil))
        ('nil (and input (or cache (setq cache (funcall fun input)))))
        ((pred stringp) (setq input action cache nil))))))

(consult--read
 (consult--dynamic-collection
  (lambda (input)
    (list
     (format "%s1" input)
     (format "%s2" input)
     (format "%s3" input)))))

[minad]

A consultified version of #634 (comment):

;; -*- lexical-binding: t -*-

(defun consult-info--candidates (manuals input)
  (cl-loop
   for manual in (ensure-list manuals) nconc
   (with-temp-buffer
     (info-setup manual (current-buffer))
     (let ((regexp (string-join (split-string input) ".*")))
       (cl-loop
        while (ignore-errors (Info-search regexp nil))
        collect
        (cons (format "%-50s %s"
                      Info-current-node
                      (propertize
                       (string-trim (buffer-substring-no-properties
                                     (line-beginning-position)
                                     (line-end-position)))
                       'face 'font-lock-comment-face))
              (format "(%s)%s" manual Info-current-node)))))))

(defun consult-info (manuals)
  (interactive
   (list
    (progn
      (info-initialize)
      (completing-read-multiple
       "Manuals: "
       (info--manual-names current-prefix-arg)
       nil t))))
  (info
   (consult--read
    (consult--dynamic-collection
     (apply-partially #'consult-info--candidates manuals))
    :prompt "Search: "
    :require-match t
    :lookup #'consult--lookup-cdr)))

(defun consult--dynamic-collection (fun)
  (consult--async-split (consult--dynamic-collection-1 fun)))

(defun consult--dynamic-collection-1 (fun)
  (let (input cache)
    (lambda (action)
      (pcase action
        ('nil (and input (or cache (setq cache (funcall fun input)))))
        ("" (setq input nil cache nil))
        ((pred stringp) (setq input action cache nil))))))

Missing features:

• Preview
• Jump to the position of the match
• Better formatting and search result highlighting
• Use consult--regepx-compiler to convert input to regexp

[minad]

cc @okamsn since you've contributed the info command to the wiki and started #128.

[minad]

It would be nice to have this but it also needs a considerable amount of work to implement the points mentioned in #634 (comment). For now I moved this issues to #6. PR welcome.

[alphapapa]

@minad Of course, this is your project, but FWIW, it makes it harder for me to keep track of this issue's progress if it's closed before it's done. Tracking it in a meta issue may make for a shorter issue list, but for contributors who use issues to help track their unfinished work, it can lead to them being forgotten--or they might see that it's been closed and assume that someone else finished the work.

[minad]

@alphapapa Of course. I understand you.
But I also want to be honest here - closing or downgrading an issue means that the I won't act - if not contributed, the feature won't be added.

Generally I am closing issues quite eagerly to keep track of the ones I consider truly important (urgent bugs etc). Feature requests like this one do not fall und this category. Anyway, a contribution would be very welcome if it meets the quality requirements.

[alphapapa]

But I also want to be honest here - closing or downgrading an issue means that the I won't act - if not contributed, the feature won't be added.

I have issues like that on my projects as well; for them, I add the help wanted tag and comment that I don't intend to work on it myself. Then in a few weeks or months, when the next person wanting that feature comes along, they can easily find that issue rather than opening another one--which inevitably happens otherwise, because no one searches closed issues. :)

[minad]

Yeah, that's another approach. I think the difference is that I find it justified to close a feature request as acknowledged but not implemented and not planned. It would be nice to have the proposed feature but I can very well live without it. You did some preliminary work in #634 (comment) and I did some minor experimentation in #634 (comment), but polishing this up to a fully functional command in the style of other Consult commands needs much more work. We are still on a conceptual/design phase, so I feel that the wish list is just the right place to mention the idea. The wish list is pinned and I expect people to find it.

[minad]

I've implemented a more efficient consult-line-multi command and added the consult--dynamic-collection helper. We can now implement an efficient consult-info based on that. See 1247248.

[alphapapa]

Thanks, that looks very cool.

[minad]

I started a draft for a consult-info command in #727. It needs a bit more polishing, but not much. If you are interested, please give it a try.

[minad]

Merged #727