Compatibility: Disable in source blocks

[fleimgruber]

A source block (for motivation see below after debug trace) such as

#+BEGIN_SRC python
def new_algo(arg):
    """Science based on new_algo

    Reference :cite:`author_title_2017`
    """
    # nifty new algorithm
#+END_SRC

trips up org-ref with

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  search-forward(nil 23419)
  org-ref-get-bibtex-key-under-cursor()
  (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor))
  (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string)))
  (save-excursion (goto-char position) (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string))))
  (progn (save-excursion (goto-char position) (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string)))))
  (if org-ref-show-citation-on-enter (progn (save-excursion (goto-char position) (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string))))))
  (when org-ref-show-citation-on-enter (save-excursion (goto-char position) (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string)))))
  (lambda (window object position) (when org-ref-show-citation-on-enter (save-excursion (goto-char position) (let ((s (org-ref-format-entry (org-ref-get-bibtex-key-under-cursor)))) (with-temp-buffer (insert s) (fill-paragraph) (buffer-string))))))(#<window 3 on *Backtrace*> #<buffer python.org> 23337)
  help-at-pt-string()
  help-at-pt-kbd-string()
  display-local-help(t)
  help-at-pt-maybe-display()
  apply(help-at-pt-maybe-display nil)
  timer-event-handler([t 0 0 100000 t help-at-pt-maybe-display nil idle 0])

Motivation: The docstring above uses the :cite: from https://github.com/mcmtroffaes/sphinxcontrib-bibtex.

At first I thought this a corner case not worth covering in org-ref but then this could also creep up in other situations with ref: etc. So the first thing that came to my mind: Is there a way to selectively disable org-ref for certain document parts? If not, would this be a viable feature?

[jkitchin]

I can sort of reproduce this. I get some annoying messages in the Messages buffer, but no actual traceback.

This is buggy, but I feel like it might be a bug in org-mode. :cite: should not be getting recognized as a link IMHO. I will follow up on this on the mailing list.

[jkitchin]

It appears org-mode should recognize that as a link.

I don't think you want to disable org-ref in some parts. If you use real citations in those parts, or sometimes (e.g. for docstrings of another language that allow org markup), they would be disabled in a non-obvious way. Maybe it could be a user option, but it seems like harmless failure is a better solution to me. That is basically what I observe. How are you getting that?

[fleimgruber]

Thanks for checking back and your thoughts on this. I am with you on the non-obvious and harmless failure parts. To look more into it, I just updated to latest org-ref from Melpa.

Now given (part of the same section as above - again this is documentation material on using sphinxcontrib-bibtex)

#+BEGIN_SRC rst
.. bibliography:: o4c.bib
#+END_SRC

I get

Debugger entered--Lisp error: (error #(": o4c.bib does not seem to exist" 0 9 (fontified nil)))
  signal(error (#(": o4c.bib does not seem to exist" 0 9 (fontified nil))))
  error("%s does not seem to exist" #(": o4c.bib" 0 9 (fontified nil)))
  org-ref-find-bibliography()
  org-ref-cite-link-face-fn("`shakouri_g._multi-objective_2017")
  funcall(org-ref-cite-link-face-fn "`shakouri_g._multi-objective_2017")

[jkitchin]

That is the same problem. it looks like you have a bibliography link, but it is not valid.

[fleimgruber]

AFAICS, the problem is that the rst source block content line is parsed as containing an org-ref bibliography: link pointing to : o4c.bib which is indeed not valid. That is one of the cases where I thought it would be viable to have the org-ref parsing disabled for certain parts of the document, because the rst source block content line is a perfectly valid reStructuredText directive.

[jkitchin]

That is the problem, and it is the same problem. The issue originates mostly in org-mode, because it should not be recognizing any links in these blocks. It does because it uses regexps in fontification, rather than the org-parser. Since it does recognize them though, the problem is compounded by org-ref which tries to make the links functional. There isn't a way to make org-ref not parse this, since it doesn't do the parsing. The only real option is graceful failing.

I still don't understand why you are triggering the tracebacks though. For me it just silently fails and causes no issues.

[fleimgruber]

Maybe the full traceback contains a hint? https://gist.github.com/fleimgruber/ba9b6c19d5fc0097da4fc094ab482d3c

FWIW, this is on
org-ref 20170808.625
org-mode release_9.1

[brandonwillard]

I kept getting this error when trying to export. It stems from the function org-ref-find-bibliography, which uses a standard regex search for bibliography entries over the entire buffer. There's no org-mode machinery involved in that search, and — as a result — no "context" awareness (e.g. whether search results are in a src block or not).

It's a pretty big problem for org-scripting that, say, automatically configures bib settings.

My solution was to override org-ref-find-bibliography and make it use org-mode's options (i.e. #+BIBLIOGRAPHY and #+BIBLIOGRAPHYSTYLE) instead of links. Going a little further, I changed the export code to use those option values and output the bibliography block at the location of the last #+BIBLIOGRAPHY. The code is in this gist.

Those changes solve the problem observed here, but they seem to go against org-ref's general design. Are there reasons for preferring links over existing org-mode objects — like options and/or keywords?

[jkitchin]

The links offer syntax and fontification benefits, e.g. the bibliography link is red if the bibfile doesn't exist, and it is easy to access on export.

@brandonwillard can you post a small org file that is a problem for you? You gist is quite interesting, and indeed quite different than how org-ref is typically structured. I have tried to avoid things like modifying org-export-filter-parse-tree-functions.

It seems like it should be feasible to add some context awareness to org-ref-find-bibliography.

[brandonwillard]

Problems arise when calling org-ref-helm-insert-cite-link or org-html-export-to-html in an org file containing only the following:

[[cite:test]]

#+BEGIN_SRC elisp :exports both :results raw
(concat "bibliography:" (car-safe (f-glob "../tex/*.bib")))
#+END_SRC

As well, there are display issues that produce the following messages:

!!! No entry found !!! [2 times]
Error during redisplay: (jit-lock-function 1) signaled (file-missing "Opening input file" "No such file or directory" "~/projects/papers/tex-project-templates/src/org/zsh:1: unmatched \".bib")

I also wanted to avoid dealing with the parse tree, but I couldn't quickly figure out how to get keywords like #+BIBLIOGRAPHY to show up in filters added to org-export-filter-keyword-functions (or the key :filter-keyword in org-export-filters-alist). Such filters should make it possible to avoid the parse tree and produce the same results.

[jkitchin]

It looks like it is not hard to add some context awareness to org-ref-find-bibliography. e.g. to not find bibliographies that are in src blocks. Do you think that is sufficient? Are there other places that you think it should be disabled?

looking at this code, I wonder why it doesn't use org machinery to get the links. I guess there is some legacy there from trying to support keywords and links in the past.

[brandonwillard]

Org-mode options are extremely well suited for specifying bibliography files and styles, so is it just a matter of highlighting and other display features?

Extensions to the appropriate export backends sound like the right approach to injecting bibliographies. This is — in part — due to the ability it provides regarding the location of a bibliography in the structure of a document (via the document template). As well, the specifics of each export type are better organized across their respective backends (e.g. LaTeX and beamer getting a \bibliographystyle).

Unfortunately, it's not so simple and/or clean to make small changes at specific points in the template translation functions of existing backends (e.g. see org-latex-template). The next best thing might be a general filter for the body of an exported document via org-export-filter-body-functions. Per-backend customizations could be implemented using multiple dispatch/generic functions, e.g.

(cl-defgeneric btw//org-export-filter-body (body backend info)
  body)

(cl-defmethod btw//org-export-filter-body
  ((body t)
   (backend (eql latex))
   (info t))
  (if-let* ((bib-style (or (plist-get info :bibliographystyle)
                           ""))
            (bib-value (plist-get info :bibliography)))
      (progn
        (setq bib-value (format "\\bibliography{%s}"
                                (replace-regexp-in-string "\\.bib"
                                                          ""
                                                          (mapconcat 'file-relative-name bib-value ","))))
        (if (not (string-blank-p bib-style))
            (concat (format "\\bibliographystyle{%s}\n" bib-style)
                    bib-value))
        (concat body "\n" bib-value))
    body))

(add-to-list 'org-export-filter-body-functions 'btw//org-export-filter-body)

There's a fairly strong assumption built into this approach implying that a bibliography should always be at the end of a document's body. This assumption may, or may not, hold depending on what the backend does. Regardless, it's a way to avoid the parse tree.

[jkitchin]

It is not a good assumption that the bibliography should always be at the end. It should always be where the user puts it. There are times you need the bibliography before an appendix, for example.

That is also a very interesting snippet, thanks for sharing it! It doesn't seem to address the issue in org-ref-find-bibliography though. I think if we fix that, this issue would be resolved.

I also don't think it should be necessary to modify things like org-export-filter-body-functions. The export function of the links works as expected here.

[brandonwillard]

Using org-mode's options is a concise way to solve the problem(s) in this issue (e.g. the override in the gist), but it works outside of the link mechanism and — as a result — needs a means of exporting the bibliography.

There's more than one way to accomplish the latter, including the use of links. There also doesn't appear to be anything stopping one from using a single link to specify where the bibliography is exported. This link, in conjunction with bib files and styles specified by options, should do the job.