bibtex-completion-find-pdf-in-field not working with zotero and better-bibtex export format

[abradd]

I am running Emacs 24.5 on OS X with helm-bibtex (v. 20160823.900) and org-ref (v .20160816.1723) with the helm-bibtex backend. I switched my workflow from Mendeley to Zotero + better-bibtex (BBT) and it broke the open pdf functionality of org-ref. I believe I have tracked down the source of the issue in bibtex-completion-find-pdf-in-field which I am pretty confident is unaltered by org-ref. It doesn't seem to handle the export format of BBT. An example format for BBT export is:

file = {2011_Optogenetics.pdf:/Deisseroth/2011_Optogenetics.pdf:application/pdf}

BBT uses full paths or relative paths for the location of the pdf depending upon the location of your bibtex file (in my case they are in the same directory so the path is relative). I set bibtex-completion-library-path to accommodate this.

It seems that the

(--map (f-join it path)
  (-flatten bibtex-completion-library-path)) ; Jabref #100

command should capture this case, but it doesn't work for some reason. I was only able to open the pdf through org-ref by adding an additional function to bibtex-completion-find-pdf-field:

(--map (concat it path)
 (-flatten bibtex-completion-library-path)) ; Zotero + BBT export

[tmalsburg]

Hm, that format should actually work -- at least that was my memory. I'm currently traveling and will look into it after my return.

[tmalsburg]

Just tested this and it's working on my machine. So either this has been fixed or it has something to do with org-ref after all (I don't use org-ref) or it's a Windows-specific issue or I misunderstood the problem. Could you please test again?

[shenshifo]

Hi, I am running Emacs 25.1 on OS X with ivy-bibtex. I am trying to open pdf with ivy-bibtex. The file field in Zotero+BBT is relative path. However, ivy-bibtex seems can not find the file. Here is my configuration part for ivy-bibtex:

(use-package ivy-bibtex :ensure t)
(setq   bibtex-completion-notes-path "~/Dropbox/OrgMode/Papers.org"
        bibtex-completion-bibliography '("~/Dropbox/Zotero/Zotero.bib")
        bibtex-completion-library-path '("~/Dropbox/Zotero/"))
(setq   bibtex-completion-pdf-field "file")
(--map (concat it fpath)
 (-flatten bibtex-completion-library-path)) 
(setq   ivy-bibtex-open-pdf-function
        (lambda (fpath)
          (call-process "open" nil 0 nil "-a" "/Applications/Skim.app" fpath)))

Is there any way to tell the open-pdf-function the absolute path? Thanks.

[tmalsburg]

You say that your paths are relative, but relative to what? Ivy-bibtex tries to combine bibtex-completion-library-path with the path in the file field. Is that where your PDFs are located?

[shenshifo]

All file fields in bib file are relative to a certain folder, which I set the same as the bibtex-completion-library-path ( "~/Dropbox/Zotero/" in this case).

Here is a quick example in my .bib file

@Article{bell2008,
title = {Audit Labor Usage and Fees under Business Risk Auditing},
volume = {46},
timestamp = {2016-04-06T00:55:10Z},
number = {4},
urldate = {2012-12-28},
journal = {Journal of Accounting Research},
author = {Bell, T. B. and Doogar, R. and Solomon, I.},
year = {2008},
pages = {729--760},
file = {/Bell_2008_Journal_of_Accounting_Research.pdf}
}

The absolute path of the pdf file is "~/Dropbox/Zotero/Bell_2008_Journal_of_Accounting_Research.pdf".

As you suggest that Ivy-bibtex tries to combine bibtex-completion-library-path with the path in the file field, I think ivy-bibtex should be able to find the file. But it does not work somehow. Thanks for your time.

[tmalsburg]

Was the value in the file field generated by Zotero? I was under the impression that Zotero like most other bibliography managers uses this format (OP's example above):

file = {2011_Optogenetics.pdf:/Deisseroth/2011_Optogenetics.pdf:application/pdf}

[wadkar]

I am finding a similar issue while opening the PDF. In my case, the PDF filename contains spaces. When I try to open the PDF I get following error in the *Messages* buffer:

bibtex-completion-get-entry1: Search failed: "^@\\([^^\"@\\&$#%',={}()  
]*\\)[[:space:]]*[({][[:space:]]*DL_handbook[[:space:]]*,"

Also, I am using this solution to open the PDF file.

[tmalsburg]

@wadkar your problem seems to be related to org-ref. Different project. As far as helm-bibtex is concerned, spaces in file names should be ok.

[wadkar]

@tmalsburg thanks. I will take it over to org-ref. BTW, one can change BBT settings to remove the file name and file type from the field. This way the "File" field will contain the absolute path of the PDF file.

Check BBT's option in the Export tab.

[shenshifo]

@tmalsburg I used an add-on for Zotero (zotfile) to link all my files to a central dropbox folder. In Zotero, each item only has a link to the pdf file. However, BBT can successfully identify the relative path of the pdf, which is something as below

file = {/Bell_2008_Journal_of_Accounting_Research.pdf} for each item in bib file

@wadkar I will try it. It sounds the right path to this issue.

[abradd]

@wadkar I get the same error when my bibtex library isn't properly defined for org-ref, does it happen when opening the org-ref link or only when trying to access the pdf from the helm buffer? For me, it happens when I try to open (C-c C-o) the org-ref link as a helm buffer.

@tmalsburg This issue is still present for me even with updated emacs (25.1.1), org-ref (20161117.630) and helm-bibtex (20161119.412) on OS X 10.10.5. I am also using zotfile to relocate my pdfs to a designated location, BBT is able to find these pdfs just fine as well.

I cannot really see the issue being caused by org-ref, but I don't really know enough about debugging emacs to look much deeper at the moment. Do you have a reference for how to trace these function calls? I basically used C-h k and C-h f to see what functions were being called and traced them down to bibtex-completion-find-pdf-in-field in the helm-bibtex folder.

The only way I have been able to correct this is by adding the adapted bibtex-completion-find-pdf-in-field to my .emacs to add:

(--map (concat it path)
(-flatten bibtex-completion-library-path)) ; Zotero + BBT export

If I remove this from bibtex-completion-find-pdf-in-field I am unable to find the pdfs again. This shows up as "Try to get pdf" in the helm buffer.

It seems that @shenshifo is encountering the same issues on OS X, so perhaps this is an OS X specific issue with f-join. As @tmalsburg mentioned, the format used by BBT should work with the bibtex-completion-find-pdf-in-field as is... and yet it doesn't.

@tmalsburg, what OS are you using? I noticed you referenced Windows, but I am not using Windows so I do not know if this is a Windows issue as well.

@shenshifo I noticed that you are using the concat method in your configuration, but I don't understand why you put it where you did. Either I am missing something or things were lost in translation so I have posted the full bibtex-completion-library-path function with the concat adaptation here.

As a side note, BBT is generating the leading / as you have shown so you probably don't need the trailing / in your bibtex-completion-library-path, my example relative path is:

file = {2016_In Vivo Neural Recording and Electrochemical Performance of Microelectrode.pdf:/Zhao et al/2016_In Vivo Neural Recording and Electrochemical Performance of Microelectrode.pdf:application/pdf}

and my bibtex-completion-library-path is:

(setq bibtex-completion-library-path '("~/Google Drive/literature" "~/Google Drive/newliterature"  "~/Google Drive/zotero_data/storage"))

I think is this most important when using the concat hack as it isn't intelligent, like f-join, about dealing with superflous /s. This could be generating ~/Google Drive/literature//Zhao er al/2016_In Vivo Neural Recording and Electrochemical Performance of Microelectrode.pdf if you put a trailing / in bibtex-completion-library-path which is invalid.

[shenshifo]

@abradd Thanks a lot. It works after I add the function from your gist. Great!

[tmalsburg]

To be clear, helm-bibtex / ivy-bibtex currently don't support plain relative paths in the file field. So @shenshifo's problem is not a due to a bug or the OS. We currently have support for absolute paths and for specifications following this format:

file = {2011_Optogenetics.pdf:/Deisseroth/2011_Optogenetics.pdf:application/pdf}

I can add support for plain relative paths if this a standard format that enough people are using. It will slow down parsing of the library for everyone who's using the file field to link PDFs. To avoid this slowdown we'd have to revamp the code for finding PDFs completely. This is actually overdue but I don't have the time to work on it.

[tmalsburg]

Bare relative paths should now work. Please confirm. Details here 9a48c31.

[gelbwald]

I had the same problem as @abradd and @shenshifo on helm-bibtex-20170808.1124 . The problem was that f-join ignores prefixes if one of the arguments starts with /. So (f-join "/path/to/library" "/mypath/to/myfile.pdf") becomes "/mypath/to/myfile.pdf". The concat workaround in @abradd 's gist worked for me.

[tmalsburg]

Oh, I assumed that this was fixed. If not, please consider making a PR.

[abradd]

This broke on me again recently due to a change in BetterBibtex. This thread is quite old now but maybe some of the following will be useful to someone.

Bare relative paths should now work. Please confirm. Details here 9a48c31.

This does work for singular bare relative paths, but doesn't work if there is a second path:

"Wise et al/2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf;/Users/user/zotero/storage/7C8H6QW9/4545308.html"

as it takes the html file instead of the pdf and would not have worked with the previous examples in this thread because it uses f-join which doesn't handle the leading / on the relative path as mentioned above.

As an aside, it turns out the f-join doesn't handle the leading / in a relative path because it uses expand-file-name which explicitly requires that the path not start with a / or ~.

The reason this workflow broke for me was because BetterBibtex relative paths stopped working, they have since been restored and took on the format:

file = {/Wise et al/2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf;/Users/user/zotero/storage/7C8H6QW9/4545308.html}

Again the relative path failed because of the leading /. I tested adapting the the code in bibtex-completion-find-pdf-in-field to handle the prepended /, but it seemed far too tedious and likely to break. Instead I requested the prepended / be removed when exporting in BetterBibtex here. This produces the following file entry:

file = {Wise et al/2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf;/Users/user/zotero/storage/7C8H6QW9/4545308.html}

which both isn't support and still doesn't work without changes to the code. The changes required to get it to work are minor:

for result = (-first (lambda (path)
                         (if (and (not (s-blank-str? path))
                                  (f-exists? path)
                                  (not (f-directory? path)))

The check for path being a directory is required as (f-join (f-root) path) produces / as a result and the code considers it a valid file because path = "" and file-name = Wise et al/2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf. Because / is one of the first entries in the list it is selected as the return value even if the file-name is valid. The result of that long-winded explanation is that it works with minor changes and I can make a PR if you like.

But I wanted to discuss the topology of the the file entry in the bibtex file. At one point the field generated by BetterBibtex was:

file = {2011_Optogenetics.pdf:/Deisseroth/2011_Optogenetics.pdf:application/pdf}

which would have worked if not for the preceding /. Rather than re-inventing the wheel it may make more sense to see if BetterBibtex can return to the previous format above for relative paths (with the removal of the preceding /) to produce:

file = {2011_Optogenetics.pdf:Deisseroth/2011_Optogenetics.pdf:application/pdf}

which should work with the code as is. Or perhaps there is a more common format for the file field that would be preferable?

[retorquere]

BBT will actually return

file = {2011_Optogenetics.pdf:Deisseroth/2011_Optogenetics.pdf:application/pdf}

rather than

file = {Deisseroth/2011_Optogenetics.pdf}

depending on how you set the JabRef format in the preferences. If it's off, you get only file paths; if either 3 or 4, you get the 3-part format. But then you get it for all file fields, not just for those with relative paths.

[abradd]

I wasn't aware of that setting. Thanks for letting me know. The file field now looks like:

file = {2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf:Wise et al/2008_Microelectrodes, Microelectronics, and Implantable Neural Microsystems.pdf:application/pdf;IEEE Xplore Abstract Record:/Users/user/zotero/storage/7C8H6QW9/4545308.html:text/html}

and works with helm-bibtex without any changes.

But then you get it for all file fields, not just for those with relative paths.

Good point. The html content above also takes on the 3 part style as you mentioned. From my brief testing of the code it seems that even a full path used in the 3 part format works. As such, I'll close this issue.

[retorquere]

Wrt the problem as initially reported here, the "relative" paths starting with / was an error in BBT that I didn't know about but which is fixed now.

I'm also going to add an option to always export full absolute paths because relative paths don't use the BBT cache.

[tmalsburg]

Thanks for the update, @retorquere.

[hkarl]

This seems to be a bit brittle?

• I have bibtex-completion-library-path set to an absolute path /Users/... (no trailing slash)
• My bibtex files have a field file with the path RELATVE to to bibtex-completion-library-path (no starting /)
• If look I concatenate these two paths in a terminal (and add a slash between them, of course), the files are there
• but ivy-bibtex does not find them, just opens a Finder window pointing to root ?

Any idea how to debug this?

And: thanks a bundle for this excellent tool!