New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to open PDF specified by "file" field in bibtex file generated using Mendeley #105

Closed
shivamkalra opened this Issue Apr 10, 2016 · 22 comments

Comments

Projects
None yet
5 participants
@shivamkalra

shivamkalra commented Apr 10, 2016

I use Mendeley for updating the bibtex files.. I've (setq helm-bibtex-pdf-field "file") in my configuration but helm-bibtex is unable to find PDFs. I've temporarily fixed the issue by changing following code:

https://github.com/tmalsburg/helm-bibtex/blob/master/bibtex-completion.el#L413

to

if (f-file? (concat "/" path))
           collect (concat "/" path)

It seems like problem is f-full as it returns wrong path. I had to replace it with (concat "/" path). Does anyone have similar issue for Mendeley desktop generated bibtex file? I'm on Arch Linux using latest versions of both the programs.

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 10, 2016

Owner

I don't use Mendeley so I don't know whether the current code is working properly. But I haven't received any complaints about it recently. It's slightly weird that your code works because the path separator character on Linux is / not \. Could you please post a sample BibTeX entry for testing? Please also let me know the path where your PDFs are located. Thank you.

Owner

tmalsburg commented Apr 10, 2016

I don't use Mendeley so I don't know whether the current code is working properly. But I haven't received any complaints about it recently. It's slightly weird that your code works because the path separator character on Linux is / not \. Could you please post a sample BibTeX entry for testing? Please also let me know the path where your PDFs are located. Thank you.

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 10, 2016

Titus von der Malsburg notifications@github.com writes:

[..] It's slightly weird that your code works because the path
separator character on Linux is / not . Could you please post a
sample BibTeX entry for testing?

Actually my code also has /, sorry I made mistake before (updated now).
Here is the sample article entry from my bibtex library file.

@article{Ellabaan_2009,
author = {Ellabaan, Mostafa M H},
doi = {10.1109/ISDA.2009.115},
file = {:home/skalra/Dropbox/Bibliography/pdfs/2009/Ellabaan. 2009.pdf:pdf},
isbn = {9780769538723},
pages = {1--6},
title = {{Valley-Adaptive Clearing Scheme for Multimodal Optimization Evolutionary Search}},
year = {2009}
}

I'm aware that you've already taken care of missing /. However, it seems
like (f-full "home/skalra/Dropbox/Bibliography/pdfs/2009/Ellabaan. 2009.pdf") is not appending / instead it appends CWD.

Shivam

shivamkalra commented Apr 10, 2016

Titus von der Malsburg notifications@github.com writes:

[..] It's slightly weird that your code works because the path
separator character on Linux is / not . Could you please post a
sample BibTeX entry for testing?

Actually my code also has /, sorry I made mistake before (updated now).
Here is the sample article entry from my bibtex library file.

@article{Ellabaan_2009,
author = {Ellabaan, Mostafa M H},
doi = {10.1109/ISDA.2009.115},
file = {:home/skalra/Dropbox/Bibliography/pdfs/2009/Ellabaan. 2009.pdf:pdf},
isbn = {9780769538723},
pages = {1--6},
title = {{Valley-Adaptive Clearing Scheme for Multimodal Optimization Evolutionary Search}},
year = {2009}
}

I'm aware that you've already taken care of missing /. However, it seems
like (f-full "home/skalra/Dropbox/Bibliography/pdfs/2009/Ellabaan. 2009.pdf") is not appending / instead it appends CWD.

Shivam

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 10, 2016

Owner

I think I was relying on f-full doing something that isn't really specified, so I should change that. Thanks for reporting. In the medium term, I want to rewrite the whole system for finding PDFs. Currently, the code tries to make everyone happy, but the result is somewhat slow and not always predictable. I'm thinking about having a plug-in system, where people can choose the Mendeley-plugin for finding PDFs or whatever else plug-in they want. It would also be possible to activate several plug-ins, which may be useful for people with messier bibliographies.

Owner

tmalsburg commented Apr 10, 2016

I think I was relying on f-full doing something that isn't really specified, so I should change that. Thanks for reporting. In the medium term, I want to rewrite the whole system for finding PDFs. Currently, the code tries to make everyone happy, but the result is somewhat slow and not always predictable. I'm thinking about having a plug-in system, where people can choose the Mendeley-plugin for finding PDFs or whatever else plug-in they want. It would also be possible to activate several plug-ins, which may be useful for people with messier bibliographies.

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 10, 2016

Titus von der Malsburg notifications@github.com writes:

I think I was relying on f-full doing something that isn't really
specified, so I should change that. Thanks for reporting. In the
medium term, I want to rewrite the whole system for finding PDFs.
Currently, the code tries to make everyone happy, but the result is
somewhat slow and not always predictable. I'm thinking about having a
plug-in system, where people can choose the Mendeley-plugin for
finding PDFs or whatever else plug-in they want. It would also be
possible to activate several plug-ins, which may be useful for people
with messier bibliographies.

Sounds good. There is one more oddity I noticed with Mendeley. It is
that, when you have _ in file's path, it tends to escape it like
{\_}. May be it is standard way of escaping _ character in bib files.

Shivam

shivamkalra commented Apr 10, 2016

Titus von der Malsburg notifications@github.com writes:

I think I was relying on f-full doing something that isn't really
specified, so I should change that. Thanks for reporting. In the
medium term, I want to rewrite the whole system for finding PDFs.
Currently, the code tries to make everyone happy, but the result is
somewhat slow and not always predictable. I'm thinking about having a
plug-in system, where people can choose the Mendeley-plugin for
finding PDFs or whatever else plug-in they want. It would also be
possible to activate several plug-ins, which may be useful for people
with messier bibliographies.

Sounds good. There is one more oddity I noticed with Mendeley. It is
that, when you have _ in file's path, it tends to escape it like
{\_}. May be it is standard way of escaping _ character in bib files.

Shivam

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 12, 2016

Owner

The first issue should be fixed now. Could you please ttry the latest version and tell me if it works?

It is that, when you have _ in file's path, it tends to escape it like {\_}.

What do you mean by "it tends"? Are there also cases were it is not escaped like that?

Owner

tmalsburg commented Apr 12, 2016

The first issue should be fixed now. Could you please ttry the latest version and tell me if it works?

It is that, when you have _ in file's path, it tends to escape it like {\_}.

What do you mean by "it tends"? Are there also cases were it is not escaped like that?

@cryoguy

This comment has been minimized.

Show comment
Hide comment
@cryoguy

cryoguy Apr 15, 2016

In my case, Mendeley always escapes _ like {\_}, and I end up with "no pdf" when trying to open. Any fix to this? or is there sth else causing this issue? I am using Arch linux also. Thanks.

@Article{Fujishima1972,
abstract = {, Nature 238 (1972) 37–38.},
archivePrefix = {arXiv},
arxivId = {arXiv:1011.1669v3},
author = {FUJISHIMA, AKIRA and HONDA, KENICHI},
doi = {10.1038/238037a0},
eprint = {arXiv:1011.1669v3},
file = {:mnt/2nd{\_}BAY/papers/FUJISHIMA, HONDA - 1972 - Electrochemical Photolysis of Water at a Semiconductor Electrode - Nature.pdf:pdf},
isbn = {0028-0836},
issn = {0028-0836},
journal = {Nature},
keywords = {TiO2},
mendeley-tags = {TiO2},
month = {jul},
number = {5358},
pages = {37--38},
pmid = {12635268},
title = {{Electrochemical Photolysis of Water at a Semiconductor Electrode}},
url = {http://www.nature.com/doifinder/10.1038/238037a0},
volume = {238},
year = {1972}
}

cryoguy commented Apr 15, 2016

In my case, Mendeley always escapes _ like {\_}, and I end up with "no pdf" when trying to open. Any fix to this? or is there sth else causing this issue? I am using Arch linux also. Thanks.

@Article{Fujishima1972,
abstract = {, Nature 238 (1972) 37–38.},
archivePrefix = {arXiv},
arxivId = {arXiv:1011.1669v3},
author = {FUJISHIMA, AKIRA and HONDA, KENICHI},
doi = {10.1038/238037a0},
eprint = {arXiv:1011.1669v3},
file = {:mnt/2nd{\_}BAY/papers/FUJISHIMA, HONDA - 1972 - Electrochemical Photolysis of Water at a Semiconductor Electrode - Nature.pdf:pdf},
isbn = {0028-0836},
issn = {0028-0836},
journal = {Nature},
keywords = {TiO2},
mendeley-tags = {TiO2},
month = {jul},
number = {5358},
pages = {37--38},
pmid = {12635268},
title = {{Electrochemical Photolysis of Water at a Semiconductor Electrode}},
url = {http://www.nature.com/doifinder/10.1038/238037a0},
volume = {238},
year = {1972}
}

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 15, 2016

@tmalsburg Thank you. Latest version fixed the / issue.

@cryoguy I believe Mendley uses {_} for maths where as {\_} for delimiter.

shivamkalra commented Apr 15, 2016

@tmalsburg Thank you. Latest version fixed the / issue.

@cryoguy I believe Mendley uses {_} for maths where as {\_} for delimiter.

@cryoguy

This comment has been minimized.

Show comment
Hide comment
@cryoguy

cryoguy Apr 15, 2016

@shivamkalra Is this the reason I can not open the pdf file? my pdfs are stored in a path-name with a {_}which becomes {\_}after Mendeley conversion. Thanks.

cryoguy commented Apr 15, 2016

@shivamkalra Is this the reason I can not open the pdf file? my pdfs are stored in a path-name with a {_}which becomes {\_}after Mendeley conversion. Thanks.

@shivamkalra shivamkalra reopened this Apr 15, 2016

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 15, 2016

Owner

The current code replaces {\_} by _ does that solve your problem?

Owner

tmalsburg commented Apr 15, 2016

The current code replaces {\_} by _ does that solve your problem?

@cryoguy

This comment has been minimized.

Show comment
Hide comment
@cryoguy

cryoguy Apr 16, 2016

@tmalsburg I realize that and still try to figure out why it failed to locate the pdf. Is it because the file name? Thanks for you assistance.

cryoguy commented Apr 16, 2016

@tmalsburg I realize that and still try to figure out why it failed to locate the pdf. Is it because the file name? Thanks for you assistance.

@cryoguy

This comment has been minimized.

Show comment
Hide comment
@cryoguy

cryoguy Apr 16, 2016

I just figure out that the problem lies in org-ref. Helm-bibtex opens pdf ok (with pdf-tools in my case) when I disabled org-ref. Now I need to figure out the problem there. Thanks for all your assistance.

cryoguy commented Apr 16, 2016

I just figure out that the problem lies in org-ref. Helm-bibtex opens pdf ok (with pdf-tools in my case) when I disabled org-ref. Now I need to figure out the problem there. Thanks for all your assistance.

@cryoguy

This comment has been minimized.

Show comment
Hide comment
@cryoguy

cryoguy Apr 16, 2016

It is fixed now. jkitchin/org-ref#184 Thanks!

cryoguy commented Apr 16, 2016

It is fixed now. jkitchin/org-ref#184 Thanks!

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 17, 2016

Owner

Ah, glad to hear that it's working now! I'm closing this now. @shivamkalra feel free to reopen if you're still experiencing problems with underscores.

Owner

tmalsburg commented Apr 17, 2016

Ah, glad to hear that it's working now! I'm closing this now. @shivamkalra feel free to reopen if you're still experiencing problems with underscores.

@tmalsburg tmalsburg closed this Apr 17, 2016

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 22, 2016

@tmalsburg Could you please reopen this issue again? I'm getting same error in the latest version.

I had to replace entire chunk of sexp at https://github.com/tmalsburg/helm-bibtex/blob/master/bibtex-completion.el#L405 with following to make it work:

(f-join (f-root) (car (cdr (s-split ":" value))))

I'm assuming there is only one file.

And I changed the delimeter in Medley to - thus no funny {\_} or {_} symbols.

However, I've not got much time to debug your code. I'm also attaching the sample bibtex for which No PDF(s) found error is shown.

@article{Epitropakis_2013,
author = {Epitropakis, Michael G and Li, Xiaodong and Burke, Edmund K},
file = {:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013.pdf:pdf},
title = {{A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization}},
year = {2013}
}

You can see that file name is clean and all it needs to do is append (f-root) in front. Also take a look at what Mendley does when multiple files are added to an entry, it file field looks like this now:

file = {:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013.pdf:pdf;:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013(2).pdf:pdf},

So I believe general algorithm should be:

  1. Split by ;
  2. Map -> split by ':'
  3. Filter -> with cdr is pdf
  4. Filter -> is-file is true
  5. choose first element car

However, I believe, it must present another menu to select the files to show in case there are multiple existing files.

shivamkalra commented Apr 22, 2016

@tmalsburg Could you please reopen this issue again? I'm getting same error in the latest version.

I had to replace entire chunk of sexp at https://github.com/tmalsburg/helm-bibtex/blob/master/bibtex-completion.el#L405 with following to make it work:

(f-join (f-root) (car (cdr (s-split ":" value))))

I'm assuming there is only one file.

And I changed the delimeter in Medley to - thus no funny {\_} or {_} symbols.

However, I've not got much time to debug your code. I'm also attaching the sample bibtex for which No PDF(s) found error is shown.

@article{Epitropakis_2013,
author = {Epitropakis, Michael G and Li, Xiaodong and Burke, Edmund K},
file = {:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013.pdf:pdf},
title = {{A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization}},
year = {2013}
}

You can see that file name is clean and all it needs to do is append (f-root) in front. Also take a look at what Mendley does when multiple files are added to an entry, it file field looks like this now:

file = {:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013.pdf:pdf;:home/skalra/Dropbox/Bibliography/pdfs/2013/Epitropakis, Li, Burke - A Dynamic Archive Niching Differential Evolution Algorithm for Multimodal Optimization - 2013(2).pdf:pdf},

So I believe general algorithm should be:

  1. Split by ;
  2. Map -> split by ':'
  3. Filter -> with cdr is pdf
  4. Filter -> is-file is true
  5. choose first element car

However, I believe, it must present another menu to select the files to show in case there are multiple existing files.

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 22, 2016

Owner

Sure, go ahead. But I will very likely not have time to work on this over the next weeks. I already invested too much in time into this, especially given that I'm not even using the file field myself. But if you want to prepare a PR, I'll be happy to look at it.

Owner

tmalsburg commented Apr 22, 2016

Sure, go ahead. But I will very likely not have time to work on this over the next weeks. I already invested too much in time into this, especially given that I'm not even using the file field myself. But if you want to prepare a PR, I'll be happy to look at it.

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 22, 2016

@tmalsburg Only you have ability to open this PR (since you closed it).

Sure, I'll be happy to contribute. Although, please see my updated comment.

However, I'm thinking of just setting a variable bibtex-library-type to `mendley and then just add special conditional case for Mendley. What do you think?

shivamkalra commented Apr 22, 2016

@tmalsburg Only you have ability to open this PR (since you closed it).

Sure, I'll be happy to contribute. Although, please see my updated comment.

However, I'm thinking of just setting a variable bibtex-library-type to `mendley and then just add special conditional case for Mendley. What do you think?

@tmalsburg tmalsburg reopened this Apr 22, 2016

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 22, 2016

Owner

I think I found the problem. It occurs when the first field is empty.

Owner

tmalsburg commented Apr 22, 2016

I think I found the problem. It occurs when the first field is empty.

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 22, 2016

Owner

Should be fixed in 62593c6. Could you please check?

By the way, we already have support for multiple files separated by semicolons.

Owner

tmalsburg commented Apr 22, 2016

Should be fixed in 62593c6. Could you please check?

By the way, we already have support for multiple files separated by semicolons.

@shivamkalra

This comment has been minimized.

Show comment
Hide comment
@shivamkalra

shivamkalra Apr 22, 2016

@tmalsburg Thank you. Works nicely.

shivamkalra commented Apr 22, 2016

@tmalsburg Thank you. Works nicely.

@pangdouya

This comment has been minimized.

Show comment
Hide comment
@pangdouya

pangdouya Apr 23, 2016

Hi, I would like to report a issue. Helm-bibtex can't find the pdf file when the file path contains some nonascii character. For example the following bib item is generated by Mendeley .

@Article{Jancar2010a,
archivePrefix = {arXiv},
arxivId = {arXiv:1010.4760v4},
author = {Jan{\v{c}}ar, Petr},
eprint = {arXiv:1010.4760v4},
file = {:home/yin/Documents/Papers/arXiv preprint arXiv1010.4760/Jan{\v{c}}ar - 2010 - A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars.pdf:pdf},
journal = {arXiv preprint arXiv:1010.4760},
keywords = {decidability,deterministic context-free language,first-order gram-,language equivalence,mar,pushdown automaton,trace equivalence},
pages = {1--28},
title = {{A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars}},
url = {http://arxiv.org/abs/1010.4760},
year = {2010}
}

Helm-bibtex can't find pdf file for this item. But on my file system the path is
/home/yin/Documents/Papers/arXiv preprint arXiv1010.4760/Jančar - 2010 - A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars.pdf

pangdouya commented Apr 23, 2016

Hi, I would like to report a issue. Helm-bibtex can't find the pdf file when the file path contains some nonascii character. For example the following bib item is generated by Mendeley .

@Article{Jancar2010a,
archivePrefix = {arXiv},
arxivId = {arXiv:1010.4760v4},
author = {Jan{\v{c}}ar, Petr},
eprint = {arXiv:1010.4760v4},
file = {:home/yin/Documents/Papers/arXiv preprint arXiv1010.4760/Jan{\v{c}}ar - 2010 - A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars.pdf:pdf},
journal = {arXiv preprint arXiv:1010.4760},
keywords = {decidability,deterministic context-free language,first-order gram-,language equivalence,mar,pushdown automaton,trace equivalence},
pages = {1--28},
title = {{A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars}},
url = {http://arxiv.org/abs/1010.4760},
year = {2010}
}

Helm-bibtex can't find pdf file for this item. But on my file system the path is
/home/yin/Documents/Papers/arXiv preprint arXiv1010.4760/Jančar - 2010 - A Short Decidability Proof for DPDA Language Equivalence via First-Order Grammars.pdf

@tmalsburg

This comment has been minimized.

Show comment
Hide comment
@tmalsburg

tmalsburg Apr 23, 2016

Owner

Not sure why Mendeley escapes these characters when UTF-8 should work just fine. Anyway, I'm not going to work on this because I don't use Mendeley. PRs welcome, though.

Owner

tmalsburg commented Apr 23, 2016

Not sure why Mendeley escapes these characters when UTF-8 should work just fine. Anyway, I'm not going to work on this because I don't use Mendeley. PRs welcome, though.

@autosquid

This comment has been minimized.

Show comment
Hide comment
@autosquid

autosquid Jul 30, 2016

sorry to post to a closed issue.

@shivamkalra use '-' instead of '_' in mendeley as delimiter, while I use '-' in mine.

The following is the snippet I provide to org-ref to find pdfs for mendeley (org-ref-get-mendeley-filename). It works for me so far. Hope it helps.

(let ((e (org-ref-reftex-get-bib-field "file" entry)))
        (if (> (length e) 4)
            (let ((clean-field (replace-regexp-in-string "{\\|}\\|\\\\" "" e)))
              (let ((first-file (car (split-string clean-field ";" t))))
                (format "/%s" (substring first-file 1
                     (- (length first-file) 4)))))
          (format (concat
                   (file-name-as-directory org-ref-pdf-directory)
                   "%s.pdf")
                  key)))

autosquid commented Jul 30, 2016

sorry to post to a closed issue.

@shivamkalra use '-' instead of '_' in mendeley as delimiter, while I use '-' in mine.

The following is the snippet I provide to org-ref to find pdfs for mendeley (org-ref-get-mendeley-filename). It works for me so far. Hope it helps.

(let ((e (org-ref-reftex-get-bib-field "file" entry)))
        (if (> (length e) 4)
            (let ((clean-field (replace-regexp-in-string "{\\|}\\|\\\\" "" e)))
              (let ((first-file (car (split-string clean-field ";" t))))
                (format "/%s" (substring first-file 1
                     (- (length first-file) 4)))))
          (format (concat
                   (file-name-as-directory org-ref-pdf-directory)
                   "%s.pdf")
                  key)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment