Replace PDFContentImporter by another library

[koppor]

As user, I want to import a PDF into JabRef. Each PDF contains bibliographic information. Either by embedded XMP data or by just the txt on the title page (author, title, doi, ..., maybe included bibtex: https://www.ctan.org/pkg/coverpage or https://ctan.org/pkg/authorarchive?lang=de). This information should be extracted from the PDF.

Currently, a self-written functionality is employed. This works OK for LNCS and IEEE papers, but not for other publishers.

Solution Sketch

We have a grobid in place. This should be used. Check Apache Tika, too.

Steps:

1. If first PDF page containts @article or something direct BibTeX data (created by https://www.ctan.org/pkg/coverpage).
2. Check if a .bib file is embedded in the PDF. If yes, use this one. (created by https://ctan.org/pkg/authorarchive?lang=de)
3. Check if XMP data is available. If yes -> use that. Stop. Else continue.
4. Look for DOI in the first page. If present -> use that. Stop. Else continue.
5. Use Apache Tika/GROBID to extract PDF. Use that data.

Improvement possibility

Offer merge dialog from the different options (e.g., XMP + PDF scraping via GROBID)

Challenges

• Cover different cases (BibTeX text on the first page, .bib embedded, DOI on first page, different publisher PDFs (LNCS, LNI, IEEE, ACM, ...)
• Good test cases
  • Create test PDFs

Side notes

Check current drag'n'drop behavior. In 3.8.2, the user was asked whether (s)he wants to create a new entry or link the PDF.

In http://discourse.jabref.org/t/more-control-on-the-duplicate-finder/120/4?u=koppor the tool https://github.com/CrossRef/pdfextract was recommended. At first sight, it can fully replace our PDFContentImporter.

Refs JabRef#7209

[tobiasdiez]

But pdfextract is a (Ruby?) program and no Java library that can easily be integrated. ?

[koppor]

Depends on the definition of "easily". https://github.com/jruby/jruby/wiki/DirectJRubyEmbedding

(i) Being the initial author of PDFContentImporter, (ii) seeing that no one took over the last years, (iii) knowing the issues and (iv) limitations of the current implementation, I think, it is more easy to integrate the library than to fix PDFContentImporter.

Side story: Colleagues from an other department use the PDFContentImporter successfully with Springer and IEEE papers. Which are the two publishers it was designed for.

[koppor]

After rereading the homepage of pdfextract, the tool seems "only" to help to extract the title of the PDF. Thus, it does not help to improve the PDFContentImporter.

[koppor]

There are questions on StackExchange asking for PDF2Bib:

• Is there an open source tool for producing bibtex entries from paper PDFs?
• Extract titles from each page of a PDF?
• Extracting information from PDFs of research papers

Solutions outlined:

• cs2Bib
  • They also have XMP support
• CERMINE
• https://code.google.com/p/pdfssa4met/
• pdftk
• gscholar in combination with pdftotext
• bibfetch

More solutions:

• https://github.com/academia-edu/biblicit - a wrapper around the tools
• GROBIT/GROBID - see Think about adding GROBIT for "PDF to BibTeX" functionality #327

[koppor]

Refs JabRef#2474,

[koppor]

The important thing on this issue is to categorize "related work".

• Input: Set of represantive PDFs
• Output:
  • BibTeX generated for each PDF
  • Quality rating

[StarTrooper08]

There are questions on StackExchange asking for PDF2Bib:

• Is there an open source tool for producing bibtex entries from paper PDFs?
• Extract titles from each page of a PDF?
• Extracting information from PDFs of research papers

Solutions outlined:

• cs2Bib
  • They also have XMP support
• CERMINE
• https://code.google.com/p/pdfssa4met/
• pdftk
• gscholar in combination with pdftotext
• bibfetch

More solutions:

• https://github.com/academia-edu/biblicit - a wrapper around the tools
• GROBIT/GROBID - see Think about adding GROBIT for "PDF to BibTeX" functionality #327

Can we use python to implement pdf2bibtex convertion??