Using refextract for unstructured references

[fschwenn]

When the metadata for an article include references but only in an unstructured way, refextract should be used in the workflow after the individual spider (pipeline.py?).

At the moment refextract is only called if a fulltext is attached. But this wont be the case for all records. And in some cases it's even with fulltext better to start from a list of individual unstructured references than from the complete PDF, where refextract first has to find such list.

[michamos]

As already said though email, I don't think this should be done in hepcrawl (contents of the email follow).

There are several cases that can arise:

1. The publisher makes available a full structured reference
2. The publisher makes available a list of unstructured references
3. The publisher does not make any reference available in the metadata

In case 1., we don't need need refextract as Hepcrawl can do the
conversion from the publisher's reference format to ours, whereas in
case 3. there is nothing Hepcrawl can do besides providing the PDF.

So case 2. remains, but I think it would be better to have Hepcrawl
populate the raw references in the record, and run refextract (or in
the future maybe Grobid) in the workflow as is done curently to extract
references from PDF. We should have a task there that does reference
extraction from raw references in case they have been provided but
there are no parsed references. In this way, we cleanly separate the
task of translating between metadata formats (Hepcrawl) and parsing
references (refextract) and in the future we can easily swap refextract
for Grobid when it is mature enough.