Integrate with Relaton BibTeX functionality

[ronaldtse]

OGC needs to migrate/re-use bibliography from the BibTeX format in Metanorma.

Given Metanorma uses Relaton for references, this task relies on the completion of
relaton/relaton#48 .

This is time-sensitive work. Thanks!

[opoudjis]

Plan is as follows:

• Add bibtex document attribute to metanorma-standoc
• If present, parse it through BibTeX parsing, integration with asciidoctor-bibliography gem relaton/relaton#48 to a hash of keys to Relaton objects
• Add bibliography and citations to BibTex entities as for any other reference already in Metanorma: BibTex will be treated as equivalent to a Relaton cached entry.
• When bibliographic entries are parsed, before passing their keys on to Relaton for fetching, check to see if they already occur in the parsed BibTex file
• If they do, retrieve the Relaton bibitem XML from the parsed file, instead of Relaton
• Note that the reference in the bibliography should be not to a docidentifier, but to the BibTex database key: * [[[internal-reference,bibtex-key]]]
• Will need to write up in metanorma documentation.

[ronaldtse]

• When bibliographic entries are parsed, before passing their keys on to Relaton for fetching, check to see if they already occur in the parsed BibTex file

I think we should just treat BibTeX files as a type of Relaton database -- a relaton-db-bibtex gem? We will have other types of Relaton databases, such as Endnote and RIS. So it will be useful to have a common interface for these to hook into.

• Note that the reference in the bibliography should be not to a docidentifier, but to the BibTex database key: * [[[internal-reference,bibtex-key]]]

If the bibtex-key looks like a docidentifier, we should probably allow an explicit bibtex include, such as * [[[internal-reference,bibtex(bibtex-key)]]]?

[opoudjis]

When bibliographic entries are parsed, before passing their keys on to Relaton for fetching, check to see if they already occur in the parsed BibTex file

I think we should just treat BibTeX files as a type of Relaton database -- a relaton-db-bibtex gem? We will have other types of Relaton databases, such as Endnote and RIS. So it will be useful to have a common interface for these to hook into.

I don't care how this is implemented, because I won't be the one implementing this. Moved to relaton/relaton#48

[opoudjis]

Note that the reference in the bibliography should be not to a docidentifier, but to the BibTex database key: * [[[internal-reference,bibtex-key]]]

If the bibtex-key looks like a docidentifier, we should probably allow an explicit bibtex include, such as * [[[internal-reference,bibtex(bibtex-key)]]]?

I have no idea what this means, and I'm not interested in touching bibtex. Putting an include in means that I am having to deal with bibtex. The whole point of relaton/relaton#48 is to abstract any bibtex away so I never have to deal with it.

[ronaldtse]

@opoudjis surely you remember that Metanorma fetches according to document IDs via Relaton:

// fetches from relaton-iso
* [[[internal-ref,ISO 9001:2005]]]
// fetches from relaton-iec
* [[[internal-ref,IEC 60500-112]]]

And there is an alternative syntax where the Relaton key is explicit:

// fetches from relaton-iso
* [[[internal-ref,ISO(ISO 9001:2005)]]]
// fetches from relaton-iec
* [[[internal-ref,IEC(IEC 60500-112)]]]

By * [[[internal-reference,bibtex(bibtex-key)]]] I am referring to the second syntax. This has nothing to do with BibTeX, but it has to do with the way Metanorma uses Relaton.

[opoudjis]

I see. I think it is unnecessary. An ISO reference could in the bibtex file, or it could be retrieved live: by leaving out any bibtex prefix, you're allowing the retrieval behavior to change depending on whether you attach a bibliography file, without changing the document source. But I guess I can include that as optional behaviour, restricting the search space to the bibtex document.

[ronaldtse]

My intention with this syntax * [[[internal-reference,bibtex(bibtex-key)]]] is to avoid namespace conflicts. For example, if my BibTeX file contains ISO 9001, I might want to use that instead of a remotely fetched one.

[ronaldtse]

This is still necessary.

[opoudjis]

And it's still dependent on relaton/relaton#48, which has not been touched...

[opoudjis]

relaton/relaton#48 has been done. Reactivating.

[ronaldtse]

From @andrew2net

RelatonBib::BibtexParser.from_bibtex(bibtex)

@opoudjis this means Metanorma can support BibTeX .bib files in Relaton directly.

[ronaldtse]

@opoudjis Relaton already supports BibTeX parsing and importing as a Relaton data source: relaton/relaton-bib#57 .

This was recently requested here:
metanorma/metanorma-ogc#329

[ronaldtse]

We currently have a document attribute that sets the relaton dataset as :local-cache:, but it should likely be just a generic relaton source location:
https://www.metanorma.org/author/ref/document-attributes/

For BibTeX, we could use:

:relaton-data-source: source_one=../bibtex/my_bibtex_file.bib
:relaton-data-source: source_two=../bibtex/my_bibtex_file2.bib

and we can refer to as:

* [[[internal-reference1, source_one(key)]]] 
* [[[internal-reference2, source_two(key)]]] 

[opoudjis]

I am not, of course, going to welcome open-ended names of relaton data sources, which can potentially clash with other wrappers around bibliographic tag, and which will make parsing utterly unpredictable. The suggestion of

* [[[internal-reference1, source_one(key)]]] 
* [[[internal-reference2, source_two(key)]]] 

is discarded.

The intent is apparently for the bibliographic tag not to indicate what format the bibliographic file is in, because explicitness is not fun enough, and I should instead rely on either the file suffix, or on Metanorma peeking inside the file.

What could possibly go wrong with that.

Going along with the rest of this,

:relaton-data-source: source_one=../bibtex/my_bibtex_file.bib
:relaton-data-source: source_two=../bibtex/my_bibtex_file2.bib

will be referenced as:

* [[[internal-reference1, local-file(source_one, key)]]] 
* [[[internal-reference2, local-file(source_two, key)]]] 

and

:relaton-data-source: ../bibtex/my_bibtex_file.bib

as

* [[[internal-reference1, local-file(key)]]] 
* [[[internal-reference2, local-file(key)]]] 

I would prefer

:relaton-data-source: file=../bibtex/my_bibtex_file.bib,id=source_one,format=bibtex

But that's going to be too much for people to enter willingly.

[ronaldtse]

:relaton-data-source: file=../bibtex/my_bibtex_file.bib,id=source_one,format=bibtex

Is okay. Better than having an unpredictable key. Let's do this.

[opoudjis]

The proposed markup will not work, the repeated attribute keys will overwrite each other.

Instead, I propose we move the identifier, which is supposed to be unique, into the key:

:relaton-data-source-source_one: ../bibtex/my_bibtex_file.bib
:relaton-data-source-source_two: file=../bibtex/my_bibtex_file.bib,format=bibtex
:relaton-data-source: file=../bibtex/my_bibtex_file.bib,format=bibtex

If relaton-data-source is present, that is a unique bibliographic source. If relaton-data-source-{id} are present, they are the sources.

[ronaldtse]

We need to:

1. provide documentation
2. test it out
3. provide example files

[opoudjis]

Documentation has been provided in one of the many metanorma.org updates you have still not approved; in fact it's one of the first ones, metanorma/metanorma.org@3f3ebb2

Testing and example files are already done in 31e0fea#diff-1fd205e7e8774249755dd18f77d0000b36d8885e6c3157d67fa8d35a5e032735

[opoudjis]

metanorma.org update lost, resubmitting