Include sameAs and depiction information from EntityFacts in lobid-gnd index

[acka47]

I wrote in #44:

Possible further step: Enrich JSON-LD with links from EntityFacts. There might be a dump in the future we could build a map from.

The dump is there now (see https://data.dnb.de/opendata/) and we already set up an ES index with the data.

Reasons for including the information in the data:

• For some use cases (get all entries with a depiction, give me the GND entry with a specific ISNI etc.) we will need to query the data.
• Also, it is confusing when some things are shown in the HTML that aren't in the JSON.

[acka47]

How will we embed the information? It gets a bit problematic as we already have sameAs information from from the GND via OAI-PMH that is in part redundant but structured differently (string array vs. object array) . For example the sameAs for Lambert Heller:

Current lobid:

{
  "sameAs": [
    "http://viaf.org/viaf/313478392",
    "http://orcid.org/0000-0003-0232-7085"
  ]
}

EntityFacts:

{
  "sameAs":[
    {
      "@id":"http://d-nb.info/gnd/1066621098/about",
      "collection":{
        "abbr":"DNB",
        "name":"Gemeinsame Normdatei (GND) im Katalog der Deutschen Nationalbibliothek",
        "publisher":"Deutsche Nationalbibliothek",
        "icon":"http://www.dnb.de/SiteGlobals/StyleBundles/Bilder/favicon.png?__blob=normal&v=1"
      }
    },
    {
      "@id":"http://orcid.org/0000-0003-0232-7085",
      "collection":{
        "abbr":"ORCID",
        "name":"Open Researcher and Contributor ID",
        "publisher":"ORCID"
      }
    },
    {
      "@id":"http://viaf.org/viaf/313478392",
      "collection":{
        "abbr":"VIAF",
        "name":"Virtual International Authority File (VIAF)",
        "publisher":"OCLC",
        "icon":"http://viaf.org/viaf/images/viaf.ico"
      }
    }
  ]
}

After some discussion with @fsteeg, we prefer the following approach:

• Adjust the JSON so that there is always an object array for sameAs like

{
  "sameAs":[
    {
      "@id":"http://viaf.org/viaf/313478392"
    },
    {
      "@id":"http://orcid.org/0000-0003-0232-7085"
    }
  ]
}

• When ingesting EntityFacts datas, replace sameAs arrays.

[acka47]

Another challenge is the linking to wikipedia. From GND we get the following:

{
  "id":"http://d-nb.info/gnd/118634313",
  "wikipedia":[
    "https://de.wikipedia.org/wiki/Ludwig_Wittgenstein"
  ]
}

In EntityFacts we get links to the German and English wikipedia as part of the sameAs array:

{
  "@id":"http://d-nb.info/gnd/118634313",
  "sameAs":[
    {
      "@id":"https://de.wikipedia.org/wiki/Ludwig_Wittgenstein",
      "collection":{
        "abbr":"dewiki",
        "name":"Wikipedia (Deutsch)",
        "publisher":"Wikimedia Foundation Inc.",
        "icon":"https://de.wikipedia.org/static/favicon/wikipedia.ico"
      }
    },
    {
      "@id":"https://en.wikipedia.org/wiki/Ludwig_Wittgenstein",
      "collection":{
        "abbr":"enwiki",
        "name":"Wikipedia (English)",
        "publisher":"Wikimedia Foundation Inc.",
        "icon":"https://en.wikipedia.org/static/favicon/wikipedia.ico"
      }
    }
  ]
}

One solution would be to ignore the wikipedia links from GND and only use those from EntityFacts. This would mean that we would lose the links for all resources that are not part of EntityFacts. I think this is ok as on ~170 resources have a wikipedia link that are not also in EntitFacts, see this query for resources with wikipedia link that are neither of type person, nor corporate body nor PlaceOrGeographicName.

[acka47]

As discussed offline, I am also ok with keeping the – then mostly redundant – wikipedia links in wikipedia.

[fsteeg]

Currently processing EntityFacts data integrated in our JSON data. When we last discussed this, we said we'd like to stay close to the source data, as we do for the core GND data.

For the depiction, I only replaced @id with id (note it's not an array):

"depiction": {
  "id": "https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg",
  "thumbnail": {
    "id": "https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg?width=270"
  },
  "url": "https://commons.wikimedia.org/wiki/File:MarkTwain.LOC.jpg?uselang=en"
}

For sameAs, I replaced @id with id and added collection.name as sameAs.label:

"sameAs": [ {
  "id": "https://www.deutsche-digitale-bibliothek.de/entity/118624822",
  "label": "Deutsche Digitale Bibliothek",
  "collection": {
    "abbr": "DDB",
    "name": "Deutsche Digitale Bibliothek",
    "publisher": "Deutsche Digitale Bibliothek",
    "icon": "https://www.deutsche-digitale-bibliothek.de/appStatic/images/favicon.ico"
  }
} ]

Added the label to be consistent with the normal, non-EntityFacts sameAs structure with id and label (although we don't have actual labels in the GND sameAs data yet). The same consistency with other fields would be nice for depiction too (but not that urgent, as we don't have depiction fields with the id and label structure, so maybe a separate format for depiction is OK).

I think the whole collection thing is not ideal. It has no id, making it a bit pointless to introduce a separate entity (it's a blank node when converted to triples). The icon is the useful part, and that's similar to the depiction, also an image.

Maybe a id/label/image structure could work both for depiction:

"depiction": [ {
  "id": "https://commons.wikimedia.org/wiki/File:MarkTwain.LOC.jpg?uselang=en",
  "label": "Mark Twain",
  "image": "https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg"
} ]

And for sameAs:

"sameAs": [ {
  "id": "https://www.deutsche-digitale-bibliothek.de/entity/118624822",
  "label": "Deutsche Digitale Bibliothek",
  "image": "https://www.deutsche-digitale-bibliothek.de/appStatic/images/favicon.ico"
} ]

Current implementation uses the EntityFacts JSON and inserts the data into the framed, compacted GND JSON. We will have to look into the context and add some things there to keep our JSON consumable as RDF (depiction and image if we go with the unified approach, a few more if we stay closer to the EntityFacts format).

[acka47]

Maybe a id/label/image structure could work both for depiction:

"depiction": [ {
 "id": "https://commons.wikimedia.org/wiki/File:MarkTwain.LOC.jpg?uselang=en",
 "label": "Mark Twain",
 "image": "https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg"
} ]

If we create a better structure from this, I would like to not to go off too far from the source. Maybe something like this?

{
  "depiction":[
    {
      "id":"https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg",
      "url":"https://commons.wikimedia.org/wiki/File:MarkTwain.LOC.jpg?uselang=en",
      "thumbnail":"https://commons.wikimedia.org/wiki/Special:FilePath/MarkTwain.LOC.jpg?width=270"
    }
  ]
}

And for sameAs:

"sameAs": [ {
  "id": "https://www.deutsche-digitale-bibliothek.de/entity/118624822",
  "label": "Deutsche Digitale Bibliothek",
  "image": "https://www.deutsche-digitale-bibliothek.de/appStatic/images/favicon.ico"
} ]

In this case, I'd rather keep the structure as the icon is for the collection and not the linked thing/entry. The abbreviation etc. is also useful for some and should be included. We should probably add ids for the collections, e.g.:

{
  "id":"https://www.deutsche-digitale-bibliothek.de/entity/118624822",
  "collection":{
    "id":"http://d-nb.info/gnd/1070828033",
    "abbr":"DDB",
    "name":"Deutsche Digitale Bibliothek",
    "publisher":"Deutsche Digitale Bibliothek",
    "icon":"https://www.deutsche-digitale-bibliothek.de/appStatic/images/favicon.ico"
  }
}