cross reference wikidata Q id

[keighrim]

New Feature Summary

Current implementation performs NEL to a dbpedia URI (@wricketts correct me if I'm wrong about this). It would be great if the app can also map the dbpedia entry to a wikidata entry (references by Q id). This issue is for further discussion of implementation of the feature.

@wricketts can you start by doing some research and compare two databases(?) and write a comment below about possibility and requirements of automatic mapping?

[wricketts]

@keighrim from some initial searching it seems the DBpedia dataset contains mappings between DBpedia URIs and Wikidata URIs via a owl:sameAs relation. It would be necessary to use SPARQL to query the DBpedia endpoint and get this mapping.

[owencking]

Having the Q id would be very desirable from the GBH perspective. Besides querying a DBpedia endpoint, another approach might be to query a Wikidata endpoint. Also, a simple-minded approach might be to periodically generate a static look-up table with mappings between DBpedia and Wikidata identifiers.

[keighrim]

Another problem we found is that DBPedia is a "done" project as of 2016 so nothing from wikidata side added since around 2016 can be mappable to DBPedia-based effort. Once this is wrapped up, we should find a better linking strategy/tool that directly utilizes wikidata as the backend.

[wricketts]

Something else to note is that a DBpedia entry can map to multiple Wikidata nodes.
So for example, the entry for Ukraine maps to 3 different Q ids, the last of which is actually a Polish village different from the country. I'm not sure how we would want to deal with these cases.

[keighrim]

I guess in the most naive way, we can throw all Q ids in to a list of "grounding" field in the NE annotation type. That is, what we currently have is

app-dbpedia-spotlight-wrapper/app.py

Line 63 in 45d8df2

anno.add_property('dbpedia_link', entities[e]['dbpedia_link'])

But we haven't really fixated the name to use for encoding grounding (NEL) information in the NE type, and the type definition at http://vocab.lappsgrid.org/NamedEntity.html is underspecified. So the most naive implementation I'm suggesting is having a field like grounding that is a standard JSON list of URIs (and list all DBPedia and Wikidata URIs).

A question we might need to answer sometime soon is how we want to upgrade the lapps at_types for additional field names we used in the CLAMS project, and whether we want to maintain lapps vocab under clams responsibility. @marcverhagen please let us know if you have thought about this.

[marcverhagen]

My idea was similar to the above, have some field with a URI or list URIs. At first I was thinking a field named dbpedia, but I think grounding is better. With that we would need some metadata to determine what the grounding is to.

At first I was thinking it would be be silly not to use and adapt the LAPPS vocabulary, and once we have revived vocab.lappsgrid.org that should be easy. But since maintaining one repository is better than maintaining two I am also starting to feel good about adding NamedEntity to the CLAMS vocabulary.

[keighrim]

I was testing the app and got into this problem

# sending this reqeust
$ clams source text:/data/wgbh/transcripts/cpb-aacip-507-zw18k75z4h.trn.json | curl localhost:5055 -d@-
# got the following error response 

{
  "metadata": {
    "mmif": "http://mmif.clams.ai/0.4.2"
  },
  "documents": [
    {
      "@type": "http://mmif.clams.ai/0.4.2/vocabulary/TextDocument",
      "properties": {
        "mime": "text",
        "id": "d1",
        "location": "file:///data/wgbh/transcripts/cpb-aacip-507-zw18k75z4h.trn.json"
      }
    }
  ],
  "views": [
    {
      "id": "v_0",
      "metadata": {
        "timestamp": "2023-06-29T14:37:48.268764",
        "app": "http://apps.clams.ai/dbpedia-spotlight-wrapper",
        "error": {
          "message": "<class 'SPARQLWrapper.SPARQLExceptions.QueryBadFormed'>: QueryBadFormed: A bad request has been sent to the endpoint: probably the SPARQL query is badly formed. \n\nResponse:\nb'Virtuoso 37000 Error SP030: SPARQL compiler, line 5: syntax error at \\'magazine\\' before \\')\\'\\n\\nSPARQL query:\\n#output-format:application/sparql-results+json\\n\\n                PREFIX : <http://dbpedia.org/resource/>\\n                SELECT DISTINCT ?wikidata_concept\\n                WHERE {dbr:Money_(magazine) owl:sameAs ?wikidata_concept\\n                  FILTER (regex(str(?wikidata_concept), \"www.wikidata.org\"))\\n                }\\n                LIMIT 10\\n            \\n'",
          "stackTrace": "  File \"/usr/local/lib/python3.8/site-packages/clams/restify/__init__.py\", line 146, in post\n    return self.json_to_response(self.cla.annotate(raw_data, **self.annotate_param_caster.cast(raw_params)))\n\t\n  File \"/usr/local/lib/python3.8/site-packages/clams/app/__init__.py\", line 109, in annotate\n    annotated = self._annotate(mmif, **runtime_params)\n\t\n  File \"/app/app.py\", line 104, in _annotate\n    entities = _get_ne_links(res_json)\n\t\n  File \"/app/app.py\", line 93, in _get_ne_links\n    grounding.extend(list(_get_qid(uri)))\n\t\n  File \"/app/app.py\", line 47, in _get_qid\n    res = sparql.query().convert()\n\t\n  File \"/usr/local/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py\", line 960, in query\n    return QueryResult(self._query())\n\t\n  File \"/usr/local/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py\", line 930, in _query\n    raise QueryBadFormed(e.read())\n"
        }
      },
      "annotations": []
    }
  ]

The stack trace after replacing escaped whitespaces,

"message": "
<class 'SPARQLWrapper.SPARQLExceptions.QueryBadFormed'>: QueryBadFormed: A bad request has been sent to the endpoint: probably the SPARQL query is badly formed. 

Response:
b'Virtuoso 37000 Error SP030: SPARQL compiler, line 5: syntax error at 'magazine' before ')'

SPARQL query:
#output-format:application/sparql-results+json

                PREFIX : <http://dbpedia.org/resource/>
                SELECT DISTINCT ?wikidata_concept
                WHERE {dbr:Money_(magazine) owl:sameAs ?wikidata_concept
                  FILTER (regex(str(?wikidata_concept), "www.wikidata.org"))
                }
                LIMIT 10
            
'"

"stackTrace": "
  File \"/usr/local/lib/python3.8/site-packages/clams/restify/__init__.py\", line 146, in post
    return self.json_to_response(self.cla.annotate(raw_data, **self.annotate_param_caster.cast(raw_params)))
	
  File \"/usr/local/lib/python3.8/site-packages/clams/app/__init__.py\", line 109, in annotate
    annotated = self._annotate(mmif, **runtime_params)
	
  File \"/app/app.py\", line 104, in _annotate
    entities = _get_ne_links(res_json)
	
  File \"/app/app.py\", line 93, in _get_ne_links
    grounding.extend(list(_get_qid(uri)))
	
  File \"/app/app.py\", line 47, in _get_qid
    res = sparql.query().convert()
	
  File \"/usr/local/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py\", line 960, in query
    return QueryResult(self._query())
	
  File \"/usr/local/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py\", line 930, in _query
    raise QueryBadFormed(e.read())
"

[wricketts]

@keighrim When debugging I found that URIs such as https://dbpedia.org/page/Money_(magazine) or https://dbpedia.org/page/Rock_Hill,_South_Carolina resulted in bad sparql queries because of the special characters in the entity names. I pushed code to escape special characters and the app can fully process that transcript now. But it's possible that unaccounted edge cases can still come up in other data.

[keighrim]

Looks like the SPARQL spec has a predefined character set for escaping. https://www.w3.org/TR/2013/REC-sparql11-query-20130321/#rPN_LOCAL (see PN_LOCAL_ESC row).

[wricketts]

Actually, disregard my previous comment. I made another fix and it should work without needing to escape characters.