Provide mechanism to get info on total number of results

[acka47]

Currently, API users don't get any information of the total number of results a specific query yields. This is no problem if the number of hits is < 50 as all results show up in one document. If the number of hits is > 50 one would have to page through all results and add all results together to get to know the total number (which would add unnecessary API load.
Thus, the total number of hits should be delivered by default with with every query response or users should have the possibility to somehow ask for the total number of hits.

[acka47]

As said in #230 (comment) I would look into using the Sindice Search Vocabulary for providing information on a query and its results with total number of results being one part of this information.

[literarymachine]

Bump - when using the API to discover link targets for workflows such as @edoweb's, this is a really important feature.

[fsteeg]

+1, we also need this for NWBib.

Technically this is no problem at all, the only issue is how to serve it as RDF.

How do we make statements about the query? Can we simply add a generic statement like:

<http://sindice.com/vocab/search#Query> <http://sindice.com/vocab/search#totalResults> 42 .

Or do we need to include the actual query in some way?

[literarymachine]

I would argue that the query itself can be referenced by it's URL, e.g. http://api.lobid.org/resource?name=Faust. So maybe this would be cleaner?

<http://api.lobid.org/resource?name=Faust> <http://sindice.com/vocab/search#totalResults> 42 .

Also see #230 (comment)

[acka47]

The first thing we should do before implementing this is discontinuing support for other RDF serializations than JSON-LD (and RDFa) in query results (as @jschnasse already proposed some time ago).

The results of curl -H "Accept: text/turtle" http://lobid.org/resource?name=Faust are of no use anyway, because they don't deliver information on ranking. JSON-LD already is the default, see curl http://lobid.org/resource?name=Faust and it already provides ranking information through delivering an array. Conversely, this means that content negotiation for other RDF serializations (turtle, N-Triples) should only be supported for querying resource URIs like http://lobid.org/resource/HT014834500. (We could differentiate these two options (conneg for RDF vs. no conneg for RDF) by the URL schemes http://api.lobid.org vs. http://lobid.org.)

This said, here's my first take on how to provide the search results in JSON-LD:

{
    "@context": { 
        "search": "http://sindice.com/vocab/search#",
        "issued": {
            "@id": "http://purl.org/dc/terms/issued",
            "@type": "xsd:dateTime"
            }
    },
    "@type": "search:Query",
    "@id": "http://api.lobid.org/resource?name=Faust",
    "issued": "2014-05-15T09:34:51",
    "search:first": "http://api.lobid.org/resource?name=Faust",
    "search:last": "http://api.lobid.org/resource/$lastResultPage",
    "search:next": "http://api.lobid.org/resource/$secondResultPage",
    "search:itemsPerPage": "50",
    "search:totalResults": "4273",
    "search:result": [
        { "@id": "$FirstSearchResultInNestedJSON-LD" },
        { "@id": "$FirstSearchResultInNestedJSON-LD" },
        { "further": "results" }
    ]
}

(This example applies for the ``?format=fullquery. For?format=id` etc. the result would be adjusted accordingly.)

[literarymachine]

The first thing we should do before implementing this is discontinuing support for other RDF serializations than JSON-LD (and RDFa) in query results (as @jschnasse already proposed some time ago).

The first thing you should do is ask if someone uses other RDF serializations. Second of all - if search results are valid JSON-LD, what stops you from providing other serializations?

<http://api.lobid.org/resource?name=Faust>
  a search:Query ;
  dc:issued "2014-05-15T09:34:51" ;
  search:first <http://api.lobid.org/resource?name=Faust> ;
  search:last <http://api.lobid.org/resource/$lastResultPage> ;
  search:next <http://api.lobid.org/resource/$secondResultPage> ;
  search:itemsPerPage "50" ;
  search:totalResults "4273" ;
  search:result (
    <http://lobid.org/resource/HT014009200>
    <http://lobid.org/resource/HT013815600>
    <http://lobid.org/resource/CT003041072>
    ...
  ) .

Yes, this uses RDF-lists, which are not the prettiest thing in the world. But to make the JSON-LD not only implicitly (based on JSON-syntax that is) deliver the proper result order, they would have to be modeled as a list anyways.

[acka47]

@literarymachine Do you use the RDF serializations for lobid search results? :)

[literarymachine]

Yes, I currently use ntriples.

[fsteeg]

Implementing the changes suggested by @acka47 would imply an incompatible 2.0 API release, since the basic response structure would be completely different: we'd be returning a JSON object as the top level element, instead of an array.

The approach that I started implementing yesterday is to add a new entity for the query itself as the first entity in the response. In JSON-LD, this would look like this:

[{
  @id: "http://api.lobid.org/resource?name=Faust&format=full",
  http://sindice.com/vocab/search#totalResults: 4218
},{
  @graph:
  ...
}]

Removing the RDF search results has the same issue: we'd break API, we can only do this in a 2.0 release. The approach above would e.g. yield this if N-Triples are requested:

curl -L --header "Accept: text/plain" "http://localhost:9000/resource?name=Faust"

<http://api.lobid.org/resource?name=Faust> <http://sindice.com/vocab/search#totalResults> "4218"^^<http://www.w3.org/2001/XMLSchema#integer> .
<http://d-nb.info/gnd/118524461> <http://d-nb.info/standards/elementset/gnd#preferredNameForThePerson> "Delacroix, Eug\u00E8ne"^^<http://www.w3.org/2001/XMLSchema#string> .
...

[acka47]

As interim solution the proposal by @fsteeg would be ok with me. Would this also be ok for you, @literarymachine, for the time being?

[literarymachine]

As interim solution the proposal by @fsteeg would be ok with me. Would this also be ok for you, @literarymachine, for the time being?

+1 (see #297 (comment))

We should probably discuss any further (and incompatible) modifications of the result format somewhere else. Just a short addition to #297 (comment): marking the search:result in the JSON-LD as an ordered list can be done from within the context:

"@context": {
    "search": "http://sindice.com/vocab/search#",
    "issued": {
        "@id": "http://purl.org/dc/terms/issued",
        "@type": "xsd:dateTime"
    },
    "search:result" {
        "@container": "@list"
    }
}

[fsteeg]

Deployed for testing:

http://test.lobid.org/resource?name=Typee&format=full&size=1

curl -L --header "Accept: text/plain" "http://test.lobid.org/resource?name=Typee&size=1"

[literarymachine]

+1

[fsteeg]

Deployed to production, closing:

http://lobid.org/resource?name=Typee&format=full&size=1

curl -L --header "Accept: text/plain" "http://lobid.org/resource?name=Typee&size=1"

[fsteeg]

Reopening, does not work correctly for RDF+XML:

curl -L -H "Accept: application/rdf+xml" "http://test.lobid.org/resource/HT009442672"

Has two root elements, should have only one. See possibly similar issue in hbz/oerworldmap.

[fsteeg]

Deployed fix to staging (don't add query info on path-style requests for single resources):

http://test.lobid.org/resource?name=Typee&format=full&size=1
curl -L -H "Accept: text/plain" "http://test.lobid.org/resource?name=Typee&size=1"
curl -L -H "Accept: application/rdf+xml" "http://test.lobid.org/resource/HT009442672"

[fsteeg]

Deployed to production, closing.

Opened new issue for RDF+XML results for multiple hits with one root: #463 (independent from the topic of this issue, did already return multiple root elements before total number of results were added).