Limit query result sizes in Knora

[benjamingeer]

Using Gravsearch's owlim:query-limit-results setting is not great, because if the limit is exceeded, it silently truncates the query result, which can result in strange errors or inconsistencies. It would be better to handle this in Knora.

• Limit the number of properties in a class definition.
• If a property has a lot of values, require retrieving them in pages.
• Limit query depth in Gravsearch.

Others?

[gfoo]

Limit the number of properties in a class definition.

in this ticket, are you talking about only the gravsearch queries content or also about ontologies?
(because we already have ontologies with many properties in one class)

[benjamingeer]

I’m talking about ontologies. If someone makes a class definition with cardinalities on 10000 different properties, we’re not going to be able to serve instances of that class. There has to be some kind of limit.

[mrivoal]

What would be the maximum number of different properties allowed for a resource?

[benjamingeer]

I don’t know, what you you think would be reasonable?

[mrivoal]

For example we already have a project with a resource class of 118 properties. I don't think this is really reasonable or good modeling practices, but we have to deal with the weakness of the current GUI (Salsah 1.5 resource viewer and advanced search features).

[benjamingeer]

Understood. How about a limit of 150 for now?

[mrivoal]

Yes, we could definitely live with that :)

But @loicjaouen had an idea/suggestion to help users and GUI deal with complex data models which are more in line with good modeling. It would imply a resource class defined in knora-base ontology has being a compound of another class. See #785.

And @gfoo also suggests to integrate it Knora API simple schema. It would further help to deal with complex data model from the front-end side. But I will let him speak for himself.

[benjamingeer]

If I remember correctly, the compound resource idea would just be an optimisation to make it more efficient to query a tree of resources (e.g. in Gravsearch). It wouldn't create the illusion that the "root" resource has all the properties of its component resources, so it wouldn't help with the limitations of extended search in Salsah 1.5.

And even if we created such an illusion, the problem would still remain: we have to limit the amount of data that the triplestore returns per request. This means we can't allow the user to query 1000 resources in a single request, regardless of whether we make the response look like 1000 resources or 1 resource. This is why Gravsearch has a page limit of 25 resources, and this is also why we will need to limit the query depth in Gravsearch queries.

[loicjaouen]

limiting a resource's number of properties can only increase the graph depth... there's no magic :(

Resource aggregation (or compound) could be a way to "page", by delegating to the client, who has the business knowledge, what it wants to unfold, and tell a generic gui that what is compounded should be presented in the same window as the main resource.

[benjamingeer]

limiting a resource's number of properties can only increase the graph depth... there's no magic :(

But generally you don't need to get a whole tree at once. You can get just one node, and then the user can request parent or child nodes if they're really interested. Or you can just get the IRIs and labels of all the resources in the tree (e.g. using the graph route), display those as a network diagram, and let the user click on different nodes to get more detail about them (which would make additional requests).

Resource aggregation (or compound) could be a way to "page"

Yes. The client would have to make multiple requests to get more and more of the tree.

[gfoo]

(just ideas)

By adding a kind of composition semantic between a root resource and its related resources viewed as a part of a composition (see model below), Knora could provide these functionnalities on a group of resources:

• manage last modification date
• get version history
• simplify the data access, instead of making several http v2/resources requests, we could have all data in one request (of course we could use gravsearch, but it is designed to search and not really to access data)
• (BONUS simplify the editing form in Salsah)

[benjamingeer]

simplify the data access, instead of making several http v2/resources requests, we could have all data in one request

We want the client to make several requests in order to limit the size of each response. That's what this issue is about.

[mrivoal]

limiting a resource's number of properties can only increase the graph depth... there's no magic :(

I agree with @loicjaouen.

I think this is the point where we should give up on Git and discuss the options on the table this face-to-face. The discussion should probably enroll @lrosenth.

Same thing for the composition suggestion (#785), which is another issue (even though related to this one). Let's discuss the options and involve @kilchenmann and @flavens because it has a lot to do with the front-end side and the way complex resources are displayed (the same editing form could be used for queries: FileMaker style).

[benjamingeer]

@mrivoal OK!

[benjamingeer]

@tobiasschweizer would like to implement the querying of compound resources in https://github.com/dhlab-basel/knora-api-js-lib.