Persistent Nemo engine with preloaded graph and runtime queries?

[nicola883]

Hi, we are evaluating Nemo for querying a large biomedical graph that changes only periodically.

Can Nemo preload the graph once into a long-lived engine and then accept runtime queries without reloading it each time?

Ideally, we would like to send a query that includes Datalog rules plus parameter facts.
If that is not supported, can Nemo at least preload the graph and rules once, and then accept only parameter facts at runtime?

If not, what is the closest supported pattern today?

[mmarx]

Neither of this is, unfortunately, not currently supported. While proper support for answering arbitrary queries over a loaded data set will take quite a bit of work (there is substantial overlap with #735, e.g.) we might be able to address your use case with something much simpler: perhaps it would suffice if the engine could be restarted with a completely different program, but would reuse already imported data for @import directives?

How would you use Nemo – as a standalone client, or through one of the APIs (Rust/Python/Javascript)? Do you have any rules you always apply over the graph before answering queries, or do you just load the graph (in which format?) and then want to answer queries using varying rules?

[nicola883]

We would use Nemo through the Rust API, but we are flexible if another integration works better.
Yes, we can have predefined rules. In a future version we would like to provide rules at query time, but for now fixed rules are fine.
The graph format can be anything supported by Nemo (we can adapt).
However, at query time, we would need to provide parameters, e.g.:
?- path("start", "end").

[n9]

I have a similar use case: reasoning over a large static external list of [subject, predicate, object, statementId], where object and statementId are optional.

[n9]

I think it might also need to (somehow) cache the results of transitivity rules, such as the ones mentioned in #763 (comment)?

[mmarx]

Are those rules always the same, or do they vary with the queries?

[n9]

Those rule are always the same.

[mmarx]

Then, at least as a workaround, you could load the graph, materialise the transitivity rules, export the resulting graph, and query over that one, could you not?

[n9]

Yes.

[mmarx]

perhaps it would suffice if the engine could be restarted with a completely different program, but would reuse already imported data for @import directives?

I've created #784 for this.

[nicola883]

Yes, of course. Is it possible?

[mmarx]

We'll try to have this in the next release, yes, hopefully somewhere in the next few weeks.

[nicola883]

That is great. The goal is not to spend time loading the graph (that can be minutes) for each query. If the graph is already in memory and Nemo can use it, perfect!