Performance Issue - Clingo in concurrent threads

[marycore]

Hello,
I am using Clingo 4.5.3 which is assumed to be considered as a stream reasoner. The idea is to feed the incremental ASP solver with the data that is continuously coming from a number of sensors. In order to model such system, I have implemented two concurrent threads in Java. The first thread reads the database table at each second to fetch the newly arrived data and share it with the second thread which has initialized clingo and is waiting to feed it with the data. This model works fine only during the first two minutes after running the threads. The second thread starts to get slower after about 2 minutes (~ timestampe 200) and thus causes considerable delays in reading and parsing the sensor data in real time.
However, when I run the solver alone and feed it manually (not through threads) it quickly solves the program with the given data and without getting slower even at timestamps larger than 200.

I appreciate if you let me know how to deal with the performance issue, or any hints about modeling or initializing the solver.
Best

[rkaminsk]

Hi, I do not know much about stream reasoning but can tell you what the limitations of the clingo system are.

• Continuously adding new facts to a program and then ground and solve will certainly cause slow downs/failure at some point because the system is accumulating more and more memory.
• What can be done is to write an encoding that takes a fixed set of inputs (*). The inputs can then be assigned before solving. This approach will not consume unbounded amounts of memory but is harder to implement. Also if the domains of your inputs are too large the approach might be infeasible.
• If you can solve you problem in reasonable time when solving each problem from scratch, why bother with anything more complicated (except as a research topic).

(*) It is also possible to extend the inputs dynamically but this is getting complicated.

[pobermei]

Hi,
As Roland pointed out, continuously adding new program increments for new time steps won't scale. We noticed this problem too and to address it, we introduced in this paper a technique to implement a time-based sliding window. The primary idea is as follows:

• The discrete time steps are arranged in a cycle (modulo-ring) which loops after a fixed number of steps.
• The problem is encoded with rules containing only regular facts and a fixed horizon = the modulo-ring size
• At each time point t, arriving streamed data represented as external facts are continuously assigned (via equivalence) to corresponding regular facts for timestamp t modulo window-length.

With that, only the assignment rules have to be continuously grounded and added as new modules but not the whole problem encoding. Naturally, this won't scale ad infinitum either but should provide a notable improvement to your original approach.

[marycore]

Pobermei and rkaminsk, thanks a lot for your answer.
I read the paper and understood the sliding window approach. Given streams of sensor data (as external predicates), the clingo solver (plus a python scripts handling the incremental behavior) in my application is used as a stream reasoner in order to recognize activities in a smart home. It means that for some activities the solver needs to keep track of some events (as the precondition of other upcoming events) for a long period of time e.g., 4 hours equivalent to timestamp t = 4*3600 (the resolution of timestamp is in second). I think for such situations, the sliding window solution doesn't seem effective. Right?

My second question: Is it possible to remove some obsolete predicates from the grounding set of the solver during the solving process?

Best

[rkaminsk]

It means that for some activities the solver needs to keep track of some events (as the precondition of other upcoming events) for a long period of time e.g., 4 hours equivalent to timestamp t = 4*3600 (the resolution of timestamp is in second). I think for such situations, the sliding window solution doesn't seem effective. Right?

Representing so many time points with atoms will certainly not scale. You might want to have a look into clingcon, which can handle variables over (finite) numbers. But note that it does not support multi-shot solving yet.

My second question: Is it possible to remove some obsolete predicates from the grounding set of the solver during the solving process?

To remove something from a logic program, externals can be used. Releasing a. in

#external a.
b :- a.

will remove a and b from the program. By calling Control.cleanup(), both atoms will also be removed from clingo's domains and not used for grounding in later steps. Actually, anything the solver determined to be false is removed. But note that this does not free all memory associated with the atoms but will keep the symbolic representation in memory. I thought about also collecting such kind of memory but this is difficult to implement and won't happen anytime soon. The solving backend will also keep a small amount of memory because even though all constraints over a variable will be removed the variable as such is not.