push event performance issue

[tango-controls-bot]

When using push_change_event and push_archive_event there is a big difference in performances if using it with the "criteria managed by library" option (second argument of set_xxx_event) set to true or false.
In the first case there is a lot of CPU consuption in the push_xxx_event execution also when thresholds criteria for sending the event are not met. Furthermore there is a quite high variance in the execution time of push_xxx_event.
Finally in both cases there is an increase of CPU consuption in the push_xxx_event execution if there are client subscribed to the event.

Attached there is a document with some results on a real case and a simple test device server to reproduce.

Note: in the second case threshold are read from the DB only at startup of the server.

Reported by: scalamera

Original Ticket: tango-cs/bugs/803

[tango-controls-bot]

Hi Graziano,

Thank you very much for the report.
I had a look at the Tango core code and at your code.
What is clear is that the code parts described in your 2 different configurations from the pdf are not equivalent in terms of features.

In your configuration 2, you invoke push_archive_event() only when the "archive criterias" are met, which means that the management of Tango alarms is not done in the cases where push_archive_event is not called.
You might miss the events indicating the first time an alarm is triggered for instance.
Yes, this is something I learnt: push_archive_event() will send an event in any case if there is a change of quality factor, even if the archive criterias are not met (archive period, absolute and relative thresholds).

What I've seen and understood is the following:
In the configuration 1 (criteria managed by Tango library), the Tango synchronisation monitor is taken during the whole call to push_archive_event().
This might explain why it takes more time because you might need to wait for the monitor if you have some polling enabled or some other clients.
And this waiting period can be long if someone is reading/writing another attribute for instance or executing a command.
push_archive_event() will call Attribute::set_value() in any case, with some memory allocation (and memory freeing at the end), even when there is no event subscriber connected.
When there is no connected subscriber, the calls to push_archive_event() should be equivalent in configuration 1 and configuration 2.

In your pdf, some important information is missing, which is how often is the push_archive_event() method called in your configuration.
If it is called only one loop iteration over 10, since it doesn't have to take the monitor, it is perfectly normal this will be faster.
But be aware that you might miss some alarms in this case.

In any case, it seems the Tango core code could be optimized because it is not efficient to allocate some memory for nothing and release it immediately if there is no connected subscriber.

In your tests, were the subscribers running on the same host as the device server?

As I told you during the last Tango meeting, please note that Tango will continue to act as if there is still a subscriber until 10 minutes after the last subscriber disappeared. So the CPU consumption will be the same as when there is a subscriber during these 10 minutes.

I couldn't reproduce the 100% CPU consumption issue in configuration 1 with your code sample with Tango 9 on our machines up to now. This might be due to the fact that there are much less attributes in the PushEventTest example.

To be continued...
Reynald

Original comment by: bourtemb

[bourtemb]

This problem should be solved in Tango 9 LTS with PR #430

[bourtemb]

Should be solved with the combination of PR #430 and #458