Change event subscription blind to change events right after device server restart

[moved here from https://sourceforge.net/p/tango-cs/bugs/834/]

Background
If a device server is restarted, all connected event subscribers will try to reconnect to it after a timeout. Once the client has reconnected, it will receive the latest value of the attribute. So far, so good.

Expected behaviour:
After the connection has been reestablished, from there on the event subscriber should receive every attribute value change that occurs (e.g. a change event on the State attribute).

Actual behaviour:
After the event subscriber has reconnected to the device server, there is a time window in which the event subscriber will not be informed about attribute changes.
E.g. when subscribing to change events on the State attribute, any State changes won't be reported to the event subscribers during that time window. The length of the time window seems to depend on the polling period. A polling period of 10s on State makes the event channel blind to State changes for at least a couple of seconds.

In other words, the event subscriber gets unsynchronized with the device server. And it will remain unsynchronized for potentially a long time, until another attribute change on the server is prompted.

Steps to reproduce:
If the attribute is configured with a large polling period (e.g. the default 3000ms), the bug is easy and reliably to reproduce.

1. Tested with Tango 9.2.5a, OmniORB 4.2.1, libzmq v4.1.4, PyTango 4.2.1 on Ubuntu 16.04 [amd64]
2. start TangoTest device server ( (Mthreaded_impl: false; State polling enabled and set to 10s)

Ready to accept request

3. start attached event_bug_demo_client.py
   [this script connects to State change events and prints out the events]

State: UNKNOWN

4. Quit and restart TangoTest
   wait until the event subscriber has reestablished the connection

Error state: Event channel is not responding anymore, maybe the server or event system is down
State:  UNKNOWN

5. Now quickly execute attached event_bug_demo_switch_state.py
   [this will call SwitchStates on TangoTest, causing State to go from UNKNOWN to RUNNING]

If done quickly enough, the event_bug_demo_client.py script will stay stuck at UNKNOWN forever.

Please let me know if there is anything further I can provide to help analyzing this issue.

And here are the python test scripts mentioned above:

python_test_scripts.zip

Note: I've created the ticket here because it looks to me like a bug in the C++ Tango device server.

[bourtemb]

Thank you very much for re-creating the ticket here.
I could reproduce the behavior you are describing with TangoTest and a C++ client.
Many thanks for the detailed bug report.

[bourtemb]

It looks like this problem is due to the fact that the first time EventSupplier::detect_and_push_archive_event() and EventSupplier::detect_and_push_archive_event() methods are called, no event is sent unless the attribute reading is throwing an exception or the user is forcing event push by code.
I personally think that it would make sense to push an event in any case the first time these methods are called.
This could be done by commenting out lines 367,368 and 370 in server/eventsupplier.cpp and by commenting out lines 729, 730 and 732 in the same file in the current Tango 9 LTS version.

This fix would introduce another issue, though: when the first (and only the first) client would subscribe to an attribute with a stable value (no value change between the event subscription synchronous call and the first time detect_and_push_change|archive_event is called (first polling loop after subscription if attribute is polled) or the first time push_change|archive_event is called with a change detection done by the library), the client callback would be called twice even though the attribute value didn't change. The timestamp values in the events would be different though.
I personally think it would be useful to introduce this small fix even though it would generate the problem mentioned above. It's better to have eventually one additional event than missing some events.

I don't know yet how to fix the potential double-event issue I just mentioned above but this does not seem straight-forward.

[Ingvord]

@bourtemb , thanks a lot for very detailed analysis of the issue!!!