Lightstreamer stops after 2 hours

[sacharbit]

Hello everyone,

Hope you're all ok in those tough times,

I have an issue with the lightstreamer stream api : it works fine for about 2 hours and then stops.
I have found online that it's a problem with the LOOP that "finishes its natural cycle" so I need to reconnect and subscribe to all the feeds that I had but I can't find a way to detect that "end of cycle".

I tried to log something everytime I receive a loop message but I don't log anything.

Please help! It drives me crazy! :)

Thanks in advance,
Sacha

[sacharbit]

I've made a workaround :
In the IGStreamService class, I've added a new thread that checks every few seconds if the data has been updated. If yes, nice, if not, disconnect and reconnect.

[oliverpolden]

In my opinion this shouldn't be a concern of the "end user" or someone using the library. The library should handle this. Thanks for your workaround @sacharbit could be a good solution until this is implemented in the library. It appears that other libraries handle a "LOOP" message.

There is useful thread here: https://labs.ig.com/node/162 that talks about using the HEARTBEAT epic which apparently keeps your session from expiring or whatever the LOOP message means has happened. This could be another workaround. I've successfully got a HEARTBEAT subscription working and I'm currently running it with a debug log level. If I don't report back here then you can probably assume that the HEARTBEAT subscription solves the issue for me.

Back to the library though, which is where I think the LOOP should be handled. I'll raise an issue to discuss if it's already handled somehow and I can have a go at doing what needs to be done when a LOOP message occurs.

For those wanting to implement a HEARTBEAT workaround, I did it like so, just under my other subscriptions:

        heartbeat_items = ["TRADE:HB.U.HEARTBEAT.IP"]
        heartbeat = Subscription(
            mode='MERGE',
            items=heartbeat_items,
            fields=["HEARTBEAT"],
        )

        heartbeat.addlistener(self.on_heartbeat_update)
        sub_heartbeat = ig_stream_service.ls_client.subscribe(heartbeat)

Then I have an "on_update" method that really does nothing apart from print so I know it's doing something:

    def on_heartbeat_update(self, item_update):
        print(item_update["values"])

Just for completeness (which I sometimes find lacking in the Python community), if you need debug level log, then add this:

    import logging
    logging.basicConfig(level=logging.DEBUG, format='%(message)s')

[oliverpolden]

Well that heartbeat didn't solve it for me.

[oliverpolden]

So I made sure to be up to date with what's in Master and ran my stream in debug mode. I expected it to fail but by the time I went to bed it was still running and it was even still running in the morning. I looked back through the log and could see it rebinding successfully. I've updated everything on my server and am now running it with logging turned on. Hopefully it's just to ensure you're running the latest code to fix this issue.

[oliverpolden]

Nope, got a LOOP with no rebind.

[sacharbit]

woaw! That was a rollercoaster of emotions :D

[sacharbit]

Make sure that in lightstreamer file in line around 350, you have rebind=True and receive=False
because for me it works!

[RRMXkun]

@oliverpolden Have you made progress on handling LOOP in the library? If not I will give it a try; been facing that same issue and I'm tired of it! But wouldn't want to start from scratch if you've already got uncommitted code.

-Edit- @sacharbit There is a rebind variable indeed. It looks like there is some kind of handling of LOOP commands already, have you been able to verify that it definitely works?

[oliverpolden]

No I’m afraid not.

…

On 5 Jul 2020, at 13:52, RRMXkun ***@***.***> wrote:  @oliverpolden Have you made progress on handling LOOP in the library? If not I will give it a try; been facing that same issue and I'm tired of it! But wouldn't want to start from scratch if you've already got uncommitted code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[sacharbit]

I've made a pull request. Basically, you have to add the version and LS_CONTENT_LENGTH.
My pull request : #127

[bug-or-feature]

Pull request has been merged in

[chis86]

Hi, I'm seeing a disconnect after about 6 hours, and experienced this every time I've subscribed. I'm curious about the setting Message = None, and then stopping the While Loop by setting receive = False, when I received a Timeout. Here is my stack trace. Is there a way to handle the connection drops for this timeout error? Would adding rebind = True in the first If Message is None cause issues?
Thanks

Traceback (most recent call last):
File "C:\Python38\lib\site-packages\trading_ig\lightstreamer.py", line 333, in _receive
message = self._read_from_stream()
File "C:\Python38\lib\site-packages\trading_ig\lightstreamer.py", line 170, in _read_from_stream
line = self._stream_connection.readline().decode("utf-8").rstrip()
File "C:\Python38\lib\http\client.py", line 659, in readline
result = self.fp.readline(limit)
File "C:\Python38\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "C:\Python38\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "C:\Python38\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

[RRMXkun]

@chis86

Would adding rebind = True in the first If Message is None cause issues?

Yes it would, the connection is still active at this point we just received an empty message, so we wouldn't want to rebind at this stage. By the time you receive the error you're seeing we're already out of the loop as en exception has been raised by this part of the code:

try:
    message = self._read_from_stream()
    log.debug("Received message ---> <{0}>".format(message))
except Exception:
    log.error("Communication error")
    print(traceback.format_exc())
    message = None

Therefore you will never reach the if message=None part.

The only way to address this that I can see is to make use of the Heartbeat:

heartbeat = Subscription(mode='MERGE', items=["TRADE:HB.U.HEARTBEAT.IP"], fields=["HEARTBEAT"])
    heartbeat.addlistener(heartbeat_update)
    sub_heartbeat = ig_stream_service.ls_client.subscribe(heartbeat)

and re-establish a connection if the heartbeat stops. We could try to implement this in the LSClient class of lightstreamer.py.