Lost connection in longPoll

[anusoft]

This error sometimes occur every 10 minutes, but sometime connection last many hours.

Traceback (most recent call last):
File "echobot.py", line 30, in
for op in client.longPoll():
File "/usr/local/lib/python2.7/dist-packages/line-0.1.8-py2.7.egg/line/client.py", line 379, in longPoll
operations = self._fetchOperations(self.revision, count)
File "/usr/local/lib/python2.7/dist-packages/line-0.1.8-py2.7.egg/line/api.py", line 255, in _fetchOperations
return self._client.fetchOperations(revision, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 482, in fetchOperations
self.send_fetchOperations(localRev, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 492, in send_fetchOperations
self._oprot.trans.flush()
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 106, in _f
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 139, in flush
self.__http.endheaders()
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

[carpedm20]

Actually, This is not a new issue because some people told about this but I couldn't find the answer for this issue yet..

[carpedm20]

b5397aa, problem solved.

[anusoft]

I've still found issue after upgraded:

Traceback (most recent call last):
File "echobot.py", line 34, in
for op in client.longPoll():
File "/usr/local/lib/python2.7/dist-packages/line-0.4.1-py2.7.egg/line/client.py", line 504, in longPoll
operations = self._fetchOperations(self.revision, count)
File "/usr/local/lib/python2.7/dist-packages/line-0.4.1-py2.7.egg/line/api.py", line 308, in _fetchOperations
return self._client.fetchOperations(revision, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 482, in fetchOperations
self.send_fetchOperations(localRev, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 492, in send_fetchOperations
self._oprot.trans.flush()
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 106, in _f
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 139, in flush
self.__http.endheaders()
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

[carpedm20]

I think this is not the problem of expiration of authToken. I guess Name or service not known error can be a problem of internet. How about making a exception around the longPoll and bypass this error?

[anusoft]

There're other few error, I'll try to collect here and make exceptions for retrying.

But there're sometimes bots just freeze without disconnect and not responding, that is harder to fix and I think there should be some NOOP or PING to probe if connection is alive too.

[anusoft]

I attempted to reconnect when socket error but still not working, any advise?

        if self._check_auth():
            """Check is there any operations from LINE server"""
            OT = OperationType

            try:
                operations = self._fetchOperations(self.revision, count)
            except SocketError as serr:
                print "Socket Error serr.errno %s" % serr.errno

                self.login()
                self.ready()

When I tried to call self.login(), self.read() it's show valid authToken but not able to continue

LoginResult(authToken='XXX, verifier=None, type=1, pinCode=None, certificate=None)
'XXX'
[*] DUMMY
Operation(status=None, checkSum=None, param3=None, param2=None, param1=None, createdTime=1427958741545, message=None, reqSeq=0, type=48, revision=747022)

[carpedm20]

I think you'd better use

client.updateAuthToken()

when the exception is occurred. It is a method that refresh authToken without PinCode authentication.

[anusoft]

When the connection freezes, I've tried to check with netstats found that the connection is still alive, but no data receiving

# netstat -anp | grep 13114
tcp        1      0 172.16.1.1:59796        203.104.142.1:80        CLOSE_WAIT  13114/python    
tcp        0      0 172.16.1.1:60061        203.104.142.1:80        ESTABLISHED 13114/python    

# strace -p 13114
Process 13114 attached
recvfrom(4, 

What method do you suggest for periodically check if connection is alive?

Edited: I tried to create work around for freezing bots by getProfile every minute.

t1 = time.time()
t2 = time.time()

while True:
    op_list = []

    if  (time.time() - t1) >= 60:
        print "Idle 60 Second"
        t1 = time.time()
        client.getProfile()

    if  (time.time() - t2) >= 7000:
        print "Idle 7000 Second"
        t2 = time.time()
        client.updateAuthToken()

Edited 2:

self.login() + self.ready() won't work wells when socket error, it will cause more socket error or another error. I decided to quit the program then rerun it again when socket error using shell

while [ true ]; do python echobot.py; sleep 10; done

Also getProfile() every minute and updateAuthToken() every almost two hours seem does the job well, I can run bots continuously more than 24 hours now without getting any socket error.

Edited 3:

It still freeze from time to times so I'm using threading to check if it's still response.

import os
import threading

def DEAD():
    print 'DEAD'
    os._exit(1)    

tt = threading.Timer(65.0, DEAD)
tt.start()

while True:
    op_list = []

    if  (time.time() - t1) >= 60:
        print time.strftime("%Y-%m-%d %H:%M:%S"), "Idle 60 Second"
        t1 = time.time()
        client.getProfile()

        tt.cancel()
        tt = threading.Timer(65.0, DEAD)
        tt.start()

[anusoft]

#25 seem to be also fixed this.