Connection lost on Linux server #103

dgambin · 2020-03-20T08:32:01Z

Hi,

I am getting connect/disconnect events all the time on my Linux (debian) server. The same script works ok on my local machine (mac). The pattern is always the same, like:

CONNECTED->SUBSCRIBED->CONN CLOSE NORMALLY/DISONNECT after exactly 4 minutes->CONNECTED->CONN CLOSE NORMALLY/DISONNECT after exactly 2 minutes. Then it repeats every 2 minutes many times, then again 4 minutes/2 minutes.

I don't find anything in my server logs and there is only "503: mqtt session connection was closed (Unexpected disconnect)" message logged on flespi platform.

Any clue on what this could be? Probably some server/network related stuff but don't know how to identify it
I am attaching the screenshot from my terminal.

Thx on any help that could help me resolve this.

regards, dejan

Lenka42 · 2020-03-20T09:36:25Z

@dgambin could you please set debug-level logging?
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s')

dgambin · 2020-03-20T09:55:33Z

@Lenka42 debug log attached
debug.log

Lenka42 · 2020-03-25T12:01:49Z

From both broker and client sides it looks like socket was closed unexpectedly :( I guess more research on system and network settings is needed

ijl20 · 2020-04-01T15:45:07Z

I see the same or similar issue - in my case a connection to a mostly default (apart from logging) configured localhost Ubuntu mosquitto broker has timeout disconnects resulting in missed messages. It looks to me (logs below) like the gmqtt client sends PINGREQ messages every (default) 60 seconds, which maintains the connection to the mosquitto broker fine. But when the subscription receives a message it then (usually, I think) stops sending the PINGREQs, the mosquitto broker drops the client (by closing the socket) after 1.5 x the keepalive period, the gmqtt client immediately wakes up on the disconnect and reconnects and resubscribes (taking ~5 seconds to reconnect and missing any messages during that period). When the gmqtt client reconnects, it starts sending PINGREQs again. In this non-PINGREQ period the gmqtt client subscription continues to work fine, i.e. I can publish more messages and they will all be received. But the mqtt client will still be kicked off the mosquitto broker after 1.5 x the keepalive and have to reconnect.

Reproducible with the 'simple example' subscriber program given on the homepage README of this repo. I modified this provided gmqtt example simply to include timestamps and to connect to my default localhost mosquitto broker in my github here.

I published simple test messages on topic TEST/user about every 10 seconds using mosquitto_pub, giving the seconds timestamp as the message (seen as the mosqpub client connections in the mosquitto log).

I ran mosquitto_sub on the same host (seen as the mosqsub client), listing all topics/messages, giving:
Local subscription to mosquitto: me@my-machine:~$ mosquitto_sub -v -t '#'

TEST/user 1585751829
TEST/user 1585751840
TEST/user 1585751851
TEST/user 1585751861
TEST/user 1585751871

The example program subscribes to TEST/# and prints out these messages.
Running the 'example program': me@my-machine:~$ python3 test_timeout.py

1585751818.500474 RECV MSG: b'1585751818'
1585751829.868452 RECV MSG: b'1585751829'
1585751840.392649 RECV MSG: b'1585751840'
1585751851.807200 RECV MSG: b'1585751851'
1585751859.924842 Disconnected
[TRYING WRITE TO CLOSED SOCKET]
1585751865.934178 Connected
1585751865.934926 SUBSCRIBED
1585751871.934515 RECV MSG: b'1585751871'

The example program output above shows it missing the 1585751861 message.

The mosquitto broker drops the GMQTT client at 1585751859, as shown in the
mosquitto broker log. Note the example program is client 'client-id', and also there is mosqpub and mosqsub to send messages and subscribe respectively.

me@my-machine:~$ tail -f /var/log/mosquitto/mosquitto.log

1585751769: New client connected from 127.0.0.1 as client-id (c0, k60, u'foo').
1585751769: Sending CONNACK to client-id (1, 0)
1585751769: Received SUBSCRIBE from client-id
1585751769:     TEST/# (QoS 0)
1585751769: client-id 0 TEST/#
1585751769: Sending SUBACK to client-id
1585751796: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751796: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751818: New connection from 127.0.0.1 on port 1883.
1585751818: New client connected from 127.0.0.1 as mosqpub|5983-ijl20-tosh (c1, k60).
1585751818: Sending CONNACK to mosqpub|5983-ijl20-tosh (0, 0)
1585751818: Received PUBLISH from mosqpub|5983-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Received DISCONNECT from mosqpub|5983-ijl20-tosh
1585751818: Client mosqpub|5983-ijl20-tosh disconnected.
1585751829: New connection from 127.0.0.1 on port 1883.
1585751829: New client connected from 127.0.0.1 as mosqpub|5993-ijl20-tosh (c1, k60).
1585751829: Sending CONNACK to mosqpub|5993-ijl20-tosh (0, 0)
1585751829: Received PUBLISH from mosqpub|5993-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Received DISCONNECT from mosqpub|5993-ijl20-tosh
1585751829: Client mosqpub|5993-ijl20-tosh disconnected.
1585751840: New connection from 127.0.0.1 on port 1883.
1585751840: New client connected from 127.0.0.1 as mosqpub|6002-ijl20-tosh (c1, k60).
1585751840: Sending CONNACK to mosqpub|6002-ijl20-tosh (0, 0)
1585751840: Received PUBLISH from mosqpub|6002-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Received DISCONNECT from mosqpub|6002-ijl20-tosh
1585751840: Client mosqpub|6002-ijl20-tosh disconnected.
1585751851: New connection from 127.0.0.1 on port 1883.
1585751851: New client connected from 127.0.0.1 as mosqpub|6009-ijl20-tosh (c1, k60).
1585751851: Sending CONNACK to mosqpub|6009-ijl20-tosh (0, 0)
1585751851: Received PUBLISH from mosqpub|6009-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Received DISCONNECT from mosqpub|6009-ijl20-tosh
1585751851: Client mosqpub|6009-ijl20-tosh disconnected.
1585751856: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751856: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751859: Client client-id has exceeded timeout, disconnecting.
1585751859: Socket error on client client-id, disconnecting.
1585751861: New connection from 127.0.0.1 on port 1883.
1585751861: New client connected from 127.0.0.1 as mosqpub|6022-ijl20-tosh (c1, k60).
1585751861: Sending CONNACK to mosqpub|6022-ijl20-tosh (0, 0)
1585751861: Received PUBLISH from mosqpub|6022-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751861: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751861: Received DISCONNECT from mosqpub|6022-ijl20-tosh
1585751861: Client mosqpub|6022-ijl20-tosh disconnected.
1585751865: New connection from 127.0.0.1 on port 1883.
1585751865: Client client-id disconnected.
1585751865: New client connected from 127.0.0.1 as client-id (c0, k60, u'foo').
1585751865: Sending CONNACK to client-id (1, 0)
1585751865: Received SUBSCRIBE from client-id
1585751865:     TEST/# (QoS 0)
1585751865: client-id 0 TEST/#
1585751865: Sending SUBACK to client-id
1585751871: New connection from 127.0.0.1 on port 1883.
1585751871: New client connected from 127.0.0.1 as mosqpub|6034-ijl20-tosh (c1, k60).
1585751871: Sending CONNACK to mosqpub|6034-ijl20-tosh (0, 0)
1585751871: Received PUBLISH from mosqpub|6034-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Received DISCONNECT from mosqpub|6034-ijl20-tosh
1585751871: Client mosqpub|6034-ijl20-tosh disconnected.
1585751915: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751915: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751925: Received PINGREQ from client-id
1585751925: Sending PINGRESP to client-id
1585751976: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751976: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751985: Received PINGREQ from client-id
1585751985: Sending PINGRESP to client-id

Another run with logging set to debug, this time with keepalive=20:

(venv) me@my-machine:~/src/acp_mqtt_decoders$ python3 test_timeout.py
2020-04-01 17:04:40,044 [CONNECTION MADE]
2020-04-01 17:04:40,045 [CMD 0x20] b'\x00\x00'
2020-04-01 17:04:40,045 [CONNACK] flags: 0x0, result: 0x0
1585757080.045449 Connected
2020-04-01 17:04:40,045 NEW ID: 1
2020-04-01 17:04:40,045 [SEND SUB] 1 ['TEST/#']
2020-04-01 17:04:40,045 [QoS query IS EMPTY]
2020-04-01 17:04:40,045 Sending PUBLISH (q1), 'TIME', ... (18 bytes)
2020-04-01 17:04:40,045 NEW ID: 2
2020-04-01 17:04:40,046 [CMD 0x90] b'\x00\x01\x00'
2020-04-01 17:04:40,046 [SUBACK] 1 (0,)
1585757080.046218 SUBSCRIBED
2020-04-01 17:04:40,046 FREE MID: 1
2020-04-01 17:04:40,046 [CMD 0x40] b'\x00\x02'
2020-04-01 17:04:40,046 [RECEIVED PUBACK FOR] 2
2020-04-01 17:04:40,046 FREE MID: 2
2020-04-01 17:04:40,046 [REMOVE MESSAGE] 2
2020-04-01 17:04:45,051 [QoS query IS EMPTY]
2020-04-01 17:04:50,052 [QoS query IS EMPTY]
2020-04-01 17:04:55,056 [QoS query IS EMPTY]
2020-04-01 17:05:00,056 [CMD 0xd0] b''
2020-04-01 17:05:00,057 [PONG REQUEST] 0xd0 b''
2020-04-01 17:05:00,059 [QoS query IS EMPTY]
2020-04-01 17:05:05,065 [QoS query IS EMPTY]
2020-04-01 17:05:07,567 [CMD 0x30] b'\x00\tTEST/user1585757107'
2020-04-01 17:05:07,567 [RECV TEST/user with QoS: 0] b'1585757107'
1585757107.568310 RECV MSG: b'1585757107'
2020-04-01 17:05:07,568 FREE MID: None
2020-04-01 17:05:10,066 [QoS query IS EMPTY]
2020-04-01 17:05:15,072 [QoS query IS EMPTY]
2020-04-01 17:05:20,073 [QoS query IS EMPTY]
2020-04-01 17:05:25,079 [QoS query IS EMPTY]
2020-04-01 17:05:29,939 [RECV EMPTY] Connection will be reset automatically.
2020-04-01 17:05:29,940 [CONN CLOSE NORMALLY]
2020-04-01 17:05:29,940 [CMD 0xe0] b''
1585757129.941526 Disconnected
2020-04-01 17:05:29,942 [TRYING WRITE TO CLOSED SOCKET]
2020-04-01 17:05:35,949 [CONNECTION MADE]
2020-04-01 17:05:35,951 [CMD 0x20] b'\x00\x00'
2020-04-01 17:05:35,951 [CONNACK] flags: 0x0, result: 0x0
1585757135.952271 Connected
2020-04-01 17:05:35,952 NEW ID: 3
2020-04-01 17:05:35,952 [SEND SUB] 3 ['TEST/#']
2020-04-01 17:05:35,953 [QoS query IS EMPTY]
2020-04-01 17:05:35,953 [CMD 0x90] b'\x00\x03\x00'
2020-04-01 17:05:35,954 [SUBACK] 3 (0,)
1585757135.954416 SUBSCRIBED
2020-04-01 17:05:35,954 FREE MID: 3
2020-04-01 17:05:40,960 [QoS query IS EMPTY]
2020-04-01 17:05:45,961 [QoS query IS EMPTY]
^C2020-04-01 17:05:47,423 [CONN CLOSE NORMALLY]
2020-04-01 17:05:47,424 [CMD 0xe0] b''
1585757147.424661 Disconnected

For which the mosquitto broker log (last PING at 1585757100, dropped at 1585757129 :

1585757080: New client connected from 127.0.0.1 as client-id (c1, k20, u'foo').
1585757080: Sending CONNACK to client-id (0, 0)
1585757080: Received SUBSCRIBE from client-id
1585757080:     TEST/# (QoS 0)
1585757080: client-id 0 TEST/#
1585757080: Sending SUBACK to client-id
1585757080: Received PUBLISH from client-id (d0, q1, r0, m2, 'TIME', ... (18 bytes))
1585757080: Sending PUBACK to client-id (Mid: 2)
1585757080: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TIME', ... (18 bytes))
1585757100: Received PINGREQ from client-id
1585757100: Sending PINGRESP to client-id
1585757107: New connection from 127.0.0.1 on port 1883.
1585757107: New client connected from 127.0.0.1 as mosqpub|10410-ijl20-tos (c1, k60).
1585757107: Sending CONNACK to mosqpub|10410-ijl20-tos (0, 0)
1585757107: Received PUBLISH from mosqpub|10410-ijl20-tos (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Received DISCONNECT from mosqpub|10410-ijl20-tos
1585757107: Client mosqpub|10410-ijl20-tos disconnected.
1585757129: Client client-id has exceeded timeout, disconnecting.
1585757129: Socket error on client client-id, disconnecting.
1585757135: New connection from 127.0.0.1 on port 1883.
1585757135: New client connected from 127.0.0.1 as client-id (c0, k20, u'foo').
1585757135: Sending CONNACK to client-id (0, 0)
1585757135: Received SUBSCRIBE from client-id
1585757135:     TEST/# (QoS 0)
1585757135: client-id 0 TEST/#
1585757135: Sending SUBACK to client-id
1585757136: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757136: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757147: Received DISCONNECT from client-id
1585757147: Client client-id disconnected.
1585757196: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757196: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757256: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757256: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757316: Received PINGREQ from mosqsub|4950-ijl20-tosh

Mixser · 2020-04-02T08:20:13Z

Hi @ijl20, thanks for your investigation, it will help us to understand and find out where is the mistake.

ijl20 · 2020-04-02T14:36:31Z

On further debugging the issue seems to be that gmqtt is treating an incoming message from a mosquitto subscription as ok to reset internal timestamp MQTTConnection._last_data_in, but mosquitto is not using that publish to reset its internal keepalive timer.

I modified MQTTConnection._keep_connection as below,

to add the additional logger writes to show the internal timestamp (which shows the gmqtt _last_data_in time getting reset so the next PINGREQ not getting sent),
to comment out the _last_data_in test so the PINGREQ is always sent - the timeout at mosquitto then no longer occurs.

    def _keep_connection(self):
        logger.debug('[_keep_connection] since: {:.3f}'.format(time.monotonic() - self._last_data_in))
        
        if self.is_closing():
            return

        if time.monotonic() - self._last_data_in >= 2 * self._keepalive:
            logger.warning("[LOST HEARTBEAT FOR %s SECONDS, GOING TO CLOSE CONNECTION]", 2 * self._keepalive)
            asyncio.ensure_future(self.close())
            return

        #if time.monotonic() - self._last_data_in >= 0.8 * self._keepalive:
        #    logger.debug('[_keep_connection] send_ping_request()')
        self._send_ping_request()
            
        logger.debug('[_keep_connection] resetting own loop.call_later()]')
        self._keep_connection_callback = asyncio.get_event_loop().call_later(self._keepalive / 2, self._keep_connection)

I think using _last_data_in from MQTTConnection.put_package is questionable to assume the broker has also reset its timeout (certainly doesn't work with mosquitto). Maybe _last_data_out would work but that variable is initialized in MQTTConnection but never used AFAIK.

Thanks for the effort supporting this - IMHO gmqtt is a nice product which sensibly aims at providing just the client well rather than client+broker.

ijl20 · 2020-04-07T13:03:54Z

Quick heads up... on further testing with patch #105 I think there remains a similar timeout issue for a client publisher. (I haven't properly debugged yet, just wanted to post quickly)

I modified my earlier test_timeout.py (itself based on the gmqtt homepage sample) to remove the subscription in on_connect and add another co-routine to publish every 10 seconds ( full code ):

def ts_string():
    return '{:.6f}'.format(time.time())

async def publisher(client):
    while True:
        await asyncio.sleep(10)
        ts = ts_string()
        print("{} [test_pub_timeout] publishing".format(ts))
        client.publish('TIME', ts, qos=0)

async def main(broker_host):
    client = MQTTClient("client-id")

    client.on_connect = on_connect
    client.on_message = on_message
    client.on_disconnect = on_disconnect
    client.on_subscribe = on_subscribe

    client.set_auth_credentials('mqtt-user','mqtt-password')
    await client.connect(broker_host, keepalive=20, version=MQTTv311)

    client.publish('TIME', str(time.time()), qos=0)

    asyncio.ensure_future(publisher(client))

    await STOP.wait()
    await client.disconnect()

Although the regular publishes succeed, the responses from mosquitto do not cause the connection.py self._last_data_in to be updated, and the gmqtt client disconnects. Pls note for this test the client has keepalive=20.

2020-04-07 13:20:46,746 [CONNECTION MADE]
2020-04-07 13:20:46,746 [CMD 0x20] b'\x01\x00'
2020-04-07 13:20:46,747 [CONNACK] flags: 0x1, result: 0x0
1586262046.747338 Connected
2020-04-07 13:20:46,747 [QoS query IS EMPTY]
2020-04-07 13:20:51,748 [QoS query IS EMPTY]
1586262052.676289 [test_pub_timeout] publishing
2020-04-07 13:20:52,676 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:20:56,748 [_keep_connection fix] since: 10.001
2020-04-07 13:20:56,749 [QoS query IS EMPTY]
2020-04-07 13:21:01,752 [QoS query IS EMPTY]
1586262062.678739 [test_pub_timeout] publishing
2020-04-07 13:21:02,678 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:06,752 [_keep_connection fix] since: 20.006
2020-04-07 13:21:06,753 [QoS query IS EMPTY]
2020-04-07 13:21:11,758 [QoS query IS EMPTY]
1586262072.680279 [test_pub_timeout] publishing
2020-04-07 13:21:12,680 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:16,757 [_keep_connection fix] since: 30.010
2020-04-07 13:21:16,760 [QoS query IS EMPTY]
2020-04-07 13:21:21,765 [QoS query IS EMPTY]
1586262082.682315 [test_pub_timeout] publishing
2020-04-07 13:21:22,682 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:26,761 [_keep_connection fix] since: 40.014
2020-04-07 13:21:26,761 [LOST HEARTBEAT FOR 40 SECONDS, GOING TO CLOSE CONNECTION]
2020-04-07 13:21:26,761 [CONN CLOSE NORMALLY]
2020-04-07 13:21:26,762 [CMD 0xe0] b''
1586262086.762310 Disconnected
2020-04-07 13:21:26,762 [TRYING WRITE TO CLOSED SOCKET]
1586262092.689112 [test_pub_timeout] publishing
2020-04-07 13:21:32,689 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:32,689 [TRYING WRITE TO CLOSED SOCKET]
2020-04-07 13:21:32,763 [CONNECTION MADE]
2020-04-07 13:21:32,763 [CMD 0x20] b'\x01\x00'
2020-04-07 13:21:32,764 [CONNACK] flags: 0x1, result: 0x0
1586262092.764317 Connected
2020-04-07 13:21:32,764 [QoS query IS EMPTY]
2020-04-07 13:21:37,770 [QoS query IS EMPTY]
1586262102.695539 [test_pub_timeout] publishing
2020-04-07 13:21:42,695 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:42,763 [_keep_connection fix] since: 10.000

From a simple mosquitto_sub output, the messages published to TIME were as below, i.e. 1586262092 was lost :

1586262052.676289
1586262062.678739
1586262072.680279
1586262082.682315
1586262102.695539

Lenka42 · 2020-04-07T14:29:44Z

@ijl20 thanks for investigation 🙂
This is an interesting case, when messages from client (Publish qos 0) are coming quit often, but broker does not respond in any way. I pushed commit with a fix for this case #105 , could you please check if looks good to you?

ijl20 · 2020-04-07T18:20:29Z

Fix looks good to me (I cloned master from Lenka42/gmqtt), and my pub & sub test code all behaves fine now. I hadn't considered the significance of frequent + qos=0 publishes being the special case in my 'publisher' issue but see exactly what you mean.

Lenka42 · 2020-04-08T13:24:39Z

@dgambin Could you please check the newest version 0.6.4 and see if it solves your problem?

dgambin · 2020-04-14T12:01:02Z

I have just tried and for now I don't see any problem :-)

Lenka42 · 2020-04-22T14:03:12Z

@dgambin @ijl20 it seems to me that issue is solved. If you have any other consideration, feel free to reopen 🙂

Lenka42 mentioned this issue Apr 3, 2020

Enhancements on keepalive functionality #105

Merged

Lenka42 closed this as completed Apr 22, 2020

Mixser mentioned this issue Jun 3, 2020

Periodically disconnects with mosquitto 1.4.15 #110

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection lost on Linux server #103

Connection lost on Linux server #103

dgambin commented Mar 20, 2020

Lenka42 commented Mar 20, 2020

dgambin commented Mar 20, 2020 •

edited

Loading

Lenka42 commented Mar 25, 2020

ijl20 commented Apr 1, 2020 •

edited

Loading

Mixser commented Apr 2, 2020

ijl20 commented Apr 2, 2020 •

edited

Loading

ijl20 commented Apr 7, 2020 •

edited

Loading

Lenka42 commented Apr 7, 2020

ijl20 commented Apr 7, 2020 •

edited

Loading

Lenka42 commented Apr 8, 2020 •

edited

Loading

dgambin commented Apr 14, 2020

Lenka42 commented Apr 22, 2020

Connection lost on Linux server #103

Connection lost on Linux server #103

Comments

dgambin commented Mar 20, 2020

Lenka42 commented Mar 20, 2020

dgambin commented Mar 20, 2020 • edited Loading

Lenka42 commented Mar 25, 2020

ijl20 commented Apr 1, 2020 • edited Loading

Mixser commented Apr 2, 2020

ijl20 commented Apr 2, 2020 • edited Loading

ijl20 commented Apr 7, 2020 • edited Loading

Lenka42 commented Apr 7, 2020

ijl20 commented Apr 7, 2020 • edited Loading

Lenka42 commented Apr 8, 2020 • edited Loading

dgambin commented Apr 14, 2020

Lenka42 commented Apr 22, 2020

dgambin commented Mar 20, 2020 •

edited

Loading

ijl20 commented Apr 1, 2020 •

edited

Loading

ijl20 commented Apr 2, 2020 •

edited

Loading

ijl20 commented Apr 7, 2020 •

edited

Loading

ijl20 commented Apr 7, 2020 •

edited

Loading

Lenka42 commented Apr 8, 2020 •

edited

Loading