Getting 404 error from WS URL in RTMClient#start() with aiohttp 3.7.0 or lower versions #912

mgtech-sydney · 2021-01-11T01:45:21Z

RTMClient.start() failing as of 2021-01-09. Connection attempted with both slack_sdk and (legacy) slackclient.

The Slack SDK version

slack-sdk 3.1.1

(Also occuring on slackclient 2.9.3)

Python runtime version

Python 3.6.7

OS info

Microsoft Windows [Version 10.0.18362.1256]

Steps to reproduce:

> import slack_sdk.rtm as rtm

> rtm_client = rtm.RTMClient(token=token) # also attempted with ssl_context provided
> rtm_client.start()

Testing via https://api.slack.com/methods/rtm.connect/test is successful with the same token as used when running code sample above.

Actual result:

  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

The text was updated successfully, but these errors were encountered:

seratch · 2021-01-11T03:04:08Z

@mgtech-sydney I guess the default SSL version in Python 3.6.7 on Windows OS may be a bit old. Can you try the following code out?

import logging
logging.basicConfig(level=logging.DEBUG)

import ssl
from slack_sdk.rtm import RTMClient

# Use this specific SSL version
ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
token = "xoxb-***"
RTMClient(token=token, ssl=ssl_context).start()

mgtech-sydney · 2021-01-11T04:45:46Z

Hi @seratch - output from running the above below, no change in the error.

I also ran this on python:3.7-alpine docker image and got the same result.

DEBUG:slack_sdk.rtm:Retrieving websocket info.
DEBUG:slack_sdk.web.legacy_base_client:Sending a request - url: https://www.slack.com/api/rtm.connect, query_params: {}, body_params: {}, files: {}, json_body: None, headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': '(redacted)', 'User-Agent': 'Python/3.6.7 slackclient/3.1.1 Windows/10'}
DEBUG:slack_sdk.web.legacy_slack_response:Received the following response - status: 200, headers: {'date': 'Mon, 11 Jan 2021 03:39:22 GMT', 'server': 'Apache', 'x-xss-protection': '0', 'pragma': 'no-cache', 'cache-control': 
'private, no-cache, no-store, must-revalidate', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-slack-req-id': 'f3ae8d4f615cb9bbf668bf34d1505b08', 'x-content-type-options': 'nosniff', 'referrer-policy': 'no-referrer', 'access-control-expose-headers': 'x-slack-req-id, 
retry-after', 'x-slack-backend': 'r', 'x-oauth-scopes': 'identify,bot:basic', 'x-accepted-oauth-scopes': 'rtm:stream,client', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'vary': 'Accept-Encoding', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-(continues...)
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    rtm.start()
...\__init__.py", line 52, in start
    rtm_client.start()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  ...\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  ...src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

seratch · 2021-01-11T05:33:55Z

Can you check the OpenSSL version you are using? You can check the version by printing ssl.OPENSSL_VERSION in Python. If you are not using the latest (1.1.1), upgrading the version and enabling it for your Python interpreter may help.

jvperrin · 2021-01-12T02:29:38Z

I'm getting this too, from an ubuntu bionic container:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/code/marley/main.py", line 297, in <module>
    main()
  File "/code/marley/main.py", line 293, in main
    marley.start()
  File "/code/marley/main.py", line 190, in start
    self.rtm.start()
  File "/code/virtualenv_run/lib/python3.7/site-packages/slack_sdk/rtm/__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/code/virtualenv_run/lib/python3.7/site-packages/slack_sdk/rtm/__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  File "/code/virtualenv_run/lib/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/code/virtualenv_run/lib/python3.7/site-packages/aiohttp/client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

I am using Python 3.7.5 and OpenSSL 1.1.1:

$ /code/virtualenv_run/bin/python
Python 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1  11 Sep 2018'

Adding in ssl.SSLContext(ssl.PROTOCOL_SSLv23) and RTMClient(token=token, ssl=ssl_context).start() seems to have no effect either, unfortunately.

Harpsichord1207 · 2021-01-12T03:06:15Z

I got the same error since last night(01/11), without modifying any code. Maybe it's a slack server error?

KIM-DONGWON · 2021-01-12T04:09:56Z

I am using Python 3.7 and docker environment and the same error occurs but I cannot find the cause. Is it an error on the slack server?

seratch · 2021-01-12T04:15:37Z

We are sorry for the disruption.

I have been trying to reproduce the situation on my end. However, I haven't managed to see the error with my classic apps yet.

We would like to figure out if this is specific to this SDK or Python runtime first. It would be greatly appreciated if anyone could help us identify the conditions. What happens if you try to connect using the same classic bot token with @slack/rtm-api package in the Node SDK? Does the same situation arise?

If the issue occurs with Node SDK too, we will communicate to the server-side teams an closely work with them. If this happens only with this Python SDK, I will continue looking at the steps to reproduce.

ongjp · 2021-01-12T04:25:42Z

I am having the same problem here as everyone using the RTMClient. I am using Python 3.7 and my RTM chatbot was working up until about 1 hour ago.
Unfortunately I am not familiar with Node, but I can confirm on Python it is not working for me either.

undral23 · 2021-01-12T04:34:00Z

I got the same error since about 3-4 hours ago.

redliu312 · 2021-01-12T04:35:23Z

me too
OpenSSL OpenSSL 1.1.1d
Python 3.8.3

jusdino · 2021-01-12T04:35:29Z

Same problem. Python 3.6, OpenSSL 1.1.1.

Looks like there may be a lot of us. Perhaps we should switch to 👍 on the post if we have no new info to add?

Harpsichord1207 · 2021-01-12T04:38:04Z

@seratch I'm not familiar with node, i try to do the following steps:

npm install @slack/rtm-api

create a test.js with content:

const { RTMClient } = require('@slack/rtm-api');

// Read a token from the environment variables
const token = 'xoxb-xxxxxxxxxxxxxxx';

// Initialize
const rtm = new RTMClient(token);

(async () => {
  // Connect to Slack
  const { self, team } = await rtm.start();
})();

run node test.js

and there‘s no output or error with correct token. If I set a wrong token, it raise an UnhandledPromiseRejectionWarning: Error: An API error occurred: invalid_auth, so I guess Node SDK works fine.

sujan-paidipala · 2021-01-12T04:38:45Z

I see the same problem since couple of hours.
+1

Yannnnn001 · 2021-01-12T04:54:18Z

I see the same problem since couple of hours.
+1

edribeirojr · 2021-01-12T04:55:33Z

I see the same problem since couple of hours.
+1

brendan-mz · 2021-01-12T04:55:48Z

My bot started experiencing the same issue at 6:45pm PST and has not been able to re-connect since.

Python 3.6.8
OpenSSL 1.0.2s

aoberoi · 2021-01-12T05:14:33Z

For those of you experiencing this issue, we'd like to know if you are (👍) or are not (👎) using the proxy option to connect via a proxy. Would you mind reacting to this comment?

aoberoi · 2021-01-12T05:24:16Z

@jvperrin and @mgtech-sydney did you both redact the URL on the last line of this stack trace, or is this the exact URL you saw?

if you redacted it, given that those URLs expire within 10 mins, they would no longer be dangerous to share. and just being cautious, if you'd like to email me (ankur @ slack-corp dot com), seeing those URLs might be helpful to our investigation.

DEBUG:slack_sdk.rtm:Retrieving websocket info.
DEBUG:slack_sdk.web.legacy_base_client:Sending a request - url: https://www.slack.com/api/rtm.connect, query_params: {}, body_params: {}, files: {}, json_body: None, headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': '(redacted)', 'User-Agent': 'Python/3.6.7 slackclient/3.1.1 Windows/10'}
DEBUG:slack_sdk.web.legacy_slack_response:Received the following response - status: 200, headers: {'date': 'Mon, 11 Jan 2021 03:39:22 GMT', 'server': 'Apache', 'x-xss-protection': '0', 'pragma': 'no-cache', 'cache-control': 
'private, no-cache, no-store, must-revalidate', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-slack-req-id': 'f3ae8d4f615cb9bbf668bf34d1505b08', 'x-content-type-options': 'nosniff', 'referrer-policy': 'no-referrer', 'access-control-expose-headers': 'x-slack-req-id, 
retry-after', 'x-slack-backend': 'r', 'x-oauth-scopes': 'identify,bot:basic', 'x-accepted-oauth-scopes': 'rtm:stream,client', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'vary': 'Accept-Encoding', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-(continues...)
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    rtm.start()
...\__init__.py", line 52, in start
    rtm_client.start()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  ...\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  ...src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

mgtech-sydney · 2021-01-12T05:43:27Z

Hi @aoberoi

The below is a URL from our production environment (Alpine:Python3.7) which saw the error.

wss://cerberus-xxxx.lb.slack-msgs.com/websocket/6_EWYtaXs4UzN7m2JH0huysec3Di2Kgi5cF_77o8hJA_AtEfGxnKC9OI2lAJDAHPixU0ZwDo7y7UNfhspz4KHdj35qcIAhznqkuARS75wAA=

I can generate the error / URL locally in a Windows environment if that would help you as well.

jhofeditz · 2021-01-12T05:47:14Z

@aoberoi we are seeing the same thing just started in the last 3 hours. No proxy that I am aware of.

Connecting to Slack using RTM api.
Connection to slack failed: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/FGtDpGKyogzoJ3k3qzT2S5cfipIxjI4AnBZhRmp2AYk4h6FaxgNPtcV_nnnNMujbd19vwtmvWi0J-uflBvJGzGHbXcGujUh_J4TIizcuDkE=')

ongjp · 2021-01-12T05:54:38Z

I was facing the same issue. It started working after doing the following:

Upgrading aiohttp - pip install -U aiohttp

from slack import RTMClient -> from slack_sdk.rtm import RTMClient

this worked for me - thanks very much

mgtech-sydney · 2021-01-12T05:59:30Z

For me also. Thanks.

brendan-mz · 2021-01-12T06:01:37Z

Confirming the aiohttp upgrade works. I was on 3.6.2, upgraded to 3.7.3 and everything started working again.

jhofeditz · 2021-01-12T06:03:01Z

we just deployed a new release upgrading slackclient from v2.7.3 to v2.9.3 and it seems to be working now.

sujan-paidipala · 2021-01-12T06:56:18Z

Working for me too after aiohttp upgrade. Thanks for the help!

JeckPikachan · 2021-01-12T08:05:03Z

Upgrading aiohttp worked for me too. Thanks a lot!

makdee88 · 2021-01-12T09:00:25Z

Upgrading to aiohttp-3.7.3 and slackclient-2.9.3 fixed the issue. Thanks a ton.

[slackbot@xxxxslackbot]$ pip3 install -U aiohttp --proxy http://x.x.x:8888 --user slackbot
...
Successfully installed aiohttp-3.7.3 attrs-20.3.0 certifi-2020.12.5 idna-3.1 multidict-5.1.0 requests-2.25.1 urllib3-1.26.2 yarl-1.6.3

[slackbot@xxxx slackbot]$
[slackbot@xxxx slackbot]$ pip3 install -U slackclient --proxy http://x.x.x:8888 --user slackbot
...
Successfully installed idna-2.10 slackclient-2.9.3
[slackbot@xxxx slackbot]$

andrey-tikhov · 2021-01-12T09:21:43Z

Saving me. SAVING ME!!!! aiohttp-3.7.3

hkonala · 2021-01-12T10:20:04Z

@seratch I am still getting the same error even after upgrading aiohttp and slackclient. Could someone help?

[ec2-user@xxxxxxxxxxxxx skills]$ pip freeze | grep -E 'slackclient|aiohttp|opsdroid'
aiohttp==3.7.3
opsdroid==0.19.0
opsdroid-get-image-size==0.2.2
slackclient==2.9.3

Traceback (most recent call last):
  File "/usr/local/bin/opsdroid", line 11, in <module>
    load_entry_point('opsdroid==0.19.0', 'console_scripts', 'opsdroid')()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opsdroid/cli/start.py", line 43, in start
    opsdroid.run()
  File "/usr/local/lib/python3.6/site-packages/opsdroid/core.py", line 169, in run
    self.eventloop.run_until_complete(asyncio.gather(*pending))
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/slack/rtm/client.py", line 357, in _connect_and_read
    proxy=self.proxy,
  File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/fH8ZRCPmqUOTp3W58-XT7cylC8tGT5m1n_xr1U3ji9Ea2WEJQIaER14w9dNJ9qeDFwrLliVkW3ONyfE7Qs8mOp8S6e4XBcHIuDrh4US2KSA=')
ERROR: Unhandled exception in opsdroid, exiting...

seratch · 2021-01-12T10:32:40Z

File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 738, in _ws_connect
headers=resp.headers)

@hkonala The stacktrace indicates that your app still loads aiohttp v3.6

hkonala · 2021-01-12T11:03:22Z

@seratch it is saying opsdroid 0.19.0 requires slackclient==2.6.2 and aiohttp==3.6.2

[ec2-user@ip-xxxxxxxx site-packages]$ opsdroid start
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (slackclient 2.9.3 (/home/ec2-user/.local/lib/python3.6/site-packages), Requirement.parse('slackclient==2.6.2'), {'opsdroid'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/opsdroid", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3261, in <module>
    @_call_aside
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'aiohttp==3.6.2' distribution was not found and is required by opsdroid

seratch · 2021-01-12T11:25:37Z

@hkonala I am sorry to say that, in your case, there is no workaround until the issue is fixed on the server-side. I cannot tell the exact timing of the deployment, but they will resolve this early tomorrow, Pacific Time. Again, we are sorry for the disruption.

hkonala · 2021-01-12T11:27:28Z

@seratch Thanks for the info!! will wait for the fix :)

biboc · 2021-01-12T15:00:26Z

Upgraded to aiohttp==3.7.3 and it works
But there is big delay between question from user and bot receiving the question now

tgwaste · 2021-01-12T16:20:55Z

What is the status of the fix?

fndiaz · 2021-01-12T16:28:01Z

Install aiohttp, work for me too!

tgwaste · 2021-01-12T16:29:48Z

A freaking heads up from slack sure would had been nice. They sure have interrupted business a lot lately.

slackapi/python-slack-sdk#912 (comment)

spladug · 2021-01-12T20:08:58Z

For anyone else running into this that isn't using aiohttp, it appears this was the change to aiohttp that made the difference.

If your client implementation sends any capitalization other than Upgrade: websocket (e.g. Upgrade: WebSocket) you'll get the 404.

In my case, I patched autobahn here: https://github.com/crossbario/autobahn-python/blob/master/autobahn/websocket/protocol.py#L3576

seratch · 2021-01-13T02:50:20Z

According to the server-side teams, no apps should be experiencing this issue now. We apologize for the inconvenience and appreciate your patience.

hkonala · 2021-01-13T05:37:22Z

Yes @seratch .. Things seem to be working fine now.

mgtech-sydney added the untriaged label Jan 11, 2021

seratch added question M-T: User needs support to use the project rtm-client Version: 2x Version: 3x and removed untriaged labels Jan 11, 2021

seratch added bug M-T: A confirmed bug report. Issues are confirmed when the reproduction steps are documented dependency-side-issue server-side-issue and removed question M-T: User needs support to use the project labels Jan 12, 2021

stvnksslr mentioned this issue Jan 12, 2021

Issue with slackclient 2.7.3 DonDebonair/slack-machine#434

Closed

seratch added a commit to seratch/python-slack-sdk that referenced this issue Jan 12, 2021

Add aiohttp version validation (ref slackapi#912)

0d06add

seratch mentioned this issue Jan 12, 2021

Add aiohttp version validation (ref #912) #914

Merged

11 tasks

seratch added a commit that referenced this issue Jan 12, 2021

Add aiohttp version validation (ref #912)

a90ba03

powellchristoph mentioned this issue Jan 12, 2021

Fix slackmachine mailgun/aio-slack-machine#14

Merged

sosiouxme added a commit to openshift-eng/art-bot that referenced this issue Jan 12, 2021

require newer aiohttp

c08055a

slackapi/python-slack-sdk#912 (comment)

afresh1 mentioned this issue Jan 12, 2021

Slack's WebSocket API requires lowercase "websocket" in the Upgrade header vti/protocol-websocket#65

Open

seratch added a commit that referenced this issue Jan 12, 2021

Add aiohttp version validation (ref #912)

3e37278

starSweeper mentioned this issue Jan 12, 2021

Bug fixes starSweeper/AlignmentBot#16

Merged

seratch closed this as completed Jan 14, 2021

Getting 404 error from WS URL in RTMClient#start() with aiohttp 3.7.0 or lower versions #912

Getting 404 error from WS URL in RTMClient#start() with aiohttp 3.7.0 or lower versions #912

Comments

mgtech-sydney commented Jan 11, 2021 • edited by seratch

The Slack SDK version

Python runtime version

OS info

Steps to reproduce:

Actual result:

seratch commented Jan 11, 2021

mgtech-sydney commented Jan 11, 2021 • edited

seratch commented Jan 11, 2021

jvperrin commented Jan 12, 2021

Harpsichord1207 commented Jan 12, 2021

KIM-DONGWON commented Jan 12, 2021

seratch commented Jan 12, 2021

ongjp commented Jan 12, 2021

undral23 commented Jan 12, 2021

redliu312 commented Jan 12, 2021 • edited

jusdino commented Jan 12, 2021 • edited

Harpsichord1207 commented Jan 12, 2021

sujan-paidipala commented Jan 12, 2021

Yannnnn001 commented Jan 12, 2021

edribeirojr commented Jan 12, 2021

brendan-mz commented Jan 12, 2021

aoberoi commented Jan 12, 2021

aoberoi commented Jan 12, 2021

mgtech-sydney commented Jan 12, 2021

jhofeditz commented Jan 12, 2021

ongjp commented Jan 12, 2021

mgtech-sydney commented Jan 12, 2021

brendan-mz commented Jan 12, 2021

jhofeditz commented Jan 12, 2021

sujan-paidipala commented Jan 12, 2021 • edited

JeckPikachan commented Jan 12, 2021

makdee88 commented Jan 12, 2021

andrey-tikhov commented Jan 12, 2021

hkonala commented Jan 12, 2021 • edited by seratch

seratch commented Jan 12, 2021

hkonala commented Jan 12, 2021 • edited by seratch

seratch commented Jan 12, 2021

hkonala commented Jan 12, 2021

biboc commented Jan 12, 2021 • edited

tgwaste commented Jan 12, 2021

fndiaz commented Jan 12, 2021

tgwaste commented Jan 12, 2021

spladug commented Jan 12, 2021

seratch commented Jan 13, 2021

hkonala commented Jan 13, 2021

mgtech-sydney commented Jan 11, 2021 •

edited by seratch

mgtech-sydney commented Jan 11, 2021 •

edited

redliu312 commented Jan 12, 2021 •

edited

jusdino commented Jan 12, 2021 •

edited

sujan-paidipala commented Jan 12, 2021 •

edited

hkonala commented Jan 12, 2021 •

edited by seratch

hkonala commented Jan 12, 2021 •

edited by seratch

biboc commented Jan 12, 2021 •

edited