Improve IRC Gateway

[roberthoenig]

See zulip/zulip#249

Moving the issue here, let's have a discussion on what parts of the original one are still relevant.

[timabbott]

I think the big thing we need to do here is have step-by-step instructions for how to run it, documenting its limitations. We can probably just put that in the /integrations/ docs. And then we can open issues for fixing the major limitations revealed in that documentation.

[timabbott]

I think the big thing we need to do here is have step-by-step instructions for how to run it, documenting its limitations. We can probably just put that in the /integrations/ docs. And then we can open issues for fixing the major limitations revealed in that documentation.

[timabbott]

The discussion in zulip/zulip#249 should cover the special Zulip things one needs to know to run it, specifically the need to use manage.py knight to create an "API super user" bot.

[timabbott]

The discussion in zulip/zulip#249 should cover the special Zulip things one needs to know to run it, specifically the need to use manage.py knight to create an "API super user" bot.

[rht]

I tried running the mirror a while ago, found the USAGE content to be sufficient
to get me running. I modified the code so that 1. being an api-super-user is no
longer required, 2. it redirects the IRC messages to a hardcoded topic in a
hardcoded stream (this can be quickly patched as options to the current code
if necessary). Basically it is a one-liner of:
./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY.
This works.

However, a problem occurred when I tried to enable the bi-directional mode
(forwarding messages in Zulip to IRC) by uncommenting the line

python-zulip-api/zulip/integrations/irc/irc_mirror_backend.py

Line 45 in 57f7088

# self.zulip_client.call_on_each_message(forward_to_irc)

.
This line is a blocking synchronous call, and furthermore it prevents the
forwarding from IRC to Zulip from happening, i.e. this line makes everything
non-functional. The jaraco/irc library also has a problem in that it can't
sustain a connection for longer than 30 min, at which point a ping timeout will
happen.

I tried another library, pydle and reconstructed the public subset of the
mirroring script, but reached a dead end in the bidirectional area because
pydle has only one thread that captures the events happening in the IRC
channel (on_connect, on_channel_message, etc). It can't be hooked to an external
event listener such as events in a Zulip server (to have messages being
forwarded from Zulip to the IRC channel). One plus point for pydle is that it
can sustain a more stable connection than jaraco/irc.

The only stable bidirectional solution that has worked so far is via Matrix
(note: they had the entirety of freenode mirrored). So currently I have #git
bidirectional-mirrored to
https://chat.zulip.org/#narrow/stream/IRC/topic/.23git.
Another possible path is to use nodejs (if there is a better IRC library in the
ecosystem) instead of Python.

[rht]

I tried running the mirror a while ago, found the USAGE content to be sufficient
to get me running. I modified the code so that 1. being an api-super-user is no
longer required, 2. it redirects the IRC messages to a hardcoded topic in a
hardcoded stream (this can be quickly patched as options to the current code
if necessary). Basically it is a one-liner of:
./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY.
This works.

However, a problem occurred when I tried to enable the bi-directional mode
(forwarding messages in Zulip to IRC) by uncommenting the line

python-zulip-api/zulip/integrations/irc/irc_mirror_backend.py

Line 45 in 57f7088

# self.zulip_client.call_on_each_message(forward_to_irc)

.
This line is a blocking synchronous call, and furthermore it prevents the
forwarding from IRC to Zulip from happening, i.e. this line makes everything
non-functional. The jaraco/irc library also has a problem in that it can't
sustain a connection for longer than 30 min, at which point a ping timeout will
happen.

I tried another library, pydle and reconstructed the public subset of the
mirroring script, but reached a dead end in the bidirectional area because
pydle has only one thread that captures the events happening in the IRC
channel (on_connect, on_channel_message, etc). It can't be hooked to an external
event listener such as events in a Zulip server (to have messages being
forwarded from Zulip to the IRC channel). One plus point for pydle is that it
can sustain a more stable connection than jaraco/irc.

The only stable bidirectional solution that has worked so far is via Matrix
(note: they had the entirety of freenode mirrored). So currently I have #git
bidirectional-mirrored to
https://chat.zulip.org/#narrow/stream/IRC/topic/.23git.
Another possible path is to use nodejs (if there is a better IRC library in the
ecosystem) instead of Python.

[timabbott]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option). If you look at zephyr_mirror.py, it runs 2 copies of zephyr_mirror_backend.py with different options, one for each direction; that system is quite reliable.

The 30m timeout issue with the current library is annoying, though.

[timabbott]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option). If you look at zephyr_mirror.py, it runs 2 copies of zephyr_mirror_backend.py with different options, one for each direction; that system is quite reliable.

The 30m timeout issue with the current library is annoying, though.

[rht]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option).

Yeah, I forgot to tell that I did also attempt to run two specialized scripts -- one for forwarding to IRC (zulip-irc-receiver), another for forwarding from IRC (zulip-irc-sender). This works for up to 30 min only though, after which they have to be restarted again.

[rht]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option).

Yeah, I forgot to tell that I did also attempt to run two specialized scripts -- one for forwarding to IRC (zulip-irc-receiver), another for forwarding from IRC (zulip-irc-sender). This works for up to 30 min only though, after which they have to be restarted again.

[rheaparekh]

The current IRC gateway manages to run with two copies (for both side message forwarding) running. However all the Issues I identified were:

• Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio jaraco/irc#135, with an open PR on it!)
• Message formatting isn't correct, the users sending the messages in Zulip are not being shown in IRC.
• Messages are replicated (When we forward a message from Zulip->IRC, IRC forward the same message back to Zulip)

[rheaparekh]

The current IRC gateway manages to run with two copies (for both side message forwarding) running. However all the Issues I identified were:

• Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio jaraco/irc#135, with an open PR on it!)
• Message formatting isn't correct, the users sending the messages in Zulip are not being shown in IRC.
• Messages are replicated (When we forward a message from Zulip->IRC, IRC forward the same message back to Zulip)

[rht]

Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio jaraco/irc#135, with an open PR on it!)

@rheaparekh the PR was merged on May 21 2018. Did this resolve the timeout issue?

[rht]

Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio jaraco/irc#135, with an open PR on it!)

@rheaparekh the PR was merged on May 21 2018. Did this resolve the timeout issue?

[rht]

Confirmed that irc.client_aio.AioReactor fixes the timeout. I ran ./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY again and had observed the 15-30 min disconnect become stable (1 hour and counting).

Then the remaining problem of running 2 scripts for each direction can be solved with threading once for all.

The patch: rht@7c08c98

[rht]

Confirmed that irc.client_aio.AioReactor fixes the timeout. I ran ./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY again and had observed the 15-30 min disconnect become stable (1 hour and counting).

Then the remaining problem of running 2 scripts for each direction can be solved with threading once for all.

The patch: rht@7c08c98