IRC puppeting

[qaisjp]

This project looks amazing. I don't use this project yet. I currently have a custom setup for three-way bridging.

1. custom software for irc <-> discord
2. open-source bridge for slack <-> discord.

My custom bridge for Discord/IRC which has some advanced features, namely, dummy accounts! See qaisjp/go-discord-irc.

I'd like to merge this into matterbridge.

Features:

• There exists an IRC user per Discord user (name suffixed with ~d, like john~d`) ([Feature] Allow listing of members across bridges #361)
• Each IRC user has a unique hostname containing the Discord UID
  • It achieves this using WebIRC (requires special server tokens). See docs for WebIRC (quick read).
• IRC away status updates based on Discord status (Status monitoring #116)
• IRC user connects when Discord user comes "online" (or speaks whilst offline)
• IRC user disconnects from server after the Discord user is "offline" for a period of time (not just a period of general inactivity) ([Discord] Support join/parts  #654)
• PMing IRC users send PMs to Discord users (a special syntax is needed for Discord to IRC) (Feature Request: DMing users across platform #191)
• Nickname mention conversions (mentioning qaisjp~d on IRC will convert to a correct @qaisjp on DIscord; mentioning @Some Weird Name on Discord will translate to Some_Weird_Name~d on IRC)
• IRC to Discord formatting conversions (inverse planned)
• Webhook support on Discord to have username/avatar spoofing (using instant automatically created webhooks)
  • uses avatar of a Discord user with the same name
  • planned: send Discord avatar to IRC (IRCv3)
• Live updates by editing the config file (partial) (Accommodate reloading of gateway config when watching file #589)
• Discord message edits are transcribed as name~d meant to say "...."
• maybe some more?

I currently have two instances of go-discord-irc deployed to two IRC servers / discord guilds:

1. one with no custom config (so it has no spoofing, only one central listener user), and
2. one with custom config.

The one with custom config has the following features:

• IRC user per Discord member
• Server provides WebIRC credentials
• Server has a special username suffix: allows users with the same username, making all Discord users are suffixed with ~d, making
  • Note: ~ is ordinarily an illegal character, so this is a server with custom code
  • In my setup, Slack users don't appear as separate IRC users, they just appear as a Discord message from the central listener user)
  • If we supported multiple, it would be ~d, ~s, etc

This would also solve the several issues for IRC/Discord, at least when dummy user support is enabled (see list above with tagged issues).

Thoughts? I am happy to send pull requests to the project to make this a reality.

I would need: support from the community (i.e, coping with any questions I have whilst learning the codebase and implementing features).

The aim is to deprecate my project completely and for me to move completely to matterbridge.

[42wim]

I'd welcome support for this, but we have to do some changes to make this a reality.
At the moment, bridges aren't aware of users on other bridges, so this needs to be done first before we can integrate your irc spoofing feature

[patcon]

This all sounds super fucking rad @qaisjp! 🎉

It might take a bit to work through all these pieces as atomic feature requests, but my experience is that if you are patient in working things in, then others (incl myself in whatever way I can) would be really eager to support :)

[qaisjp]

Slowly but steadily we can make this happen! 🙂

[mithro]

This sounds super awesome and I would love to see it happen!

[qaisjp]

I've been looking into this, and I'm not sure what's the best way to go about this config-wise.

Example base config to work off

Take this simple bi-directional configuration that bridges between Discord and IRC.

[discord]
[discord.compsoc]
Token="<some value>"
Server="<some value>"
WebhookURL="<some value>"

[irc]
[irc.compsoc]
Server="localhost:6697"
Nick="DiscordBot"

[[gateway]]
name="compsoc.bottest"
enable=true

[[gateway.inout]]
account="irc.compsoc"
channel="#bottest"

[[gateway.inout]]
account="discord.compsoc"
channel="bottest"

Naming

Discord already support puppets, but they are... "ephemeral". We don't want to people to use a Puppets setting and realise they actually need to use WebhookURL. That can be considered poor user experience.

We know that Discord puppets are not persistent, unlike IRC. So we could name our setting something like PersistentPuppets. Assuming that we set it to a boolean value... which we shouldn't, because....

Flow of knowledge

User information flows from discord to irc. How do we represent this information in the config?

And we probably want a configuration design that can gracefully support more than just IRC puppets.

Suggestion

What if we did something like this? I think including the term bridge might confuse people that are writing their config:

[[gateway]]
name="compsoc.bottest"
enable=true

[[gateway.bridge]]
account="irc.compsoc"
channel="#bottest"
chat="inout"
users="out"   # output

[[gateway.bridge]]
account="discord.compsoc"
channel="bottest"
chat="inout"
users="in"    # input

---

What do you think? How should this be designed? Please give me feedback! I want to know if there's a better way to design the config.

[poVoq]

Hmm, as for naming, I always considered the webhook method as user "spoofing" while what you are proposing for IRC seems to be proper user "puppeteering".

[qaisjp]

I only called it spoofing because when I wrote the issue I had not found a better word for it (which is 'puppeting')

[jlu5]

Hi all,

Lately I've been working on an alternative implementation that allows doing this in a stateless fashion. It involves an IRCd extension and a separate command to translate relayed messages into PRIVMSGs, and allows user spoofing in a way fairly similar to e.g. Discord webhooks. Although this limits the UI slightly (virtual users can't be DMed for example because they aren't actually connected), it simplifies how much state relay bots need to track.

I've tentatively placed the details in a repo: relaymsg.md. At least one IRCd vendor has shown interest in this, so I'm hoping to finalize a draft spec soon and send it to IRCv3 for consideration. See ircv3/ircv3-specifications#417

[Justinzobel]

I'm running a private bridge just for myself to bridge three IRC rooms to private Telegram rooms. Is there a way I could have IRC PMs go to the bot's PM inside Telegram and responses like discussed above? ie user2@irc: Here's my reply to your PM.

[qaisjp]

I'm running a private bridge just for myself to bridge three IRC rooms to private Telegram rooms. Is there a way I could have IRC PMs go to the bot's PM inside Telegram and responses like discussed above? ie user2@irc: Here's my reply to your PM.

Not at the moment

[Justinzobel]

I'm not at my PC at the moment. Do private messages show up in the debug log?

…

On Thu, 6 Aug 2020, 7:46 pm Qais Patankar, ***@***.***> wrote: I'm running a private bridge just for myself to bridge three IRC rooms to private Telegram rooms. Is there a way I could have IRC PMs go to the bot's PM inside Telegram and responses like discussed above? ie ***@***.***: Here's my reply to your PM. Not at the moment — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#667 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKOQBMEWE27AB7UGRW4YA3R7J7G5ANCNFSM4GMXSKNA> .

[westor7]

Will @qaisjp go-discord-irc be merged on @42wim matterbridge ? does it needs anything else to be finished?

This is a most-wanted merge :P i hope to be in the next version..

• Thanks!

[qaisjp]

I've slowly been adding some of the smaller features — the long term aim is to replace my bridge entirely with matterbridge, yes

[Mikaela]

I am not sure if this is the right place to ask, but are there any plans to support the RELAYMSG cap mentioned above (#667 (comment)) or merging the support back to upstream Matterbridge?

I have been lately using a couple of Oragono based networks which would support it and make Matterbridge more pleasant to use on them.

[qaisjp]

The RELAYMSG cap is nice but I still plan to work on a stateful approach that makes it possible for users to actually show up in user lists.

One problem with this stateful approach is that restarting the bridge causes a massive influx of joinquit spam. I'd like to solve this issue (in my bridge) before bringing this feature to matterbridge.

I'm thinking of solving this problem by making it possible (but not required) for a separate process of matterbridge to "manage" the irc connections.

We'd need to decide a way for matterbridge to communicate with this standalone "irc process".

• grpc + protobuf
  • pros: good versioning guarantees
  • cons:
    • imports a large set of packages
    • complicated tooling (but only those working on this component of the bridge would need to set it up, as the corresponding generated Go code would be checked into master — if we don't check the generated code into master, we break go get)
• net/rpc + gob
  • pros:
    • both these packages are in stdlib
    • no extra tooling required
    • gob has strong bindings to Go (therefore easier to use than protobufs)
  • cons: net/rpc is frozen, and has "many bugs" that will never be fixed
• websocket + json:
  • pros:
    • json package is in stdlib (and is literally Go, so v. easy to use)
    • there are fairly mature implementations for websocket (gorilla, and there's also the other newer one)
    • easy to debug using postman or other websocket tools traditionally available for the web
  • cons:
    • have to implement our own versioning
    • no request/response system unlike net/rpc and grpc

The general idea is for the API/ABI to be fairly low level — examples of RPC functions would be:

• making a puppet connect
• making a puppet quit
• sending an arbitrary command as a puppet (NICK, PRIVMSG, etc)
• set puppet custom metadata (e.g. discord user ID, slack user ID, etc)
• get a puppet
• get list of puppets that are connected + their custom metadata + their user info (nickname)

The last RPC function being the magic that allows matterbridge to restart, connect to an existing matterbridge --irc-srv, and recover.

@42wim + anyone else — any comments or thoughts on the tech stack for this component?

Additionally, Wim, you've expressed reservations about keeping the "make many connections" system, and mentioned you were interested in replacing it with an ircd such as Oragono. I imagine both systems could be abstracted away from matterbridge using this same RPC-based API.

[westor7]

@qaisjp
One problem with this stateful approach is that restarting the bridge causes a massive influx of joinquit spam. I'd like to solve this issue (in my bridge) before bringing this feature to matterbridge.

The same is happening when netsplit servers, i think this is not a problem but its by design, you cannot avoid that stuff and i personally think its ok, modern IRCd's having configuration options to hide these joins/quits, the only thing that you can do in that case is to have a specific quit message on matterbridge restart/terminate so ircd or bots could save the nicknames and not take action after on join.

[Mikaela]

One problem with this stateful approach is that restarting the bridge causes a massive influx of joinquit spam.

This is currently Oragono-specific, but would it be possible for a stateful bridge to generate SSL certificates for all bridged users and then register (with) those and set the accounts into always-on mode? Then they wouldn't quit unless the Matterbridge instance they belong to was turned off for a long time (30 days by default if I recall correctly).

Another idea would be for Matterbridge to pretend to be a IRC server, but that would practically require selfhosting IRC as at least freenode won't allow any third party to link that way to my understanding. It wouldn't resolve netsplits, for which there would be https://ircv3.net/specs/extensions/batch-3.2 which again requires a compatible client. I think IRC servers also generally have different linking protocols, so it wouldn't be universal solution (and Oragono doesn't even support that currently ergochat/ergo#26).

[qaisjp]

The same is happening when netsplit servers

Yes, but I've observed that a genuine netsplit happens less frequently than wanting to update the bridge or the bridge crashing.

Another idea would be for Matterbridge to pretend to be a IRC server, but that would practically require selfhosting IRC as at least freenode won't allow any third party to link that way to my understanding.

Am I correct in believing that this idea is in line with the "IRCd mode" I mentioned at the very end of #667 (comment)?

I think IRC servers also generally have different linking protocols, so it wouldn't be universal solution (and Oragono doesn't even support that currently oragono/oragono#26).

It's unfortunate that Oragono does not yet support S2S. "IRCd mode" would have to wait on that feature from Oragono (unless there's an existing stable ircd that supports S2S and is written in Go).

[Mikaela]

Am I correct in believing that this idea is in line with the "IRCd mode" I mentioned at the very end of #667 (comment)?

I am not sure, I have difficulties following the comment.

[westor7]

@qaisjp

The same exactly is happening with ircd/services/znc/etc.. servers, when they need update, they going down updating then going up, i think this is the most common way to make it, and it would not be a problem, also 1 or 2 maximum times per month its not huge problem.

[qaisjp]

I am not sure, I have difficulties following the comment.

Yeah, the last bit in my comment is referencing a prior conversation on Discord where we discussed "[adding an] irc server to matterbridge, thinking about doing that anyway in other issue [...] like matterircd, but a cleaner one [...] integrating ircd opens up "easy" irc puppeting for the other protocols too".

[qaisjp]

The same exactly is happening with ircd/services/znc/etc.. servers, when they need update, they going down updating then going up, i think this is the most common way to make it, and it would not be a problem, also 1 or 2 maximum times per month its not huge problem.

3:30:26 PM <oper> the point is not whether they are different but the disruption they cause
3:30:54 PM <oper> whether you use C2S or S2S, the point is to keep the IRC part in a separate process from the bridge part
3:31:28 PM <oper> so you can restart the bridge part leaving the IRC users in limbo for a period of time and then resynchronise which IRC users should exist
3:32:08 PM <oper> the ircd cannot hide the activity, only the IRC clients can do that
3:33:47 PM <oper> it's easier to do it with C2S, especially as S2S varies per ircd

[42wim]

We'd need to decide a way for matterbridge to communicate with this standalone "irc process".

My vote goes to net/rpc + gob. We can do this over domain sockets, don't really need TCP or TLS I think.
(protobufs are a PITA to put it mildly)

[singpolyma]

Wouldn't restarting the irc specific service still result in the big joinquit? Is the idea just that one might restart matterbridge more often?

[qaisjp]

Is the idea just that one might restart matterbridge more often?

yes

[poVoq]

@qaisjp I guess the plan to integrate your go-discord-irc bridge with Matterbridge is postponed indefinitely?

[qaisjp]

Yes, these days I have not had much motivation to work on open source.

[ajar]

I'm game to take a crack at this one. It will be very useful for our community which still spans multiple services, including IRC which has been active since around 2013. I'll keep this thread updated.

[poVoq]

You could also try: https://github.com/OpenTTD/dibridge

[ajar]

You could also try: https://github.com/OpenTTD/dibridge

Oh neat, that is very interesting indeed. Thanks for sharing this :-) We'll survey the options for this, and glad to see we're not the only ones in this predicament (misery loves company).