email gateway: match from-email address against known users

[asah]

then synthesize a new message, as closely as possible as if the user had sent it from the UI...

(obviously, there's a security/impersonation risk for email coming from insecure domains e.g. not using DKIM/etc)

[timabbott]

@asah can you explain the use case you have in mind in more detail?

[zulipbot]

Hello @zulip/server-development, @zulip/server-integrations members, this issue was labeled with the "area: integrations", "area: emails" labels, so you may want to check it out!

[asah]

When replying to messages posted from the email gateway, they lose their association with the zulip-user who originated them:

• replies don't go to the originating user, which means that person may not see replies.
• stats, reactions etc don't accrue to the proper user.

strawman design: check the from: of the message, compare it to UserProfile.delivery_email and if they match, post the message on behalf of that user with (tbd) some indicator that it came via the gateway.

[asah]

(prototyping in my deployment)

screenshot: (no longer says "from Gateway...")

rough diff:

diff --git a/zerver/lib/email_mirror.py b/zerver/lib/email_mirror.py
index 0e9260818d..8eb7044630 100644
--- a/zerver/lib/email_mirror.py
+++ b/zerver/lib/email_mirror.py
@@ -41,6 +41,7 @@ from zerver.models import (
     get_stream_by_id_in_realm,
     get_system_bot,
     get_user,
+    get_user_by_delivery_email,
 )
 from zproject.backends import is_user_active

@@ -422,7 +423,21 @@ def process_stream_message(to: str, message: EmailMessage) -> None:
     if "include_quotes" not in options:
         options["include_quotes"] = is_forwarded(subject_header)

+    from_header = message.get("From", "")
+    try:
+        from_header = re.sub(r'^.*<(.+?)>', r'\1', from_header.strip())
+        userprofile = get_user_by_delivery_email(from_header, stream.realm)
+        options['show_sender'] = False
+    except Exception as exc:
+        userprofile = None
     body = construct_zulip_body(message, stream.realm, **options)
+    if userprofile is None:
+        print(f"couldn't find user matching delivery_address {from_header}, " +
+              "using the system_bot userprofile instead")
+        userprofile = get_system_bot(settings.EMAIL_GATEWAY_BOT, stream.realm_id)
+        body = f"From: {from_header}\n{body}"
+    else:
+        body = f"(via email) {body}"

-    send_zulip(get_system_bot(settings.EMAIL_GATEWAY_BOT, stream.realm_id), stream, subject, body)
+    send_zulip(userprofile, stream, subject, body)
     logger.info(
         "Successfully processed email to %s (%s)",
         stream.name,

[jasonhildebrand]

I recently learned that zulip allows incoming posts via the email gateway even to streams of an invite-only organization. I expect that private zulip communities will eventually receive a lot of spam, once their stream email addresses become known or get harvested.

For invite-only organizations, I think matching the from-email to a known user should be a requirement, and if a post can't be matched, then the post should be discarded (and log that this occurred).