Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
Send email notifications for missed messages #759
Conversation
dbkr
added some commits
Apr 19, 2016
Some checks were not successful
Some checks were not successful
Some checks were not successful
All checks have passed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
All checks have failed
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
dbkr
assigned
erikjohnston
Apr 29, 2016
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/config/emailconfig.py
| +# Unless required by applicable law or agreed to in writing, software | ||
| +# distributed under the License is distributed on an "AS IS" BASIS, | ||
| +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| +# See the License for the specific language governing permissions and | ||
| +# limitations under the License. | ||
| + | ||
| +# This file can't be called email.py because if it is, we cannot: | ||
| +import email.utils | ||
| + | ||
| +from ._base import Config | ||
| + | ||
| + | ||
| +class EmailConfig(Config): | ||
| + """ | ||
| + Email Configuration | ||
| + """ |
erikjohnston
Owner
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/config/emailconfig.py
| +import email.utils | ||
| + | ||
| +from ._base import Config | ||
| + | ||
| + | ||
| +class EmailConfig(Config): | ||
| + """ | ||
| + Email Configuration | ||
| + """ | ||
| + | ||
| + def read_config(self, config): | ||
| + self.email_enable_notifs = False | ||
| + | ||
| + email_config = config.get("email", None) | ||
| + if email_config: | ||
| + self.email_enable_notifs = email_config.get("enable_notifs", True) |
erikjohnston
Owner
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/config/server.py
| @@ -143,6 +148,9 @@ def default_config(self, server_name, **kwargs): | ||
| # Whether to serve a web client from the HTTP/HTTPS root resource. | ||
| web_client: True | ||
| + # The server's public-facing base URL | ||
| + # https://example.com:8448/ |
erikjohnston
Owner
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/emailpusher.py
| + should_notify_at = max(notif_ready_at, room_ready_at) | ||
| + | ||
| + if should_notify_at < self.clock.time_msec(): | ||
| + # one of our notifications is ready for sending, so we send | ||
| + # *one* email updating the user on their notifications, | ||
| + # we then consider all previously outstanding notifications | ||
| + # to be delivered. | ||
| + yield self.send_notification(unprocessed) | ||
| + | ||
| + yield self.save_last_stream_ordering_and_success(max([ | ||
| + ea['stream_ordering'] for ea in unprocessed | ||
| + ])) | ||
| + yield self.sent_notif_update_throttle( | ||
| + push_action['room_id'], push_action | ||
| + ) | ||
| + else: |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/emailpusher.py
| + @defer.inlineCallbacks | ||
| + def save_last_stream_ordering_and_success(self, last_stream_ordering): | ||
| + self.last_stream_ordering = last_stream_ordering | ||
| + yield self.store.update_pusher_last_stream_ordering_and_success( | ||
| + self.app_id, self.email, self.user_id, | ||
| + last_stream_ordering, self.clock.time_msec() | ||
| + ) | ||
| + | ||
| + def seconds_until(self, ts_msec): | ||
| + return (ts_msec - self.clock.time_msec()) / 1000 | ||
| + | ||
| + def get_room_last_notif_ts(self, last_notif_by_room, room_id): | ||
| + if room_id in last_notif_by_room: | ||
| + return last_notif_by_room[room_id] | ||
| + else: | ||
| + return 0 |
|
|
dbkr
added some commits
Apr 29, 2016
Some checks were not successful
All checks have failed
All checks have failed
Some checks were not successful
Some checks were not successful
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/mailer.py
| +from synapse.api.errors import StoreError | ||
| + | ||
| +import jinja2 | ||
| +import bleach | ||
| + | ||
| +import time | ||
| +import urllib | ||
| + | ||
| + | ||
| +MESSAGE_FROM_PERSON_IN_ROOM = "You have a message from %s in the %s room" | ||
| +MESSAGE_FROM_PERSON = "You have a message from %s" | ||
| +MESSAGES_FROM_PERSON = "You have messages from %s" | ||
| +MESSAGES_IN_ROOM = "There are some messages for you in the %s room" | ||
| +MESSAGES_IN_ROOMS = "Here are some messages you may have missed" | ||
| +INVITE_FROM_PERSON_TO_ROOM = "%s has invited you to join the %s room" | ||
| +INVITE_FROM_PERSON = "%s has invited you to chat" |
erikjohnston
Owner
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/mailer.py
| + ) | ||
| + | ||
| + ret = { | ||
| + "link": self.make_notif_link(notif), | ||
| + "ts": notif['received_ts'], | ||
| + "messages": [], | ||
| + } | ||
| + | ||
| + handler = self.hs.get_handlers().message_handler | ||
| + the_events = yield handler.filter_events_for_client( | ||
| + user_id, results["events_before"] | ||
| + ) | ||
| + the_events.append(notif_event) | ||
| + | ||
| + for event in the_events: | ||
| + vars = self.get_message_vars(notif, event, room_state) |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/mailer.py
| + | ||
| + handler = self.hs.get_handlers().message_handler | ||
| + the_events = yield handler.filter_events_for_client( | ||
| + user_id, results["events_before"] | ||
| + ) | ||
| + the_events.append(notif_event) | ||
| + | ||
| + for event in the_events: | ||
| + vars = self.get_message_vars(notif, event, room_state) | ||
| + if vars is not None: | ||
| + ret['messages'].append(vars) | ||
| + | ||
| + defer.returnValue(ret) | ||
| + | ||
| + def get_message_vars(self, notif, event, room_state): | ||
| + if event.type != "m.room.message": |
erikjohnston
Owner
|
erikjohnston
and 1 other
commented on an outdated diff
Apr 29, 2016
synapse/push/mailer.py
| + ]) | ||
| + ) | ||
| + else: | ||
| + # Stuff's happened in multiple different rooms | ||
| + return MESSAGES_IN_ROOMS | ||
| + | ||
| + def make_room_link(self, room_id): | ||
| + return "https://matrix.to/%s" % (room_id,) | ||
| + | ||
| + def make_notif_link(self, notif): | ||
| + return "https://matrix.to/%s/%s" % ( | ||
| + notif['room_id'], notif['event_id'] | ||
| + ) | ||
| + | ||
| + def make_unsubscribe_link(self): | ||
| + return "https://vector.im/#/settings" # XXX: matrix.to |
erikjohnston
Owner
|
erikjohnston
and 1 other
commented on an outdated diff
Apr 29, 2016
synapse/push/pusher.py
| def create_pusher(hs, pusherdict): | ||
| + PUSHER_TYPES = { | ||
| + "http": HttpPusher, | ||
| + } | ||
| + | ||
| + if hs.config.email_enable_notifs: | ||
| + from emailpusher import EmailPusher |
dbkr
Member
|
|
If using optional dependencies then you should check in the config that the module are importable (like we do with url previews and netaddr) |
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| } for row in after_read_receipt + no_read_receipt | ||
| ]) | ||
| @defer.inlineCallbacks | ||
| + def get_time_of_last_push_action_before(self, stream_ordering): | ||
| + def f(txn): | ||
| + sql = ( | ||
| + "SELECT e.received_ts " | ||
| + "FROM event_push_actions AS ep " | ||
| + "JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id " | ||
| + "WHERE ep.stream_ordering > ? " | ||
| + "ORDER BY ep.stream_ordering ASC " | ||
| + "LIMIT 1" |
erikjohnston
Owner
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| } for row in after_read_receipt + no_read_receipt | ||
| ]) | ||
| @defer.inlineCallbacks | ||
| + def get_time_of_last_push_action_before(self, stream_ordering): | ||
| + def f(txn): | ||
| + sql = ( | ||
| + "SELECT e.received_ts " | ||
| + "FROM event_push_actions AS ep " | ||
| + "JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id " | ||
| + "WHERE ep.stream_ordering > ? " | ||
| + "ORDER BY ep.stream_ordering ASC " | ||
| + "LIMIT 1" | ||
| + ) | ||
| + txn.execute(sql, (stream_ordering,)) | ||
| + return txn.fetchone() | ||
| + result = yield self.runInteraction("get_time_of_last_push_action_before", f) | ||
| + defer.returnValue(result[0] if result is not None else None) |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| @@ -190,6 +212,26 @@ def f(txn): | ||
| ) | ||
| defer.returnValue(result[0] or 0) | ||
| + @defer.inlineCallbacks | ||
| + def get_time_of_latest_push_action_by_room_for_user(self, user_id): | ||
| + """ | ||
| + Returns only the received_ts of the last notification in each of the | ||
| + user's rooms, in a dict by room_id | ||
| + """ | ||
| + def f(txn): | ||
| + txn.execute( | ||
| + "SELECT ep.room_id, MAX(e.received_ts) " | ||
| + "FROM event_push_actions AS ep " | ||
| + "JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id " | ||
| + "GROUP BY ep.room_id" | ||
| + ) |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| " FROM events" | ||
| " NATURAL JOIN receipts_linearized WHERE receipt_type = 'm.read'" | ||
| " GROUP BY room_id, user_id" | ||
| ") AS rl " | ||
| + "NATURAL JOIN events e " |
erikjohnston
Owner
|
erikjohnston
and 1 other
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| } for row in after_read_receipt + no_read_receipt | ||
| ]) | ||
| @defer.inlineCallbacks | ||
| + def get_time_of_last_push_action_before(self, stream_ordering): | ||
| + def f(txn): | ||
| + sql = ( | ||
| + "SELECT e.received_ts " | ||
| + "FROM event_push_actions AS ep " | ||
| + "JOIN events e ON ep.room_id = e.room_id AND ep.event_id = e.event_id " |
|
|
erikjohnston
commented on the diff
Apr 29, 2016
synapse/storage/schema/delta/31/pusher_throttle.sql
| + * | ||
| + * Unless required by applicable law or agreed to in writing, software | ||
| + * distributed under the License is distributed on an "AS IS" BASIS, | ||
| + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| + * See the License for the specific language governing permissions and | ||
| + * limitations under the License. | ||
| + */ | ||
| + | ||
| + | ||
| +CREATE TABLE pusher_throttle( | ||
| + pusher BIGINT NOT NULL, | ||
| + room_id TEXT NOT NULL, | ||
| + last_sent_ts BIGINT, | ||
| + throttle_ms BIGINT, | ||
| + PRIMARY KEY (pusher, room_id) | ||
| +); |
|
|
erikjohnston
commented on the diff
Apr 29, 2016
synapse/util/presentable_names.py
| + # at this point we're going to need to search the state by all state keys | ||
| + # for an event type, so rearrange the data structure | ||
| + room_state_bytype = _state_as_two_level_dict(room_state) | ||
| + | ||
| + # right then, any aliases at all? | ||
| + if "m.room.aliases" in room_state_bytype: | ||
| + m_room_aliases = room_state_bytype["m.room.aliases"] | ||
| + if len(m_room_aliases.values()) > 0: | ||
| + first_alias_event = m_room_aliases.values()[0] | ||
| + if first_alias_event.content and first_alias_event.content["aliases"]: | ||
| + the_aliases = first_alias_event.content["aliases"] | ||
| + if len(the_aliases) > 0 and _looks_like_an_alias(the_aliases[0]): | ||
| + return the_aliases[0] | ||
| + | ||
| + if not fallback_to_members: | ||
| + return None |
dbkr
Member
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/util/presentable_names.py
| +# distributed under the License is distributed on an "AS IS" BASIS, | ||
| +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| +# See the License for the specific language governing permissions and | ||
| +# limitations under the License. | ||
| + | ||
| +import re | ||
| + | ||
| +# intentionally looser than what aliases we allow to be registered since | ||
| +# other HSes may allow aliases that we would not | ||
| +ALIAS_RE = re.compile(r"^#.*:.+$") | ||
| + | ||
| +ALL_ALONE = "Empty Room" | ||
| + | ||
| + | ||
| +def calculate_room_name(room_state, user_id, fallback_to_members=True): | ||
| + # does it have a name? |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/storage/event_push_actions.py
| @@ -190,6 +212,26 @@ def f(txn): | ||
| ) | ||
| defer.returnValue(result[0] or 0) | ||
| + @defer.inlineCallbacks | ||
| + def get_time_of_latest_push_action_by_room_for_user(self, user_id): | ||
| + """ | ||
| + Returns only the received_ts of the last notification in each of the | ||
| + user's rooms, in a dict by room_id | ||
| + """ | ||
| + def f(txn): | ||
| + txn.execute( | ||
| + "SELECT ep.room_id, MAX(e.received_ts) " |
|
|
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/emailpusher.py
| + def seconds_until(self, ts_msec): | ||
| + return (ts_msec - self.clock.time_msec()) / 1000 | ||
| + | ||
| + def get_room_throttle_ms(self, room_id): | ||
| + if room_id in self.throttle_params: | ||
| + return self.throttle_params[room_id]["throttle_ms"] | ||
| + else: | ||
| + return 0 | ||
| + | ||
| + def get_room_last_sent_ts(self, room_id): | ||
| + if room_id in self.throttle_params: | ||
| + return self.throttle_params[room_id]["last_sent_ts"] | ||
| + else: | ||
| + return 0 | ||
| + | ||
| + def room_ready_to_notify_at(self, room_id, last_notif_time): |
|
|
|
I'm not hugely comfortable with the usages of
Is there any transaction in replacing
|
|
Do we have different behaviour between highlight and non-highlight notifications? Will we want support for doing digests for non-highlight notificaitons? |
erikjohnston
commented on an outdated diff
Apr 29, 2016
synapse/push/emailpusher.py
| +import logging | ||
| + | ||
| +from synapse.util.metrics import Measure | ||
| +from synapse.util.logcontext import LoggingContext | ||
| + | ||
| +from mailer import Mailer | ||
| + | ||
| +logger = logging.getLogger(__name__) | ||
| + | ||
| +# The amount of time we always wait before ever emailing about a notification | ||
| +# (to give the user a chance to respond to other push or notice the window) | ||
| +DELAY_BEFORE_MAIL_MS = 2 * 60 * 1000 | ||
| + | ||
| +THROTTLE_START_MS = 2 * 60 * 1000 | ||
| +THROTTLE_MAX_MS = (2 * 60 * 1000) * (2 ** 11) # ~3 days | ||
| + |
|
|
dbkr
added some commits
Apr 29, 2016
All checks have passed
Some checks were not successful
|
On There's no difference currently between highlight and non-highlight notifications. Digest mails are coming later and will look broadly similar. |
Some checks were not successful
|
@NegativeMjark How does this interact with the Slaved stuff? |
|
You'll need to add the new database methods to https://github.com/matrix-org/synapse/blob/develop/synapse/app/pusher.py#L98 If you add a cached thing then you'll need to add it to the appropriate thing in https://github.com/matrix-org/synapse/blob/develop/synapse/replication/slave/storage/ |
dbkr
and others
added some commits
May 4, 2016
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
|
@NegativeMjark: How does packaging work with the manifest stuff? I note that all the static content is in a top level |
dbkr
and others
added some commits
May 4, 2016
All checks have passed
All checks have passed
Some checks were not successful
ara4n
added some commits
May 5, 2016
Some checks were not successful
All checks have passed
|
lgtm |
ara4n
and others
added some commits
May 5, 2016
All checks have passed
All checks have passed
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
Some checks were not successful
ara4n
merged commit fe97b81
into
develop
May 10, 2016
7 of 8 checks passed
Sytest SQLite (Merged PR)
Build finished.
Details
Flake8 + Packaging (Commit)
Build #634 origin/dbkr/email_notifs succeeded in 30 sec
Details
Flake8 + Packaging (Merged PR)
Build finished.
Details
Sytest Postgres (Commit)
Build #617 origin/dbkr/email_notifs succeeded in 5 min 0 sec
Details
Sytest Postgres (Merged PR)
Build finished.
Details
Sytest SQLite (Commit)
Build #625 origin/dbkr/email_notifs succeeded in 3 min 25 sec
Details
Unit Tests (Commit)
Build #678 origin/dbkr/email_notifs succeeded in 1 min 19 sec
Details
Unit Tests (Merged PR)
Build finished.
Details
This was referenced Jun 29, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
dbkr commentedApr 29, 2016
•
edited
Most functionality here is confined to adding an email pusher and other functionality used from it. Changes to the rest of the codebase are limited to exposing the previously private _filter_events_for_client and adding an email pusher for new users who register with an email.
This code will not be enabled unless the appropriate options are added to the config on existing installations. They are disabled by default on new installations.
Newly registered users will only have an email pusher set up if email notifications are enabled on the Home Server, so users will only ever start getting mails after registering or explicitly opting in.
Fixes vector-im/vector-web#108