Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Track DB txn times w/ two counters, not histogram #13342

Merged
merged 2 commits into from
Jul 21, 2022
Merged

Conversation

DMRobertson
Copy link
Contributor

By my reckoning, this should mean that each txn desc produces 2 time
series, down from 17. If we don't like this, we could keep the histogram
but configure it with coarser buckets.

This saves on storage and CPU cycles. We don't use the full histogram distribution in our dashboards, so we think this is safe to drop.

Closes #11081.

David Robertson added 2 commits July 20, 2022 18:23
By my reckoning, this should mean that each txn `desc` produces 2 time
series, down from 17. If we don't like this, we could keep the histogram
but configure it with coarser buckets.

Closes #11081.
@@ -0,0 +1 @@
When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure if this should be .feature?

@DMRobertson DMRobertson marked this pull request as ready for review July 20, 2022 17:29
@DMRobertson DMRobertson requested a review from a team as a code owner July 20, 2022 17:29
@DMRobertson DMRobertson merged commit 34949ea into develop Jul 21, 2022
@DMRobertson DMRobertson deleted the dmr/txn_metrics branch July 21, 2022 12:23
Alladin9393 added a commit to BitorbitLabs/synapse that referenced this pull request Aug 12, 2022
Synapse 1.64.0 (2022-08-02)
===========================

No significant changes since 1.64.0rc2.

Deprecation Warning
-------------------

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406))

Synapse 1.64.0rc1 (2022-07-26)
==============================

This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.

Features
--------

- Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038))
- Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205))
- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208))
- Add support for room version 10. ([\matrix-org#13220](matrix-org#13220))
- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276))
- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263))
- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270))
- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278))
- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296))
- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350))
- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352))

Improved Documentation
----------------------

- Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231))
- Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261))
- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271))
- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314))
- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333))
- Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338))
- Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344))

Deprecations and Removals
-------------------------

- Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967))
- Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192))
- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239))
- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326))

Internal Changes
----------------

- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942))
- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943))
- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094))
- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172))
- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175))
- Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198))
- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215))
- Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218))
- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224))
- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240))
- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308))
- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251))
- Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257))
- Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258))
- Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260))
- Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266))
- Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274))
- Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279))
- Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307))
- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285))
- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292))
- Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297))
- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299))
- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300))
- Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303))
- Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310))
- Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311))
- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313))
- Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318))
- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320))
- Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323))
- Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324))
- Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362))
- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349))
- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342))
- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 23, 2022
Synapse 1.64.0 (2022-08-02)
===========================

No significant changes since 1.64.0rc2.

Deprecation Warning
-------------------

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406))

Synapse 1.64.0rc1 (2022-07-26)
==============================

This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.

Features
--------

- Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038))
- Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205))
- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208))
- Add support for room version 10. ([\matrix-org#13220](matrix-org#13220))
- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276))
- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263))
- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270))
- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278))
- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296))
- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350))
- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352))

Improved Documentation
----------------------

- Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231))
- Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261))
- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271))
- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314))
- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333))
- Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338))
- Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344))

Deprecations and Removals
-------------------------

- Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967))
- Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192))
- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239))
- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326))

Internal Changes
----------------

- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942))
- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943))
- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094))
- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172))
- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175))
- Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198))
- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215))
- Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218))
- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224))
- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240))
- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308))
- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251))
- Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257))
- Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258))
- Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260))
- Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266))
- Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274))
- Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279))
- Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307))
- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285))
- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292))
- Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297))
- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299))
- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300))
- Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303))
- Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310))
- Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311))
- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313))
- Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318))
- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320))
- Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323))
- Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324))
- Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362))
- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349))
- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342))
- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmLo+zIACgkQLS76LzL7
# 4EehbRAAronXZtWM+ViMxPsiDj70KXYOKK117pGXK5XGf3Tyqb/vExA7c7bfimyW
# d3FW855fe27AMsSfcMGDpxhggVa8sZDSdvQumt5jqDXrzC348mW/FYtgcYOxkoIa
# Hh2/7V26CxWFsv8eVF3hwpualelT9lp2sedWXCQtdAkcQoWs2JwBsnoxSDliDZHg
# jc4mBFBAkah5CJ3bcZuZXRsr9doKxDOAXUv19RXhdwEGO82mpSbwQ8P0mcw2S8zr
# aAVza7jkVAza6ahg9qE0lMpi8uYE9/mt5JBnfrv/JxC7ZZfBg9jyHKaxFrzpjFsj
# 3g0jhqzcNxRskD1sk1GKGVy7D9oTg1WVpii5l3M93KguSDLKxomouhgekWOxMPBe
# 43xVdDI13ohsex+1QBnGnTSP7jZcfODnfvzSdyHQv6ef4k+OplRdfMA0QjkUcI5j
# ocJlkm2D02vw1mnU3hHNdw9ri3vkaS1Qwfsz3ZEYgn6OcZOeKAWn351WMXF/F1fm
# HYeQ5uMud+i+EekBtR8Op9ZICHt9Ogp49172enlSGzeyeD3yUk5HMAMrzJfmsp3W
# /LCCONkRrV+R8TRByUQE9YtqxUgn+eSgB5Ew/2C/WB54pZHtco+rPqkY1Bhan4QJ
# LeZTuzDKeXzgho1D5b4quEC2AWAqz3GeIvEVuOZCt8rJoMMRslg=
# =RRRX
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Aug  2 11:23:46 2022 BST
# gpg:                using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047
# gpg: Can't check signature: No public key

# Conflicts:
#	synapse/rest/client/read_marker.py
#	synapse/rest/client/receipts.py
#	synapse/storage/databases/main/events_worker.py
#	synapse/storage/databases/main/purge_events.py
#	tests/rest/client/test_rooms.py
#	tests/storage/test_event_push_actions.py
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

synapse_storage_transaction_time_bucket prometheus metric has too high cardinality due to desc label
2 participants