New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix multiple N+1s in ConversationsController #25134
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ClearlyClaire
force-pushed
the
fixes/conversations-n+1
branch
from
May 25, 2023 14:58
ba78d05
to
805f601
Compare
ClearlyClaire
added
backport
Backport already-merged features to older branches
to backport
PR needed to be backported
performance
Runtime performance
and removed
backport
Backport already-merged features to older branches
labels
May 30, 2023
Gargron
approved these changes
Jun 1, 2023
supernovae
pushed a commit
to Universeodon-com/mastodon
that referenced
this pull request
Jun 7, 2023
ClearlyClaire
added a commit
to ClearlyClaire/mastodon
that referenced
this pull request
Jun 13, 2023
Fixes a regression introduced by mastodon#25134
ClearlyClaire
added a commit
to ClearlyClaire/mastodon
that referenced
this pull request
Jun 21, 2023
ClearlyClaire
added a commit
to ClearlyClaire/mastodon
that referenced
this pull request
Jun 21, 2023
ClearlyClaire
added a commit
to ClearlyClaire/mastodon
that referenced
this pull request
Jun 21, 2023
This was referenced Jun 21, 2023
ClearlyClaire
added a commit
that referenced
this pull request
Jul 6, 2023
ClearlyClaire
added a commit
that referenced
this pull request
Jul 6, 2023
ClearlyClaire
added a commit
that referenced
this pull request
Jul 6, 2023
IsabelaY
pushed a commit
to IsabelaY/mastodon
that referenced
this pull request
Jul 6, 2023
F-Finene
pushed a commit
to F-Finene/mastodon
that referenced
this pull request
Jul 7, 2023
(cherry picked from commit 3e1724e)
skerit
pushed a commit
to 11ways/mastodon
that referenced
this pull request
Jul 7, 2023
nrdufour
added a commit
to nrdufour/home-ops
that referenced
this pull request
Aug 1, 2023
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/mastodon/mastodon](https://github.com/mastodon/mastodon) | patch | `v4.1.2` -> `v4.1.6` | --- ### Release Notes <details> <summary>mastodon/mastodon (ghcr.io/mastodon/mastodon)</summary> ### [`v4.1.6`](https://github.com/mastodon/mastodon/releases/tag/v4.1.6) [Compare Source](mastodon/mastodon@v4.1.5...v4.1.6) <h1><picture> <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true"> <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true"> <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34"> </picture></h1> >⚠️ We recently released critical security updates, so if you are still using 4.1.2 or below, 4.0.4 or below, or 3.5.8 or below, please update as soon as possible (see the release notes for [v4.1.4](https://github.com/mastodon/mastodon/releases/tag/v4.1.4)). #### Upgrade overview ℹ️ Requires streaming API restart For more information, scroll down to the upgrade instructions section. #### Changelog ##### Fixed - Fix memory leak in streaming server ([ThisIsMissEm](mastodon/mastodon#26228)) - Fix wrong filters sometimes applying in streaming ([ClearlyClaire](mastodon/mastodon#26159), [ThisIsMissEm](mastodon/mastodon#26213), [renchap](mastodon/mastodon#26233)) - Fix incorrect connect timeout in outgoing requests ([ClearlyClaire](mastodon/mastodon#26116)) #### Upgrade notes To get the code for v4.1.6, use `git fetch && git checkout v4.1.6`. > As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: `docker exec mastodon_db_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump` ##### Dependencies External dependencies have not changed compared to v4.1.5, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is: - Ruby: 2.7 to 3.0 - PostgreSQL: 9.5 or newer - Elasticsearch (optional, for full-text search): 7.x - Redis: 4 or newer - Node: >= 14, < 18 - ImageMagick: 6.9.7-7 or newer > If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](mastodon/mastodon#25776). ##### Update steps The following instructions are for updating from 4.1.5. If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations. **Non-Docker only:** 1. Install dependencies: `bundle install` and `yarn install` **Both Docker and non-Docker:** 1. Restart all Mastodon processes ### [`v4.1.5`](https://github.com/mastodon/mastodon/releases/tag/v4.1.5) [Compare Source](mastodon/mastodon@v4.1.4...v4.1.5) <h1><picture> <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true"> <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true"> <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34"> </picture></h1> >⚠️ We recently released critical security updates, so if you are still using 4.1.2 or below, 4.0.4 or below, or 3.5.8 or below, please update as soon as possible (see the release notes for [v4.1.4](https://github.com/mastodon/mastodon/releases/tag/v4.1.4)). #### Changelog ##### Added - Add check preventing Sidekiq workers from running with Makara configured ([ClearlyClaire](mastodon/mastodon#25850)) ##### Changed - Change request timeout handling to use a longer deadline ([ClearlyClaire](mastodon/mastodon#26055)) ##### Fixed - Fix moderation interface for remote instances with a .zip TLD ([ClearlyClaire](mastodon/mastodon#25885)) - Fix remote accounts being possibly persisted to database with incomplete protocol values ([ClearlyClaire](mastodon/mastodon#25886)) - Fix trending publishers table not rendering correctly on narrow screens ([vmstan](mastodon/mastodon#25945)) ##### Security - Fix CSP headers being unintentionally wide ([ClearlyClaire](mastodon/mastodon#26105)) #### Upgrade notes To get the code for v4.1.5, use `git fetch && git checkout v4.1.5`. > As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: `docker exec mastodon_db_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump` ##### Dependencies External dependencies have not changed compared to v4.1.4, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is: - Ruby: 2.7 to 3.0 - PostgreSQL: 9.5 or newer - Elasticsearch (optional, for full-text search): 7.x - Redis: 4 or newer - Node: >= 14, < 18 - ImageMagick: 6.9.7-7 or newer > If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](mastodon/mastodon#25776). ##### Update steps The following instructions are for updating from 4.1.4. If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations. **Non-Docker only:** 1. Install dependencies: `bundle install` and `yarn install` **Both Docker and non-Docker:** 1. Restart all Mastodon processes ### [`v4.1.4`](https://github.com/mastodon/mastodon/releases/tag/v4.1.4) [Compare Source](mastodon/mastodon@v4.1.3...v4.1.4) <h1><picture> <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true"> <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true"> <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34"> </picture></h1> > This release addresses a few issues that were missed in the last security update and includes changelogs for both updates. > >⚠️ It is a follow-up to the important 4.1.3 security release fixing multiple **critical security issues** (CVE-2023-36460, CVE-2023-36459). > > Corresponding security releases are available for the [4.0.x branch](https://github.com/mastodon/mastodon/releases/tag/v4.0.6) and the [3.5.x branch](https://github.com/mastodon/mastodon/releases/tag/v3.5.10). > If you are using nightly builds, **do not use this release** but update to `nightly-2023-07-07-v4.1.4` or newer instead. If you are on the `main` branch, update to the latest commit. #### Upgrade overview This release contains upgrade notes that deviate from the norm: ℹ️ Requires streaming API restart ℹ️ There are suggested reverse proxy configuration changes :warning: The minimal supported ImageMagick version has been bumped to 6.9.7-7 For more information, scroll down to the upgrade instructions section. #### Changelog (v4.1.4) ##### Fixed - Fix branding:generate_app_icons failing because of disallowed ICO coder ([ClearlyClaire](mastodon/mastodon#25794)) - Fix crash in admin interface when viewing a remote user with verified links ([ClearlyClaire](mastodon/mastodon#25796)) - Fix processing of media files with unusual names ([ClearlyClaire](mastodon/mastodon#25788)) #### Changelog (v4.1.3) ##### Added - Add fallback redirection when getting a webfinger query `LOCAL_DOMAIN@LOCAL_DOMAIN` ([ClearlyClaire](mastodon/mastodon#23600)) ##### Changed - Change OpenGraph-based embeds to allow fullscreen ([ClearlyClaire](mastodon/mastodon#25058)) - Change AccessTokensVacuum to also delete expired tokens ([ClearlyClaire](mastodon/mastodon#24868)) - Change profile updates to be sent to recently-mentioned servers ([ClearlyClaire](mastodon/mastodon#24852)) - Change automatic post deletion thresholds and load detection ([ClearlyClaire](mastodon/mastodon#24614)) - Change `/api/v1/statuses/:id/history` to always return at least one item ([ClearlyClaire](mastodon/mastodon#25510)) - Change auto-linking to allow carets in URL query params ([renchap](mastodon/mastodon#25216)) ##### Removed - Remove invalid `X-Frame-Options: ALLOWALL` ([ClearlyClaire](mastodon/mastodon#25070)) ##### Fixed - Fix wrong view being displayed when a webhook fails validation ([ClearlyClaire](mastodon/mastodon#25464)) - Fix soft-deleted post cleanup scheduler overwhelming the streaming server ([ThisIsMissEm](mastodon/mastodon#25519)) - Fix incorrect pagination headers in `/api/v2/admin/accounts` ([danielmbrasil](mastodon/mastodon#25477)) - Fix multiple inefficiencies in automatic post cleanup worker ([ClearlyClaire](mastodon/mastodon#24607), [ClearlyClaire](mastodon/mastodon#24785), [ClearlyClaire](mastodon/mastodon#24840)) - Fix performance of streaming by parsing message JSON once ([ThisIsMissEm](mastodon/mastodon#25278), [ThisIsMissEm](mastodon/mastodon#25361)) - Fix CSP headers when `S3_ALIAS_HOST` includes a path component ([ClearlyClaire](mastodon/mastodon#25273)) - Fix `tootctl accounts approve --number N` not aproving N earliest registrations ([danielmbrasil](mastodon/mastodon#24605)) - Fix reports not being closed when performing batch suspensions ([ClearlyClaire](mastodon/mastodon#24988)) - Fix being able to vote on your own polls ([ClearlyClaire](mastodon/mastodon#25015)) - Fix race condition when reblogging a status ([ClearlyClaire](mastodon/mastodon#25016)) - Fix “Authorized applications” inefficiently and incorrectly getting last use date ([ClearlyClaire](mastodon/mastodon#25060)) - Fix “Authorized applications” crashing when listing apps with certain admin API scopes ([ClearlyClaire](mastodon/mastodon#25713)) - Fix multiple N+1s in ConversationsController ([ClearlyClaire](mastodon/mastodon#25134), [ClearlyClaire](mastodon/mastodon#25399), [ClearlyClaire](mastodon/mastodon#25499)) - Fix user archive takeouts when using OpenStack Swift ([ClearlyClaire](mastodon/mastodon#24431)) - Fix searching for remote content by URL not working under certain conditions ([ClearlyClaire](mastodon/mastodon#25637)) - Fix inefficiencies in indexing content for search ([VyrCossont](mastodon/mastodon#24285), [VyrCossont](mastodon/mastodon#24342)) ##### Security - Add finer permission requirements for managing webhooks ([ClearlyClaire](mastodon/mastodon#25463)) - Update dependencies - Add hardening headers for user-uploaded files ([ClearlyClaire](mastodon/mastodon#25756)) - Fix verified links possibly hiding important parts of the URL (CVE-2023-36462) - Fix timeout handling of outbound HTTP requests (CVE-2023-36461) - Fix arbitrary file creation through media processing (CVE-2023-36460) - Fix possible XSS in preview cards (CVE-2023-36459) #### Upgrade notes To get the code for v4.1.4, use `git fetch && git checkout v4.1.4`. > As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: docker exec mastodon_db\_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump ##### Dependencies Apart from ImageMagick, external dependencies have not changed compared to v4.1.2, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is: - Ruby: 2.7 to 3.0 - PostgreSQL: 9.5 or newer - Elasticsearch (optional, for full-text search): 7.x - Redis: 4 or newer - Node: >= 14, < 18 - ImageMagick: 6.9.7-7 or newer If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more informations and ways to fix it [on this page](mastodon/mastodon#25776). ##### Update steps The following instructions are for updating from 4.1.2. If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations. **Non-Docker only:** 1. Install dependencies: `bundle install` and `yarn install` **Both Docker and non-Docker:** ℹ️ The recommended configuration for reverse proxies has been updated. Unlike updating Mastodon itself, this is not urgent, but hardening. The change is about setting `Content-Security-Policy: default-src 'none'; form-action 'none'` and `X-Content-Type-Options: nosniff` on assets. Check `dist/nginx.conf` for more information, and [the documentation](https://docs.joinmastodon.org/admin/optional/object-storage-proxy/) if you are proxying external object storage. 1. Restart all Mastodon processes ### [`v4.1.3`](https://github.com/mastodon/mastodon/releases/tag/v4.1.3) [Compare Source](mastodon/mastodon@v4.1.2...v4.1.3) <h1><picture> <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true"> <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true"> <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34"> </picture></h1> >⚠️ This release is an important security release fixing multiple **critical security issues** (CVE-2023-36460, CVE-2023-36459). > > Corresponding security releases are available for the [4.0.x branch](https://github.com/mastodon/mastodon/releases/tag/v4.0.5) and the [3.5.x branch](https://github.com/mastodon/mastodon/releases/tag/v3.5.9). > If you are using nightly builds, **do not use this release** but update to `nightly-2023-07-06-security` or newer instead. If you are on the `main` branch, update to the latest commit. #### Upgrade overview This release contains upgrade notes that deviate from the norm: ℹ️ Requires streaming API restart ℹ️ There are suggested reverse proxy configuration changes :warning: The minimal supported ImageMagick version has been bumped to 6.9.7-7 For more information, scroll down to the upgrade instructions section. #### Changelog ##### Added - Add fallback redirection when getting a webfinger query `LOCAL_DOMAIN@LOCAL_DOMAIN` ([ClearlyClaire](mastodon/mastodon#23600)) ##### Changed - Change OpenGraph-based embeds to allow fullscreen ([ClearlyClaire](mastodon/mastodon#25058)) - Change AccessTokensVacuum to also delete expired tokens ([ClearlyClaire](mastodon/mastodon#24868)) - Change profile updates to be sent to recently-mentioned servers ([ClearlyClaire](mastodon/mastodon#24852)) - Change automatic post deletion thresholds and load detection ([ClearlyClaire](mastodon/mastodon#24614)) - Change `/api/v1/statuses/:id/history` to always return at least one item ([ClearlyClaire](mastodon/mastodon#25510)) - Change auto-linking to allow carets in URL query params ([renchap](mastodon/mastodon#25216)) ##### Removed - Remove invalid `X-Frame-Options: ALLOWALL` ([ClearlyClaire](mastodon/mastodon#25070)) ##### Fixed - Fix wrong view being displayed when a webhook fails validation ([ClearlyClaire](mastodon/mastodon#25464)) - Fix soft-deleted post cleanup scheduler overwhelming the streaming server ([ThisIsMissEm](mastodon/mastodon#25519)) - Fix incorrect pagination headers in `/api/v2/admin/accounts` ([danielmbrasil](mastodon/mastodon#25477)) - Fix multiple inefficiencies in automatic post cleanup worker ([ClearlyClaire](mastodon/mastodon#24607), [ClearlyClaire](mastodon/mastodon#24785), [ClearlyClaire](mastodon/mastodon#24840)) - Fix performance of streaming by parsing message JSON once ([ThisIsMissEm](mastodon/mastodon#25278), [ThisIsMissEm](mastodon/mastodon#25361)) - Fix CSP headers when `S3_ALIAS_HOST` includes a path component ([ClearlyClaire](mastodon/mastodon#25273)) - Fix `tootctl accounts approve --number N` not aproving N earliest registrations ([danielmbrasil](mastodon/mastodon#24605)) - Fix reports not being closed when performing batch suspensions ([ClearlyClaire](mastodon/mastodon#24988)) - Fix being able to vote on your own polls ([ClearlyClaire](mastodon/mastodon#25015)) - Fix race condition when reblogging a status ([ClearlyClaire](mastodon/mastodon#25016)) - Fix “Authorized applications” inefficiently and incorrectly getting last use date ([ClearlyClaire](mastodon/mastodon#25060)) - Fix “Authorized applications” crashing when listing apps with certain admin API scopes ([ClearlyClaire](mastodon/mastodon#25713)) - Fix multiple N+1s in ConversationsController ([ClearlyClaire](mastodon/mastodon#25134), [ClearlyClaire](mastodon/mastodon#25399), [ClearlyClaire](mastodon/mastodon#25499)) - Fix user archive takeouts when using OpenStack Swift ([ClearlyClaire](mastodon/mastodon#24431)) - Fix searching for remote content by URL not working under certain conditions ([ClearlyClaire](mastodon/mastodon#25637)) - Fix inefficiencies in indexing content for search ([VyrCossont](mastodon/mastodon#24285), [VyrCossont](mastodon/mastodon#24342)) ##### Security - Add finer permission requirements for managing webhooks ([ClearlyClaire](mastodon/mastodon#25463)) - Update dependencies - Add hardening headers for user-uploaded files ([ClearlyClaire](mastodon/mastodon#25756)) - Fix verified links possibly hiding important parts of the URL (CVE-2023-36462) - Fix timeout handling of outbound HTTP requests (CVE-2023-36461) - Fix arbitrary file creation through media processing (CVE-2023-36460) - Fix possible XSS in preview cards (CVE-2023-36459) #### Upgrade notes To get the code for v4.1.3, use `git fetch && git checkout v4.1.3`. > As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: docker exec mastodon_db\_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump ##### Dependencies Apart from ImageMagick, external dependencies have not changed compared to v4.1.2, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is: - Ruby: 2.7 to 3.0 - PostgreSQL: 9.5 or newer - Elasticsearch (optional, for full-text search): 7.x - Redis: 4 or newer - Node: >= 14, < 18 - ImageMagick: 6.9.7-7 or newer If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more informations and ways to fix it [on this page](mastodon/mastodon#25776). ##### Update steps The following instructions are for updating from 4.1.2. If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations. **Non-Docker only:** 1. Install dependencies: `bundle install` and `yarn install` **Both Docker and non-Docker:** ℹ️ The recommended configuration for reverse proxies has been updated. Unlike updating Mastodon itself, this is not urgent, but hardening. The change is about setting `Content-Security-Policy: default-src 'none'; form-action 'none'` and `X-Content-Type-Options: nosniff` on assets. Check `dist/nginx.conf` for more information, and [the documentation](https://docs.joinmastodon.org/admin/optional/object-storage-proxy/) if you are proxying external object storage. 1. Restart all Mastodon processes </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4yMy4yIiwidXBkYXRlZEluVmVyIjoiMzYuMjMuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Reviewed-on: https://git.home/nrdufour/home-ops/pulls/17 Co-authored-by: Renovate <renovate@ptinem.io> Co-committed-by: Renovate <renovate@ptinem.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes MAS-154