Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Redis client and type errors introduced in #24285 #24342

Merged
merged 1 commit into from Apr 1, 2023

Conversation

VyrCossont
Copy link
Contributor

Three things:

  • The result of sscan_each is an iterator of single IDs, not an iterator of ID batches, and fails trying to each_slice a string in ids if run with a non-empty set of any kind of ID.
  • Fixing that triggers an exception because redis can't be used inside a pipelined block (see below)
  • Once that's fixed, running with a non-empty set of status IDs where the status contains a hashtag will trigger an exception but in a different place:
Pipelining commands on a Redis instance is deprecated and will be removed in Redis 5.0.0.

redis.pipelined do
  redis.get("key")
end

should be replaced by

redis.pipelined do |pipeline|
  pipeline.get("key")
end
(called from /Users/user/Projects/mastodon/app/workers/scheduler/indexing_scheduler.rb:18:in `block (3 levels) in perform'}

or

(called from /Users/user/Projects/mastodon/app/models/trends/history.rb:18:in `block in accounts'}

I think what's going on here is that this setup means that the tag indexing queue is also non-empty, and the tag index contains trend history, and calculating trend history involves another Redis call, which cannot be nested in a pipelined block. Moving the type.import! outside the pipelined block fixes this.

@nschonni
Copy link
Contributor

nschonni commented Apr 1, 2023

@mjankowski was one of your Spec PRs adding coverage for this area?

Edit: rebased the CodeCov PR and it doesn't look like it's currently covered https://app.codecov.io/gh/mastodon/mastodon/blob/jest-coverage/app/workers/scheduler/indexing_scheduler.rb

@Gargron Gargron merged commit d5ad8b6 into mastodon:main Apr 1, 2023
26 checks passed
@VyrCossont
Copy link
Contributor Author

Deeper test coverage for search stuff would be great. Even if it's not covered in that PR, maybe there are other indexing or Redis-related tests? I'll have a look through it.

arachnist pushed a commit to arachnist/mastodon that referenced this pull request Apr 4, 2023
@ClearlyClaire ClearlyClaire added the to backport PR needed to be backported label Jun 7, 2023
ClearlyClaire pushed a commit to ClearlyClaire/mastodon that referenced this pull request Jul 4, 2023
ClearlyClaire pushed a commit to ClearlyClaire/mastodon that referenced this pull request Jul 4, 2023
ClearlyClaire pushed a commit to ClearlyClaire/mastodon that referenced this pull request Jul 4, 2023
IsabelaY pushed a commit to IsabelaY/mastodon that referenced this pull request Jul 6, 2023
F-Finene pushed a commit to F-Finene/mastodon that referenced this pull request Jul 7, 2023
skerit pushed a commit to 11ways/mastodon that referenced this pull request Jul 7, 2023
@ClearlyClaire ClearlyClaire removed the to backport PR needed to be backported label Jul 18, 2023
nrdufour added a commit to nrdufour/home-ops that referenced this pull request Aug 1, 2023
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/mastodon/mastodon](https://github.com/mastodon/mastodon) | patch | `v4.1.2` -> `v4.1.6` |

---

### Release Notes

<details>
<summary>mastodon/mastodon (ghcr.io/mastodon/mastodon)</summary>

### [`v4.1.6`](https://github.com/mastodon/mastodon/releases/tag/v4.1.6)

[Compare Source](mastodon/mastodon@v4.1.5...v4.1.6)

<h1><picture>
  <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true">
  <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true">
  <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34">
</picture></h1>

> ⚠️ We recently released critical security updates, so if you are still using 4.1.2 or below, 4.0.4 or below, or 3.5.8 or below, please update as soon as possible (see the release notes for [v4.1.4](https://github.com/mastodon/mastodon/releases/tag/v4.1.4)).

#### Upgrade overview

ℹ️ Requires streaming API restart

For more information, scroll down to the upgrade instructions section.

#### Changelog

##### Fixed

-   Fix memory leak in streaming server ([ThisIsMissEm](mastodon/mastodon#26228))
-   Fix wrong filters sometimes applying in streaming ([ClearlyClaire](mastodon/mastodon#26159), [ThisIsMissEm](mastodon/mastodon#26213), [renchap](mastodon/mastodon#26233))
-   Fix incorrect connect timeout in outgoing requests ([ClearlyClaire](mastodon/mastodon#26116))

#### Upgrade notes

To get the code for v4.1.6, use `git fetch && git checkout v4.1.6`.

> As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: `docker exec mastodon_db_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump`

##### Dependencies

External dependencies have not changed compared to v4.1.5, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is:

-   Ruby: 2.7 to 3.0
-   PostgreSQL: 9.5 or newer
-   Elasticsearch (optional, for full-text search): 7.x
-   Redis: 4 or newer
-   Node: >= 14, < 18
-   ImageMagick: 6.9.7-7 or newer

> If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](mastodon/mastodon#25776).

##### Update steps

The following instructions are for updating from 4.1.5.

If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations.

**Non-Docker only:**

1.  Install dependencies: `bundle install` and `yarn install`

**Both Docker and non-Docker:**

1.  Restart all Mastodon processes

### [`v4.1.5`](https://github.com/mastodon/mastodon/releases/tag/v4.1.5)

[Compare Source](mastodon/mastodon@v4.1.4...v4.1.5)

<h1><picture>
  <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true">
  <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true">
  <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34">
</picture></h1>

> ⚠️ We recently released critical security updates, so if you are still using 4.1.2 or below, 4.0.4 or below, or 3.5.8 or below, please update as soon as possible (see the release notes for [v4.1.4](https://github.com/mastodon/mastodon/releases/tag/v4.1.4)).

#### Changelog

##### Added

-   Add check preventing Sidekiq workers from running with Makara configured ([ClearlyClaire](mastodon/mastodon#25850))

##### Changed

-   Change request timeout handling to use a longer deadline ([ClearlyClaire](mastodon/mastodon#26055))

##### Fixed

-   Fix moderation interface for remote instances with a .zip TLD ([ClearlyClaire](mastodon/mastodon#25885))
-   Fix remote accounts being possibly persisted to database with incomplete protocol values ([ClearlyClaire](mastodon/mastodon#25886))
-   Fix trending publishers table not rendering correctly on narrow screens ([vmstan](mastodon/mastodon#25945))

##### Security

-   Fix CSP headers being unintentionally wide ([ClearlyClaire](mastodon/mastodon#26105))

#### Upgrade notes

To get the code for v4.1.5, use `git fetch && git checkout v4.1.5`.

> As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: `docker exec mastodon_db_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump`

##### Dependencies

External dependencies have not changed compared to v4.1.4, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is:

-   Ruby: 2.7 to 3.0
-   PostgreSQL: 9.5 or newer
-   Elasticsearch (optional, for full-text search): 7.x
-   Redis: 4 or newer
-   Node: >= 14, < 18
-   ImageMagick: 6.9.7-7 or newer

> If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](mastodon/mastodon#25776).

##### Update steps

The following instructions are for updating from 4.1.4.

If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations.

**Non-Docker only:**

1.  Install dependencies: `bundle install` and `yarn install`

**Both Docker and non-Docker:**

1.  Restart all Mastodon processes

### [`v4.1.4`](https://github.com/mastodon/mastodon/releases/tag/v4.1.4)

[Compare Source](mastodon/mastodon@v4.1.3...v4.1.4)

<h1><picture>
  <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true">
  <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true">
  <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34">
</picture></h1>

> This release addresses a few issues that were missed in the last security update and includes changelogs for both updates.
>
> ⚠️ It is a follow-up to the important 4.1.3 security release fixing multiple **critical security issues** (CVE-2023-36460, CVE-2023-36459).
>
> Corresponding security releases are available for the [4.0.x branch](https://github.com/mastodon/mastodon/releases/tag/v4.0.6) and the [3.5.x branch](https://github.com/mastodon/mastodon/releases/tag/v3.5.10).
> If you are using nightly builds, **do not use this release** but update to `nightly-2023-07-07-v4.1.4` or newer instead. If you are on the `main` branch, update to the latest commit.

#### Upgrade overview

This release contains upgrade notes that deviate from the norm:

ℹ️ Requires streaming API restart
ℹ️ There are suggested reverse proxy configuration changes
:warning: The minimal supported ImageMagick version has been bumped to 6.9.7-7

For more information, scroll down to the upgrade instructions section.

#### Changelog (v4.1.4)

##### Fixed

-   Fix branding:generate_app_icons failing because of disallowed ICO coder ([ClearlyClaire](mastodon/mastodon#25794))
-   Fix crash in admin interface when viewing a remote user with verified links ([ClearlyClaire](mastodon/mastodon#25796))
-   Fix processing of media files with unusual names ([ClearlyClaire](mastodon/mastodon#25788))

#### Changelog (v4.1.3)

##### Added

-   Add fallback redirection when getting a webfinger query `LOCAL_DOMAIN@LOCAL_DOMAIN` ([ClearlyClaire](mastodon/mastodon#23600))

##### Changed

-   Change OpenGraph-based embeds to allow fullscreen ([ClearlyClaire](mastodon/mastodon#25058))
-   Change AccessTokensVacuum to also delete expired tokens ([ClearlyClaire](mastodon/mastodon#24868))
-   Change profile updates to be sent to recently-mentioned servers ([ClearlyClaire](mastodon/mastodon#24852))
-   Change automatic post deletion thresholds and load detection ([ClearlyClaire](mastodon/mastodon#24614))
-   Change `/api/v1/statuses/:id/history` to always return at least one item ([ClearlyClaire](mastodon/mastodon#25510))
-   Change auto-linking to allow carets in URL query params ([renchap](mastodon/mastodon#25216))

##### Removed

-   Remove invalid `X-Frame-Options: ALLOWALL` ([ClearlyClaire](mastodon/mastodon#25070))

##### Fixed

-   Fix wrong view being displayed when a webhook fails validation ([ClearlyClaire](mastodon/mastodon#25464))
-   Fix soft-deleted post cleanup scheduler overwhelming the streaming server ([ThisIsMissEm](mastodon/mastodon#25519))
-   Fix incorrect pagination headers in `/api/v2/admin/accounts` ([danielmbrasil](mastodon/mastodon#25477))
-   Fix multiple inefficiencies in automatic post cleanup worker ([ClearlyClaire](mastodon/mastodon#24607), [ClearlyClaire](mastodon/mastodon#24785), [ClearlyClaire](mastodon/mastodon#24840))
-   Fix performance of streaming by parsing message JSON once ([ThisIsMissEm](mastodon/mastodon#25278), [ThisIsMissEm](mastodon/mastodon#25361))
-   Fix CSP headers when `S3_ALIAS_HOST` includes a path component ([ClearlyClaire](mastodon/mastodon#25273))
-   Fix `tootctl accounts approve --number N` not aproving N earliest registrations ([danielmbrasil](mastodon/mastodon#24605))
-   Fix reports not being closed when performing batch suspensions ([ClearlyClaire](mastodon/mastodon#24988))
-   Fix being able to vote on your own polls ([ClearlyClaire](mastodon/mastodon#25015))
-   Fix race condition when reblogging a status ([ClearlyClaire](mastodon/mastodon#25016))
-   Fix “Authorized applications” inefficiently and incorrectly getting last use date ([ClearlyClaire](mastodon/mastodon#25060))
-   Fix “Authorized applications” crashing when listing apps with certain admin API scopes ([ClearlyClaire](mastodon/mastodon#25713))
-   Fix multiple N+1s in ConversationsController ([ClearlyClaire](mastodon/mastodon#25134), [ClearlyClaire](mastodon/mastodon#25399), [ClearlyClaire](mastodon/mastodon#25499))
-   Fix user archive takeouts when using OpenStack Swift ([ClearlyClaire](mastodon/mastodon#24431))
-   Fix searching for remote content by URL not working under certain conditions ([ClearlyClaire](mastodon/mastodon#25637))
-   Fix inefficiencies in indexing content for search ([VyrCossont](mastodon/mastodon#24285), [VyrCossont](mastodon/mastodon#24342))

##### Security

-   Add finer permission requirements for managing webhooks ([ClearlyClaire](mastodon/mastodon#25463))
-   Update dependencies
-   Add hardening headers for user-uploaded files ([ClearlyClaire](mastodon/mastodon#25756))
-   Fix verified links possibly hiding important parts of the URL (CVE-2023-36462)
-   Fix timeout handling of outbound HTTP requests (CVE-2023-36461)
-   Fix arbitrary file creation through media processing (CVE-2023-36460)
-   Fix possible XSS in preview cards (CVE-2023-36459)

#### Upgrade notes

To get the code for v4.1.4, use `git fetch && git checkout v4.1.4`.

> As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: docker exec mastodon_db\_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump

##### Dependencies

Apart from ImageMagick, external dependencies have not changed compared to v4.1.2, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is:

-   Ruby: 2.7 to 3.0
-   PostgreSQL: 9.5 or newer
-   Elasticsearch (optional, for full-text search): 7.x
-   Redis: 4 or newer
-   Node: >= 14, < 18
-   ImageMagick: 6.9.7-7 or newer

If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more informations and ways to fix it [on this page](mastodon/mastodon#25776).

##### Update steps

The following instructions are for updating from 4.1.2.

If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations.

**Non-Docker only:**

1.  Install dependencies: `bundle install` and `yarn install`

**Both Docker and non-Docker:**

ℹ️ The recommended configuration for reverse proxies has been updated. Unlike updating Mastodon itself, this is not urgent, but hardening. The change is about setting `Content-Security-Policy: default-src 'none'; form-action 'none'` and `X-Content-Type-Options: nosniff` on assets. Check `dist/nginx.conf` for more information, and [the documentation](https://docs.joinmastodon.org/admin/optional/object-storage-proxy/) if you are proxying external object storage.

1.  Restart all Mastodon processes

### [`v4.1.3`](https://github.com/mastodon/mastodon/releases/tag/v4.1.3)

[Compare Source](mastodon/mastodon@v4.1.2...v4.1.3)

<h1><picture>
  <source media="(prefers-color-scheme: dark)" srcset="./lib/assets/wordmark.dark.png?raw=true">
  <source media="(prefers-color-scheme: light)" srcset="./lib/assets/wordmark.light.png?raw=true">
  <img alt="Mastodon" src="./lib/assets/wordmark.light.png?raw=true" height="34">
</picture></h1>

> ⚠️ This release is an important security release fixing multiple **critical security issues** (CVE-2023-36460, CVE-2023-36459).
>
> Corresponding security releases are available for the [4.0.x branch](https://github.com/mastodon/mastodon/releases/tag/v4.0.5) and the [3.5.x branch](https://github.com/mastodon/mastodon/releases/tag/v3.5.9).
> If you are using nightly builds, **do not use this release** but update to `nightly-2023-07-06-security` or newer instead. If you are on the `main` branch, update to the latest commit.

#### Upgrade overview

This release contains upgrade notes that deviate from the norm:

ℹ️ Requires streaming API restart
ℹ️ There are suggested reverse proxy configuration changes
:warning: The minimal supported ImageMagick version has been bumped to 6.9.7-7

For more information, scroll down to the upgrade instructions section.

#### Changelog

##### Added

-   Add fallback redirection when getting a webfinger query `LOCAL_DOMAIN@LOCAL_DOMAIN` ([ClearlyClaire](mastodon/mastodon#23600))

##### Changed

-   Change OpenGraph-based embeds to allow fullscreen ([ClearlyClaire](mastodon/mastodon#25058))
-   Change AccessTokensVacuum to also delete expired tokens ([ClearlyClaire](mastodon/mastodon#24868))
-   Change profile updates to be sent to recently-mentioned servers ([ClearlyClaire](mastodon/mastodon#24852))
-   Change automatic post deletion thresholds and load detection ([ClearlyClaire](mastodon/mastodon#24614))
-   Change `/api/v1/statuses/:id/history` to always return at least one item ([ClearlyClaire](mastodon/mastodon#25510))
-   Change auto-linking to allow carets in URL query params ([renchap](mastodon/mastodon#25216))

##### Removed

-   Remove invalid `X-Frame-Options: ALLOWALL` ([ClearlyClaire](mastodon/mastodon#25070))

##### Fixed

-   Fix wrong view being displayed when a webhook fails validation ([ClearlyClaire](mastodon/mastodon#25464))
-   Fix soft-deleted post cleanup scheduler overwhelming the streaming server ([ThisIsMissEm](mastodon/mastodon#25519))
-   Fix incorrect pagination headers in `/api/v2/admin/accounts` ([danielmbrasil](mastodon/mastodon#25477))
-   Fix multiple inefficiencies in automatic post cleanup worker ([ClearlyClaire](mastodon/mastodon#24607), [ClearlyClaire](mastodon/mastodon#24785), [ClearlyClaire](mastodon/mastodon#24840))
-   Fix performance of streaming by parsing message JSON once ([ThisIsMissEm](mastodon/mastodon#25278), [ThisIsMissEm](mastodon/mastodon#25361))
-   Fix CSP headers when `S3_ALIAS_HOST` includes a path component ([ClearlyClaire](mastodon/mastodon#25273))
-   Fix `tootctl accounts approve --number N` not aproving N earliest registrations ([danielmbrasil](mastodon/mastodon#24605))
-   Fix reports not being closed when performing batch suspensions ([ClearlyClaire](mastodon/mastodon#24988))
-   Fix being able to vote on your own polls ([ClearlyClaire](mastodon/mastodon#25015))
-   Fix race condition when reblogging a status ([ClearlyClaire](mastodon/mastodon#25016))
-   Fix “Authorized applications” inefficiently and incorrectly getting last use date ([ClearlyClaire](mastodon/mastodon#25060))
-   Fix “Authorized applications” crashing when listing apps with certain admin API scopes ([ClearlyClaire](mastodon/mastodon#25713))
-   Fix multiple N+1s in ConversationsController ([ClearlyClaire](mastodon/mastodon#25134), [ClearlyClaire](mastodon/mastodon#25399), [ClearlyClaire](mastodon/mastodon#25499))
-   Fix user archive takeouts when using OpenStack Swift ([ClearlyClaire](mastodon/mastodon#24431))
-   Fix searching for remote content by URL not working under certain conditions ([ClearlyClaire](mastodon/mastodon#25637))
-   Fix inefficiencies in indexing content for search ([VyrCossont](mastodon/mastodon#24285), [VyrCossont](mastodon/mastodon#24342))

##### Security

-   Add finer permission requirements for managing webhooks ([ClearlyClaire](mastodon/mastodon#25463))
-   Update dependencies
-   Add hardening headers for user-uploaded files ([ClearlyClaire](mastodon/mastodon#25756))
-   Fix verified links possibly hiding important parts of the URL (CVE-2023-36462)
-   Fix timeout handling of outbound HTTP requests (CVE-2023-36461)
-   Fix arbitrary file creation through media processing (CVE-2023-36460)
-   Fix possible XSS in preview cards (CVE-2023-36459)

#### Upgrade notes

To get the code for v4.1.3, use `git fetch && git checkout v4.1.3`.

> As always, **make sure you have backups of the database before performing any upgrades**. If you are using docker-compose, this is how a backup command might look: docker exec mastodon_db\_1 pg_dump -Fc -U postgres postgres > name_of_the_backup.dump

##### Dependencies

Apart from ImageMagick, external dependencies have not changed compared to v4.1.2, the compatible Ruby, PostgreSQL, Node, Elasticsearch and Redis versions are the same, that is:

-   Ruby: 2.7 to 3.0
-   PostgreSQL: 9.5 or newer
-   Elasticsearch (optional, for full-text search): 7.x
-   Redis: 4 or newer
-   Node: >= 14, < 18
-   ImageMagick: 6.9.7-7 or newer

If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more informations and ways to fix it [on this page](mastodon/mastodon#25776).

##### Update steps

The following instructions are for updating from 4.1.2.

If you are upgrading directly from an earlier release, please carefully read the upgrade notes for the skipped releases as well, as they often require extra steps such as database migrations.

**Non-Docker only:**

1.  Install dependencies: `bundle install` and `yarn install`

**Both Docker and non-Docker:**

ℹ️ The recommended configuration for reverse proxies has been updated. Unlike updating Mastodon itself, this is not urgent, but hardening. The change is about setting `Content-Security-Policy: default-src 'none'; form-action 'none'` and `X-Content-Type-Options: nosniff` on assets. Check `dist/nginx.conf` for more information, and [the documentation](https://docs.joinmastodon.org/admin/optional/object-storage-proxy/) if you are proxying external object storage.

1.  Restart all Mastodon processes

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4yMy4yIiwidXBkYXRlZEluVmVyIjoiMzYuMjMuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Reviewed-on: https://git.home/nrdufour/home-ops/pulls/17
Co-authored-by: Renovate <renovate@ptinem.io>
Co-committed-by: Renovate <renovate@ptinem.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants