Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Length validation error when searching for certain accounts #15025

Closed
umonaca opened this issue Oct 23, 2020 · 1 comment · Fixed by #15027
Closed

Length validation error when searching for certain accounts #15025

umonaca opened this issue Oct 23, 2020 · 1 comment · Fixed by #15027

Comments

@umonaca
Copy link
Contributor

umonaca commented Oct 23, 2020

Expected behaviour

The search should return a normal result showing the target account.

Actual behaviour

When searching for board@gup.pe, it shows 422 Mastodon::LengthValidationError.

Steps to reproduce the problem

  1. On mastodon.social open the search bar on web page.
  2. Type in board@gup.pe and enter.
  3. Wait a few seconds. Then it will show this error.

Specifications

Tested on mastodon.social and other instances running latest version of Mastodon on master branch.
@ClearlyClaire
Copy link
Contributor

This happens when fetching the account's outbox (to know the number of posted messages), which is nearly 3MB, while Mastodon refuses anything more than 1MB.

Instead of erroring out, Mastodon should probably consider it cannot know the number of posts.

(And Guppe should probably implement some pagination instead of returning all the items at once)

ClearlyClaire added a commit to ClearlyClaire/mastodon that referenced this issue Oct 23, 2020
@ClearlyClaire
Fix account processing failing because of large collections
955e3cf 
Fixes mastodon#15025
@ClearlyClaire ClearlyClaire mentioned this issue Oct 23, 2020
Merged
Gargron pushed a commit that referenced this issue Oct 23, 2020
@ClearlyClaire
Fix account processing failing because of large collections (#15027)
fb58658 
Fixes #15025
mashirozx added a commit to mashirozx/mastodon that referenced this issue Oct 26, 2020
@mashirozx @joshleeb
@abcang @ClearlyClaire @dunn @dependabot
merge upstream (bump version to 3.2.1) (#242)
70fdfc0 
* Add support for Gemini urls (mastodon#15013)

This PR updates the `valid_url` regex and sanitizer allowlist to provide
support for Gemini urls.

Closes mastodon#14991

* Removed disabling comments for Style/MethodMissingSuper (mastodon#15014)

* Removed disabling comments for Style/MethodMissingSuper

* Update rubocop for codeclimate

* Add follower synchronization mechanism (mastodon#14510)

* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each

* helm: bump version to 3.2.1 (mastodon#15019)

* Bump @testing-library/react from 11.0.4 to 11.1.0 (mastodon#14992)

Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 11.0.4 to 11.1.0.
- [Release notes](https://github.com/testing-library/react-testing-library/releases)
- [Changelog](https://github.com/testing-library/react-testing-library/blob/master/CHANGELOG.md)
- [Commits](testing-library/react-testing-library@v11.0.4...v11.1.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/core from 7.11.6 to 7.12.3 (mastodon#14993)

Bumps [@babel/core](https://github.com/babel/babel/tree/HEAD/packages/babel-core) from 7.11.6 to 7.12.3.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.3/packages/babel-core)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/runtime from 7.11.2 to 7.12.1 (mastodon#14994)

Bumps [@babel/runtime](https://github.com/babel/babel/tree/HEAD/packages/babel-runtime) from 7.11.2 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-runtime)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump webmock from 3.9.1 to 3.9.3 (mastodon#14996)

Bumps [webmock](https://github.com/bblimke/webmock) from 3.9.1 to 3.9.3.
- [Release notes](https://github.com/bblimke/webmock/releases)
- [Changelog](https://github.com/bblimke/webmock/blob/master/CHANGELOG.md)
- [Commits](bblimke/webmock@v3.9.1...v3.9.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/plugin-transform-react-inline-elements from 7.10.4 to 7.12.1 (mastodon#14998)

Bumps [@babel/plugin-transform-react-inline-elements](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-transform-react-inline-elements) from 7.10.4 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-plugin-transform-react-inline-elements)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump active_record_query_trace from 1.7 to 1.8 (mastodon#14999)

Bumps [active_record_query_trace](https://github.com/brunofacca/active-record-query-trace) from 1.7 to 1.8.
- [Release notes](https://github.com/brunofacca/active-record-query-trace/releases)
- [Changelog](https://github.com/brunofacca/active-record-query-trace/blob/master/HISTORY.md)
- [Commits](brunofacca/active-record-query-trace@v1.7...v1.8)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump yargs from 16.0.3 to 16.1.0 (mastodon#15010)

Bumps [yargs](https://github.com/yargs/yargs) from 16.0.3 to 16.1.0.
- [Release notes](https://github.com/yargs/yargs/releases)
- [Changelog](https://github.com/yargs/yargs/blob/master/CHANGELOG.md)
- [Commits](yargs/yargs@v16.0.3...v16.1.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump oj from 3.10.14 to 3.10.15 (mastodon#15009)

Bumps [oj](https://github.com/ohler55/oj) from 3.10.14 to 3.10.15.
- [Release notes](https://github.com/ohler55/oj/releases)
- [Changelog](https://github.com/ohler55/oj/blob/develop/CHANGELOG.md)
- [Commits](ohler55/oj@v3.10.14...v3.10.15)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tzinfo-data from 1.2020.2 to 1.2020.3 (mastodon#15002)

Bumps [tzinfo-data](https://github.com/tzinfo/tzinfo-data) from 1.2020.2 to 1.2020.3.
- [Release notes](https://github.com/tzinfo/tzinfo-data/releases)
- [Commits](tzinfo/tzinfo-data@v1.2020.2...v1.2020.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump rubocop from 0.93.0 to 0.93.1 (mastodon#15004)

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.93.0 to 0.93.1.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](rubocop/rubocop@v0.93.0...v0.93.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump omniauth-saml from 1.10.2 to 1.10.3 (mastodon#15007)

Bumps [omniauth-saml](https://github.com/omniauth/omniauth-saml) from 1.10.2 to 1.10.3.
- [Release notes](https://github.com/omniauth/omniauth-saml/releases)
- [Changelog](https://github.com/omniauth/omniauth-saml/blob/master/CHANGELOG.md)
- [Commits](omniauth/omniauth-saml@v1.10.2...v1.10.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @github/webauthn-json from 0.5.6 to 0.5.7 (mastodon#14997)

Bumps [@github/webauthn-json](https://github.com/github/webauthn-json) from 0.5.6 to 0.5.7.
- [Release notes](https://github.com/github/webauthn-json/releases)
- [Commits](github/webauthn-json@v0.5.6...v0.5.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/plugin-proposal-decorators from 7.10.5 to 7.12.1 (mastodon#15008)

Bumps [@babel/plugin-proposal-decorators](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-proposal-decorators) from 7.10.5 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-plugin-proposal-decorators)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/preset-react from 7.10.4 to 7.12.1 (mastodon#15006)

Bumps [@babel/preset-react](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-react) from 7.10.4 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-preset-react)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump react from 16.13.1 to 16.14.0 (mastodon#15005)

Bumps [react](https://github.com/facebook/react/tree/HEAD/packages/react) from 16.13.1 to 16.14.0.
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/master/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v16.14.0/packages/react)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/preset-env from 7.11.5 to 7.12.1 (mastodon#15001)

Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env) from 7.11.5 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-preset-env)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @babel/plugin-transform-runtime from 7.11.5 to 7.12.1 (mastodon#14995)

Bumps [@babel/plugin-transform-runtime](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-transform-runtime) from 7.11.5 to 7.12.1.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.12.1/packages/babel-plugin-transform-runtime)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix account processing failing because of large collections (mastodon#15027)

Fixes mastodon#15025

* Fix followers synchronization mechanism not being triggered on mentions (mastodon#15026)

e.g. if someone on an instance that previously had followers gets mentioned
in a private toot, before this PR, they would not receive a
Collection-Synchronization header and may show the toot to the former followers
in addition to the mentioned person.

Co-authored-by: Josh Leeb-du Toit <mail@joshleeb.com>
Co-authored-by: abcang <abcang1015@gmail.com>
Co-authored-by: ThibG <thib@sitedethib.com>
Co-authored-by: Alex Dunn <dunn.alex@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@umonaca umonaca mentioned this issue Oct 28, 2020
Closed
ClearlyClaire added a commit to ClearlyClaire/mastodon that referenced this issue Dec 18, 2020
@ClearlyClaire
Fix account processing failing because of large collections (mastodon…
c5bb995 
…#15027)

Fixes mastodon#15025
Gargron added a commit that referenced this issue Dec 18, 2020
@ClearlyClaire @Gargron
Backport fixes to 3.2 (#15360)
406adfc 
* Fix 2FA/sign-in token sessions being valid after password change (#14802)

If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token

* Fix Move handler not being triggered when failing to fetch target (#15107)

When failing to fetch the target account, the ProcessingWorker fails
as expected, but since it hasn't cleared the `move_in_progress` flag,
the next attempt at processing skips the `Move` activity altogether.

This commit changes it to clear the flag when encountering any
unexpected error on fetching the target account. This is likely to
occur because, of, e.g., a timeout, when many instances query the
same actor at the same time.

* Fix slow distinct queries where grouped queries are faster (#15287)

About 2x speed-up on inboxes query

* Fix possible inconsistencies in tag search (#14906)

Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
  downcasing before query but not before insertion may mean that some
  tags with some casings are not searchable

* Fix updating account counters when account_stat is not yet created (#15108)

* Fix account processing failing because of large collections (#15027)

Fixes #15025

* Fix downloading remote media files when server returns empty filename (#14867)

Fixes #14817

* Fix webfinger redirect handling in ResolveAccountService (#15187)

* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests

* Remove dependency on unused and unmaintained http_parser.rb gem (#14574)

It seems that years ago, the “http” gem dependend on the “http_parser.rb” gem
(it now depends on the “http-parser” gem), and, still years ago, we pulled
it from git in order to benefit from a bugfix that wasn't released yet (#7467).

* Add tootctl maintenance fix-duplicates (#14860, #15201, #15264, #15349, #15359)

* Fix old migration script not being able to run if it fails midway (#15361)

* Fix old migration script not being able to run if it fails midway

Improve the robustness of a migration script likely to fail because of database
corruption so it can run again once database corruptions are fixed.

* Display a specific error message in case of index corruption

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
shouo1987 pushed a commit to CrossGate-Pawoo/mastodon that referenced this issue Dec 7, 2022
@ClearlyClaire @Gargron
Backport fixes to 3.2 (mastodon#15360)
9a45ab5 
* Fix 2FA/sign-in token sessions being valid after password change (mastodon#14802)

If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token

* Fix Move handler not being triggered when failing to fetch target (mastodon#15107)

When failing to fetch the target account, the ProcessingWorker fails
as expected, but since it hasn't cleared the `move_in_progress` flag,
the next attempt at processing skips the `Move` activity altogether.

This commit changes it to clear the flag when encountering any
unexpected error on fetching the target account. This is likely to
occur because, of, e.g., a timeout, when many instances query the
same actor at the same time.

* Fix slow distinct queries where grouped queries are faster (mastodon#15287)

About 2x speed-up on inboxes query

* Fix possible inconsistencies in tag search (mastodon#14906)

Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
  downcasing before query but not before insertion may mean that some
  tags with some casings are not searchable

* Fix updating account counters when account_stat is not yet created (mastodon#15108)

* Fix account processing failing because of large collections (mastodon#15027)

Fixes mastodon#15025

* Fix downloading remote media files when server returns empty filename (mastodon#14867)

Fixes mastodon#14817

* Fix webfinger redirect handling in ResolveAccountService (mastodon#15187)

* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests

* Remove dependency on unused and unmaintained http_parser.rb gem (mastodon#14574)

It seems that years ago, the “http” gem dependend on the “http_parser.rb” gem
(it now depends on the “http-parser” gem), and, still years ago, we pulled
it from git in order to benefit from a bugfix that wasn't released yet (mastodon#7467).

* Add tootctl maintenance fix-duplicates (mastodon#14860, mastodon#15201, mastodon#15264, mastodon#15349, mastodon#15359)

* Fix old migration script not being able to run if it fails midway (mastodon#15361)

* Fix old migration script not being able to run if it fails midway

Improve the robustness of a migration script likely to fail because of database
corruption so it can run again once database corruptions are fixed.

* Display a specific error message in case of index corruption

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix account processing failing because of large collections ClearlyClaire/mastodon
2 participants
@ClearlyClaire @umonaca