Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix client reader sharding tests #7853

Merged
merged 5 commits into from
Jul 15, 2020

Conversation

erikjohnston
Copy link
Member

This fixes a bug in the client reader sharding tests where the register requests were always going to master instead of the workers (due to a typo in using self.resource rather than resource). Fixing that leads to timeouts as the /register paths try and send a HTTP replication request to master, which we didn't handle.

Currently, the tests are set up so that you need to explicitly call self.handle_http_replication_attempt() after a connection attempt has been made. This is impossible to do if sending a request, as render blocks until the request is completed. We could solve this by trying and making it possible to render call self.handle_http_replication_attempt at the same time, but it feels nicer to instead have the test base class automatically handle the http replication call.

All in all, this PR:

  1. Lifts the duplicated code for the federation sender and client reader sharding tests to a new base class. (tests/replication/)
  2. Add the ability to our test reactor to get notified on connection attempts (tests/server.py)
  3. Have the new base class automatically handle connection attempts to the replication HTTP listener on master HS (tests/replication/_base.py)
  4. Fix http client to use the right reactor for its body producer. (synapse/http/client.py)

changelog.d/7853.misc Outdated Show resolved Hide resolved
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
@erikjohnston erikjohnston requested a review from clokep July 15, 2020 11:16
Copy link
Contributor

@clokep clokep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall! Thanks for fixing all this up, I think it'll greatly improve being able to make other tests with multiple homeservers.


return resource

def make_worker_hs(self, worker_app: str, extra_config={}) -> HomeServer:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth mentioning that this uses a mock for the HTTP client that can be used to assert calls back to the primary process?

While we're here can we add a type hint for extra_config?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've docced and typed this properly now. I actually decided that it was cleaner to have a **kwargs that we can pass through to setup_test_homeserver so that different tests can easily mock out different things (e.g. the pusher sharding test I'm writing needs to mock out a different http client)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense. Could also be better to sub-class this in a test-case if you want them all to have the same config.

tests/replication/_base.py Show resolved Hide resolved
tests/replication/_base.py Show resolved Hide resolved
synapse/http/client.py Show resolved Hide resolved
@erikjohnston erikjohnston merged commit f13061d into develop Jul 15, 2020
@erikjohnston erikjohnston deleted the erikj/fix_client_reader_shard_tests branch July 15, 2020 14:27
babolivier pushed a commit that referenced this pull request Sep 1, 2021
* commit 'a973bcb8a':
  Add some tiny type annotations (#7870)
  Remove obsolete comment.
  Ensure that calls to `json.dumps` are compatible with the standard library json. (#7836)
  Avoid brand new rooms in `delete_old_current_state_events` (#7854)
  Allow accounts to be re-activated from the admin APIs. (#7847)
  Fix tests
  Fix typo
  Newsfile
  Use get_users_in_room rather than state handler in typing for speed
  Fix client reader sharding tests (#7853)
  Convert E2E key and room key handlers to async/await. (#7851)
  Return the proper 403 Forbidden error during errors with JWT logins. (#7844)
  remove `retry_on_integrity_error` wrapper for persist_events (#7848)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants