Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add service discovery and connection caching to Sentinel, and make send/receive more deterministic #43

Merged
merged 38 commits into from
May 18, 2017

Conversation

pseudoben
Copy link

There are several large changes here. The primary purpose is to add some key capabilities to the Sentinel channel layer. In the process, I've made some improvements to the core Redis channel layer and added a Docker-based testing environment.

  • Added a Docker-based testing environment using docker-compose. Provides two Redis masters, two slaves, and one Sentinel server. The Redis servers are configured to not persist data to disk.
  • Made send/receive logic more deterministic for normal (non-single-reader) channels. When sending to or receiving from a normal channel, the channel layer now cycles through all connections in order, rather than choosing at random. This guarantees that a single worker backed by multiple connections will pick up a message after calling receive() no more than once for each connection. This makes the channel layer behavior more predictable without compromising performance, and makes it possible to test channel layers with multiple connections.
  • Added two new capabilities to the Sentinel channel layer: connection caching and service discovery.
    • Connection caching is enabled by passing sentinel_refresh_interval. This will cause the channel layer to cache its master connections and only refresh them when a connection is requested and the refresh interval has passed since the last refresh.
    • Service discovery happens automatically if services is empty. The channel layer calls out to its Sentinel hosts to get all service names tracked by Sentinel, and uses that information to build its service list.

Ben Greenberg added 30 commits May 10, 2017 14:54
…vided, and provide an option to refresh masters on a cooldown rather than on every request
…l available masters when trying to get a response. This is contingent on asgiref changes
@pseudoben
Copy link
Author

Looks like I need to make some changes to the tests that they work both in Travis and in Docker.

@andrewgodwin andrewgodwin mentioned this pull request May 18, 2017
@andrewgodwin andrewgodwin merged commit 1325286 into django:master May 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants