Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: connect to redis sentinel for redis queue (backport #25506) #25556

Merged
merged 2 commits into from Mar 20, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Mar 20, 2024

Please provide enough information so that others can review your pull request:

Use redis sentinel for redis queue connection.
It is setup for production only.
Uses extra keys from common_site_config.json

Explain the details for making this change. What existing problem does the pull request solve?

At the moment we can connect to single redis master for queue. With this we can connect to sentinel cluster.

https://redis.io/docs/management/sentinel

To try it with frappe_docker/devcontainer make following changes to .devcontainer/docker-compose.yml

version: "3.7"
services:
  mariadb:
    image: docker.io/mariadb:10.6
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
      - --skip-character-set-client-handshake
      - --skip-innodb-read-only-compressed # Temporary fix for MariaDB 10.6
    environment:
      - MYSQL_ROOT_PASSWORD=123
    volumes:
      - mariadb-data:/var/lib/mysql

  redis-cache:
    image: redis:alpine

  redis-queue:
    image: redis:alpine


## Change begins here ##
  redis-queue-sentinel:
    image: docker.io/bitnami/redis-sentinel:7.2
    environment:
      - REDIS_SENTINEL_PASSWORD=sentinelpassword
      - REDIS_MASTER_HOST=redis-queue-master
      - REDIS_MASTER_PASSWORD=masterpassword

  redis-queue-master:
    image: docker.io/bitnami/redis:7.2
    environment:
      - REDIS_PASSWORD=masterpassword
## Change ends here ##

  frappe:
    image: docker.io/frappe/bench:latest
    command: sleep infinity
    environment:
      - SHELL=/bin/bash
    volumes:
      - ..:/workspace:cached
      - ${HOME}/.ssh:/home/frappe/.ssh
    working_dir: /workspace/development
    ports:
      - 8000-8005:8000-8005
      - 9000-9005:9000-9005

volumes:
  mariadb-data:

add redis_cache_* keys to common_site_config.json

{
 "background_workers": 1,
 "db_host": "mariadb",
 "db_type": "mariadb",
 "developer_mode": 1,
 "file_watcher_port": 6788,
 "frappe_user": "frappe",
 "gunicorn_workers": 25,
 "live_reload": true,
 "rebase_on_pull": false,
 "redis_socketio": "redis://redis-queue:6379",
 "redis_cache": "redis://redis-cache:6379",
 "redis_queue": "redis://redis-queue:6379",
 "redis_queue_sentinel_enabled": 1,
 "redis_queue_sentinels": ["redis-queue-sentinel:26379"],
 "redis_queue_sentinel_password": "sentinelpassword",
 "redis_queue_master_service": "mymaster",
 "redis_queue_master_password": "masterpassword",
 "restart_supervisor_on_update": false,
 "restart_systemd_on_update": false,
 "serve_default_site": true,
 "shallow_clone": true,
 "socketio_port": 9000,
 "use_redis_auth": false,
 "webserver_port": 8000
}

Note: Nothing has changed to the redis_queue key. It will be used by socketio.

Screenshots/GIFs

Docs: https://frappeframework.com/app/wiki-page-patch/372939da4e


This is an automatic backport of pull request #25506 done by [Mergify](https://mergify.com).

(cherry picked from commit e9ece3b)

# Conflicts:
#	frappe/utils/redis_queue.py
@mergify mergify bot added the conflicts label Mar 20, 2024
@mergify mergify bot requested a review from a team as a code owner March 20, 2024 01:36
@mergify mergify bot requested review from akhilnarang and removed request for a team March 20, 2024 01:36
@mergify mergify bot assigned revant Mar 20, 2024
Copy link
Contributor Author

mergify bot commented Mar 20, 2024

Cherry-pick of e9ece3b has failed:

On branch mergify/bp/version-14-hotfix/pr-25506
Your branch is up to date with 'origin/version-14-hotfix'.

You are currently cherry-picking commit e9ece3b283.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   frappe/utils/redis_queue.py

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@revant revant removed the conflicts label Mar 20, 2024
@akhilnarang akhilnarang merged commit a9ee773 into version-14-hotfix Mar 20, 2024
10 checks passed
@akhilnarang akhilnarang deleted the mergify/bp/version-14-hotfix/pr-25506 branch March 20, 2024 12:38
frappe-pr-bot pushed a commit that referenced this pull request Mar 27, 2024
# [14.69.0](v14.68.2...v14.69.0) (2024-03-27)

### Bug Fixes

* allow page length 2500 ([#25062](#25062)) ([d40a169](d40a169))
* **Contact form:** make email translatable ([1a794df](1a794df))
* **Contact form:** make title and options translatable ([6d731fa](6d731fa))
* **Contact form:** translate internal notification to system language ([2b2cf83](2b2cf83))
* DB Query distinct handling with full table name ([#25594](#25594)) ([#25596](#25596)) ([d201326](d201326))
* diff after converting to html to text ([#25582](#25582)) ([#25583](#25583)) ([68d6947](68d6947))
* don't parse CSV params for Excel ([87c6fc6](87c6fc6))
* duplicate translation of field labels ([471ad26](471ad26))
* fieldname referenced before assignment ([95a0db8](95a0db8))
* **lint:** v14 doesn't have typed doctype classes, so this was never needed ([#25578](#25578)) ([e343a32](e343a32))
* Mark totals row correctly for print ([#25629](#25629)) ([#25630](#25630)) ([a902f6d](a902f6d))
* only add title field in search if it exists ([#25634](#25634)) ([#25635](#25635)) ([446c4d6](446c4d6))
* pop from form_params ([377b35c](377b35c))
* ruff fixes ([8ecc7d9](8ecc7d9))
* test reportview ([bb5c3a1](bb5c3a1))

### Features

* add csv preview ([d89a126](d89a126))
* add test for exporting reportview as CSV ([4eb31e0](4eb31e0))
* add translation context ([88cfe4b](88cfe4b))
* allow setting a custom rate limit for `login via email link` feature ([b7a1da5](b7a1da5))
* connect to redis sentinel for redis queue (backport [#25506](#25506)) ([#25556](#25556)) ([a9ee773](a9ee773))
* CSV params for query report ([7493c4e](7493c4e))
* CSV params for report view ([9f14d65](9f14d65))
* german translations for export dialog ([eab535d](eab535d))
* hide csv settings in collapsible section ([efb70be](efb70be))
* preview real data ([91b20b8](91b20b8))

### Performance Improvements

* Faster "show title in link field" on list view ([#25597](#25597)) ([#25621](#25621)) ([7df36f6](7df36f6))
* remove useless sorting on docstatus ([#25571](#25571)) ([#25590](#25590)) ([5e65cc9](5e65cc9))

### Reverts

* Revert "fix: escape text types before setting disp area (#25520) (#25522)" (#25604) ([0bf0cb8](0bf0cb8)), closes [#25520](#25520) [#25522](#25522) [#25604](#25604)
@frappe-pr-bot
Copy link
Collaborator

🎉 This PR is included in version 14.69.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants