No support for redis sentinel caches.

[kashalls]

It seems like the redis package does not have support for redis sentinel databases. Redis sentinel allows for better high availability support than running a single redis instance.

Launching Shynet queue worker...
 
 -------------- celery@shynet-celeryworker-84bc85849d-xqb7c v5.2.7 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-6.1.0-9-amd64-x86_64-with 2023-09-19 09:36:44
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         shynet:0x7f9ba37facb0
- ** ---------- .> transport:   redis://redis.default.svc.cluster.local:6379/6
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ---- .> task events: ON
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . analytics.tasks.ingress_request
  . dashboard.tasks.send_email

[2023-09-19 09:36:44,579: INFO/MainProcess] Connected to redis://redis.default.svc.cluster.local:6379/6
[2023-09-19 09:36:44,584: INFO/MainProcess] mingle: searching for neighbors
[2023-09-19 09:36:45,602: INFO/MainProcess] mingle: all alone
[2023-09-19 09:36:45,624: CRITICAL/MainProcess] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/gossip.py", line 107, in start
    super().start(c)
  File "/usr/local/lib/python3.10/site-packages/celery/bootsteps.py", line 397, in start
    self.consumers = self.get_consumers(channel)
  File "/usr/local/lib/python3.10/site-packages/celery/worker/consumer/gossip.py", line 175, in get_consumers
    return [Consumer(
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 387, in __init__
    self.revive(self.channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 409, in revive
    self.declare()
  File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 422, in declare
    queue.declare()
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
    self._create_queue(nowait=nowait, channel=channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 617, in _create_queue
    self.queue_bind(nowait=nowait, channel=channel)
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 660, in queue_bind
    return self.bind_to(self.exchange, self.routing_key,
  File "/usr/local/lib/python3.10/site-packages/kombu/entity.py", line 669, in bind_to
    return (channel or self.channel).queue_bind(
  File "/usr/local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 562, in queue_bind
    self._queue_bind(exchange, *meta)
  File "/usr/local/lib/python3.10/site-packages/kombu/transport/redis.py", line 1033, in _queue_bind
    client.sadd(self.keyprefix_queue % (exchange,),
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 2243, in sadd
    return self.execute_command('SADD', name, *values)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.`

[milesmcc]

@kashalls thanks for the report. Are we good to close this issue now that #287 is merged in?

Btw, if you'd like to submit a PR documenting Sentinel cache support (following the change in #287) that would be quite helpful (as right now Shynet users may not know that support exists without finding this issue/reading the code).

[kashalls]

Btw, if you'd like to submit a PR documenting Sentinel cache support (following the change in #287) that would be quite helpful (as right now Shynet users may not know that support exists without finding this issue/reading the code).

I'll work on it this weekend, thanks for reminding me.

[kashalls]

Okay so it looks like the issue is a lot bigger than I initially thought it would be.

I didn't notice until today that this happened on the frontend side:

Performing startup checks...
Database is ready to go.
Startup checks complete!
Launching Shynet web server...
[2023-09-24 22:11:16 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-09-24 22:11:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2023-09-24 22:11:16 +0000] [1] [INFO] Using worker: sync
[2023-09-24 22:11:16 +0000] [9] [INFO] Booting worker with pid: 9
ERROR Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 92, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 146, in dispatch
    return super(LoginView, self).dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 74, in dispatch
    response = super(RedirectAuthenticatedUserMixin, self).dispatch(
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 101, in post
    if form.is_valid():
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 438, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 459, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.10/site-packages/allauth/account/forms.py", line 177, in clean
    user = get_adapter(self.request).authenticate(self.request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 511, in authenticate
    self._delete_login_attempts_cached_email(request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 483, in _delete_login_attempts_cached_email
    cache.delete(cache_key)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 30, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 287, in delete
    return client.delete(key)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 1567, in delete
    return self.execute_command('DEL', *names)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.
Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 92, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 146, in dispatch
    return super(LoginView, self).dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 74, in dispatch
    response = super(RedirectAuthenticatedUserMixin, self).dispatch(
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/views.py", line 101, in post
    if form.is_valid():
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 438, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 459, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.10/site-packages/allauth/account/forms.py", line 177, in clean
    user = get_adapter(self.request).authenticate(self.request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 511, in authenticate
    self._delete_login_attempts_cached_email(request, **credentials)
  File "/usr/local/lib/python3.10/site-packages/allauth/account/adapter.py", line 483, in _delete_login_attempts_cached_email
    cache.delete(cache_key)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 30, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/redis_cache/backends/base.py", line 287, in delete
    return client.delete(key)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 1567, in delete
    return self.execute_command('DEL', *names)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.10/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

So it seems DJANGO needs to be fixed to allow support for it.

The pr #287 did not fix it at all because even though json.loads can import the json from the environment variable, I think it converts it into a dict class and even though you can have nested dict's it still complains about ValueError: dictionary update sequence element #0 has length 1; 2 is required in celery when combining the default options.

Basically the options in settings.py needs to be set like this:
CELERY_BROKER_TRANSPORT_OPTIONS = { 'master_name': 'redis-master' }
and not:
CELERY_BROKER_TRANSPORT_OPTIONS: "{'master_name': 'redis-master'}"

[kashalls]

This is starting to become outside of my knowledge, and will likely require outside help to patch this.