failing requests from app due to Redis memory issue

[gassc]

Need to investigate and out of memory issue with Redis that is causing rainfall retrieval requests to fail. This only affects the aync ReST API endpoint used by the Rainfall web app.

Sample logs:

2023-05-19T12:48:45.122985+00:00 app[worker.1]: Traceback (most recent call last):
2023-05-19T12:48:45.122999+00:00 app[worker.1]: File "/app/manage.py", line 21, in <module>
2023-05-19T12:48:45.123083+00:00 app[worker.1]: main()
2023-05-19T12:48:45.123085+00:00 app[worker.1]: File "/app/manage.py", line 17, in main
2023-05-19T12:48:45.123128+00:00 app[worker.1]: execute_from_command_line(sys.argv)
2023-05-19T12:48:45.123137+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
2023-05-19T12:48:45.123217+00:00 app[worker.1]: utility.execute()
2023-05-19T12:48:45.123218+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
2023-05-19T12:48:45.123285+00:00 app[worker.1]: self.fetch_command(subcommand).run_from_argv(self.argv)
2023-05-19T12:48:45.123286+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
2023-05-19T12:48:45.123353+00:00 app[worker.1]: self.execute(*args, **cmd_options)
2023-05-19T12:48:45.123354+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
2023-05-19T12:48:45.123417+00:00 app[worker.1]: output = self.handle(*args, **options)
2023-05-19T12:48:45.123418+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django_rq/management/commands/rqworker.py", line 128, in handle
2023-05-19T12:48:45.123457+00:00 app[worker.1]: w.work(
2023-05-19T12:48:45.123458+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/rq/worker.py", line 571, in work
2023-05-19T12:48:45.123541+00:00 app[worker.1]: self.set_state(WorkerStatus.STARTED)
2023-05-19T12:48:45.123542+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/rq/worker.py", line 374, in set_state
2023-05-19T12:48:45.123618+00:00 app[worker.1]: connection.hset(self.key, 'state', state)
2023-05-19T12:48:45.123627+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/commands/core.py", line 4859, in hset
2023-05-19T12:48:45.124379+00:00 app[worker.1]: return self.execute_command("HSET", name, *items)
2023-05-19T12:48:45.124379+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1258, in execute_command
2023-05-19T12:48:45.124653+00:00 app[worker.1]: return conn.retry.call_with_retry(
2023-05-19T12:48:45.124661+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2023-05-19T12:48:45.124727+00:00 app[worker.1]: return do()
2023-05-19T12:48:45.124728+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1259, in <lambda>
2023-05-19T12:48:45.124935+00:00 app[worker.1]: lambda: self._send_command_parse_response(
2023-05-19T12:48:45.124945+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1235, in _send_command_parse_response
2023-05-19T12:48:45.125157+00:00 app[worker.1]: return self.parse_response(conn, command_name, **options)
2023-05-19T12:48:45.125157+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1275, in parse_response
2023-05-19T12:48:45.125363+00:00 app[worker.1]: response = connection.read_response()
2023-05-19T12:48:45.125372+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/connection.py", line 827, in read_response
2023-05-19T12:48:45.125499+00:00 app[worker.1]: raise response
2023-05-19T12:48:45.125523+00:00 app[worker.1]: redis.exceptions.ResponseError: OOM command not allowed when used memory > 'maxmemory'.
2023-05-19T12:48:45.320864+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.175537191.280be8e0-5e3b-4772-967e-eb52de41404b sample#memory_total=101.65MB sample#memory_rss=97.64MB sample#memory_cache=4.01MB sample#memory_swap=0.00MB sample#memory_pgpgin=27207pages sample#memory_pgpgout=1696pages sample#memory_quota=1024.00MB
2023-05-19T12:48:45.457728+00:00 heroku[worker.1]: Process exited with status 1
2023-05-19T12:48:45.486736+00:00 heroku[worker.1]: State changed from up to crashed

[gassc]

Upgraded the instance to double the RAM and that seems to have quieted things down. Will continue to monitor.