504 when opening system menu

[mitch2k]

Hi,

I have recently upgraded from Netbox 3.6.5 to 3.7.8 and then to 4.0.3. Everything works great. Except, when trying to open "system' in het menu, the browser just keeps loading, and after 2 minutes I get a 504.

journalctl -eu netbox-rq is showing:
_

May 28 11:25:42 AZURSYNDCIM01 python3[2663]: Traceback (most recent call last):
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 276, in connect
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: sock = self.retry.call_with_retry(
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: raise error
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: return do()
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 277, in
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: lambda: self._connect(), lambda error: self.disconnect(error)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 639, in _connect
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: raise err
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 627, in _connect
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: sock.connect(socket_address)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: TimeoutError: [Errno 110] Connection timed out
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: During handling of the above exception, another exception occurred:
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: Traceback (most recent call last):
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/netbox/manage.py", line 10, in
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: execute_from_command_line(sys.argv)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/init.py", line 442, in execute_from_c>
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: utility.execute()
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/init.py", line 436, in execute
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: self.fetch_command(subcommand).run_from_argv(self.argv)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: self.execute(*args, **cmd_options)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: output = self.handle(*args, **options)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/netbox/core/management/commands/rqworker.py", line 28, in handle
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: super().handle(*args, **options)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/management/commands/rqworker.py", line 90, in handle
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: w = get_worker(*args, **worker_kwargs)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/workers.py", line 50, in get_worker
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: return worker_class(
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/rq/worker.py", line 205, in init
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: connection.client_setname(self.name)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/commands/core.py", line 710, in client_setname
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: return self.execute_command("CLIENT SETNAME", name, **kwargs)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/client.py", line 542, in execute_command
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: conn = self.connection or pool.get_connection(command_name, **options)
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 1109, in get_connection
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: connection.connect()
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: File "/opt/netbox/venv/lib/python3.10/site-packages/redis/connection.py", line 280, in connect
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: raise TimeoutError("Timeout connecting to server")
May 28 11:25:42 AZURSYNDCIM01 python3[2663]: redis.exceptions.TimeoutError: Timeout connecting to server
May 28 11:25:43 AZURSYNDCIM01 systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
May 28 11:25:43 AZURSYNDCIM01 systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
May 28 11:25:43 AZURSYNDCIM01 systemd[1]: netbox-rq.service: Consumed 9.068s CPU time.

_

journalctl -eu netbox is not showing any errors. I've checked the redis and postgress log, but don't see any errors.
I'm not sure where to start troubleshooting further.

Anyone any idea?

Thanks!

[a084ed22]

You seem to have issues connecting to redis. Without additional information on your setup it is difficult to suggest debugging steps. Are the redis service and netbox on the same machine? Have you ensured that redis is running?

[mitch2k]

Hi,

redis is indeed on the same machine. It's quite default installed as in the manual. It's up and running and I don't see any issues in the Redis log:

635:M 28 May 2024 11:49:34.012 # Server initialized
635:M 28 May 2024 11:49:34.012 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
635:M 28 May 2024 11:49:34.012 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain
the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
635:M 28 May 2024 11:49:34.058 * Loading RDB produced by version 6.0.16
635:M 28 May 2024 11:49:34.058 * RDB age 17 seconds
635:M 28 May 2024 11:49:34.058 * RDB memory usage when created 1.00 Mb
635:M 28 May 2024 11:49:34.064 * DB loaded from disk: 0.051 seconds
635:M 28 May 2024 11:49:34.064 * Ready to accept connections

[a084ed22]

Can you check with something like ss -lnpt | grep 6379 whether there is a socket it is listening to? If there is, your problem is likely in netbox's configuration.py. Please check again the redis related configuration on it

[mitch2k]

Redis is listening:

LISTEN 0      511        127.0.0.1:6379      0.0.0.0:*    users:(("redis-server",pid=635,fd=6))
LISTEN 0      511            [::1]:6379         [::]:*    users:(("redis-server",pid=635,fd=7))

I have also fixed the warnings above causing them to dissapear and Redis running fine:µ

4229:C 28 May 2024 12:25:21.880 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4229:C 28 May 2024 12:25:21.880 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=4229, just started
4229:C 28 May 2024 12:25:21.880 # Configuration loaded
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 6.0.16 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 4229
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

4229:M 28 May 2024 12:25:21.882 # Server initialized
4229:M 28 May 2024 12:25:21.882 * Loading RDB produced by version 6.0.16
4229:M 28 May 2024 12:25:21.882 * RDB age 0 seconds
4229:M 28 May 2024 12:25:21.882 * RDB memory usage when created 0.90 Mb
4229:M 28 May 2024 12:25:21.883 * DB loaded from disk: 0.001 seconds
4229:M 28 May 2024 12:25:21.883 * Ready to accept connections

The redis section in the config is like this:

REDIS = {
'tasks': {
'HOST': 'fqdn',
'PORT': 6379,
'PASSWORD': '',
'DATABASE': 0,
'DEFAULT_TIMEOUT': 300,
'SSL': False,
},
'caching': {
'HOST': 'localhost',
'PORT': 6379,
'PASSWORD': '',
'DATABASE': 1,
'DEFAULT_TIMEOUT': 300,
'SSL': False,
}
}

[mitch2k]

Ah, apparantly 1 of the 2 redis entries was incorrect. Fixed it. Thanks!!