Sending HUP signal to manager causes zmq exceptions

[douglatornell]

Sending the HUP signal to the manager;
e.g. pkill --signal HUP -f nowcast.manager
causes it to apparently crash due to zmq exceptions.

Expected result is for the manager to reload the config file.

[douglatornell]

Relevant log:

2021-12-13 10:00:37,438 INFO [manager] hangup signal (SIGHUP) received; reloading configuration
2021-12-13 10:00:37,721 CRITICAL [manager] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.9/logging/config.py", line 564, in configure
    handler = self.configure_handler(handlers[name])
  File "/SalishSeaCast/nowcast-env/lib/python3.9/logging/config.py", line 745, in configure_handler
    result = factory(**kwargs)
  File "/SalishSeaCast/nowcast-env/lib/python3.9/site-packages/zmq/log/handlers.py", line 86, in __init__
    self.socket.bind(interface_or_socket)
  File "/SalishSeaCast/nowcast-env/lib/python3.9/site-packages/zmq/sugar/socket.py", line 172, in bind
    super().bind(addr)
  File "zmq/backend/cython/socket.pyx", line 540, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 267, in _process_messages
    self._try_messages()
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 286, in _try_messages
    message = self._socket.recv_string()
  File "/SalishSeaCast/nowcast-env/lib/python3.9/site-packages/zmq/sugar/socket.py", line 700, in recv_string
    msg = self.recv(flags=flags)
  File "zmq/backend/cython/socket.pyx", line 781, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 817, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 191, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 13, in zmq.backend.cython.checkrc._check_rc
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 226, in sighup_handler
    self.setup()
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 113, in setup
    msg = self._configure_logging()
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 168, in _configure_logging
    logging.config.dictConfig(logging_config)
  File "/SalishSeaCast/nowcast-env/lib/python3.9/logging/config.py", line 809, in dictConfig
    dictConfigClass(config).configure()
  File "/SalishSeaCast/nowcast-env/lib/python3.9/logging/config.py", line 571, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'zmq_pub'
2021-12-13 10:00:38,014 CRITICAL [manager] shutting down

[douglatornell]

Moved to 43ravens/NEMO_Nowcast#14