With gravity version 1.2.0 and above, graceful restart throws an exception and doesn’t restart the handlers

[cat-bro]

Galaxy release 26.0 requires gravity 1.2.0, however I am having trouble with graceful restarts. They restart things gracefully, then throw an error when they get to handlers.

cat@qaai:~$ sudo galaxyctl graceful
Performing rolling restart on service: gunicorn
Gunicorn on 0.0.0.0:8888 running, version: 26.0.1.dev1 (disk version: 26.0.1.dev1)
Restarting gunicorn instance 0
Restarted gunicorn instance 0, waiting for readiness check...
Gunicorn on 0.0.0.0:8888 running, version: 26.0.1.dev1 (disk version: 26.0.1.dev1)
Gunicorn on 0.0.0.0:8889 running, version: 26.0.1.dev1 (disk version: 26.0.1.dev1)
Restarting gunicorn instance 1
Restarted gunicorn instance 1, waiting for readiness check...
Gunicorn on 0.0.0.0:8889 running, version: 26.0.1.dev1 (disk version: 26.0.1.dev1)
Restarted: galaxy-celery.service
Restarted: galaxy-celery-beat.service
Performing rolling restart on service: handler
Traceback (most recent call last):
  File "/mnt/galaxy/venv/bin/galaxyctl", line 6, in <module>
    sys.exit(galaxyctl())
             ^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/commands/cmd_graceful.py", line 19, in cli
    pm.graceful(instance_names=instances_services)
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/process_manager/__init__.py", line 75, in decorator
    routed_func(*args, **kwargs)
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/process_manager/systemd.py", line 382, in graceful
    self.__graceful_service(config, service, service_names)
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/process_manager/systemd.py", line 370, in __graceful_service
    service.rolling_restart(restart_callbacks)
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/state.py", line 291, in rolling_restart
    if not service_instance.is_ready(quiet=False):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/galaxy/venv/lib/python3.11/site-packages/gravity/state.py", line 263, in is_ready
    raise NotImplementedError("is_ready not implemented for base Service class")
NotImplementedError: is_ready not implemented for base Service class

I’ll paste the gravity configuration that goes with the above error.

  process_manager: systemd
  galaxy_root: "{{ galaxy_server_dir }}"
  galaxy_user: "{{ galaxy_user.name }}"
  app_server: gunicorn
  virtualenv: "{{ galaxy_venv_dir }}"
  gunicorn:
    - bind: "0.0.0.0:8888"
      # performance options
      workers: 1
      # Other options that will be passed to gunicorn
      extra_args: '--forwarded-allow-ips="*"'
      preload: true
      environment: "{{ galaxy_process_env }}"
    - bind: "0.0.0.0:8889"
      # performance options
      workers: 1
      # Other options that will be passed to gunicorn
      extra_args: '--forwarded-allow-ips="*"'
      preload: true
      environment: "{{ galaxy_process_env }}"
  tusd:
    enable: true
    tusd_path: /usr/local/sbin/tusd
    upload_dir: "{{ galaxy_tus_upload_store }}"
  celery:
    concurrency: 2
    loglevel: DEBUG
  handlers:
    handler:
      processes: 2
      environment: "{{ galaxy_process_env }}"
      pools:
        - job-handlers
        - workflow-schedulers

[pauldg]

Also experiencing this issue. I believe it's due to the fact that gravity is trying to do a rolling restart on the handlers, which it shouldn't since since handlers have no readiness check implemented and should default to a regular restart.

I believe that just checking whether hasattr(self.services[0], "is_ready") in this line is not sufficient:

gravity/gravity/state.py

Line 260 in d26a732

if self.count > 1 and hasattr(self.services[0], "is_ready"):

[nsoranzo]

Ah, sorry, it must be an unintended consequence of 5cc398b