mgr/cephadm: set HEALTH warnings during apply phase in serve

[Daniel-Pivonka]

rebase and minor changes of #42565

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Daniel Pivonka dpivonka@redhat.com

[adk3798]

http://pulpito.front.sepia.ceph.com/adking-2021-10-10_15:19:57-rados:cephadm-wip-adk-testing-2021-10-09-1501-distro-basic-smithi/

Failures are known issues with cgroups and health warnings not being cleared quickly enough with agent.

[neha-ojha]

@Daniel-Pivonka are failures like https://pulpito.ceph.com/nojha-2021-10-28_22:29:54-upgrade:octopus-x-master-distro-basic-smithi/6465645/ expected in octopus to master upgrade tests at the moment? Essentially the failure is because of

2021-10-28T23:36:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_APPLY_SPEC_FAIL":{"severity":"HEALTH_WARN","summary":{"message":"Failed to apply 4 service(s): alertmanager,grafana,node-exporter,prometheus","count":4},"muted":false}},"mutes":[]}
...
021-10-28T23:36:03.552 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_neha-ojha_ceph_0ec838099064db9e5bbe31f9573445689339c4aa/qa/tasks/ceph.py", line 1469, in healthy
    manager.wait_until_healthy(timeout=300)
  File "/home/teuthworker/src/github.com_neha-ojha_ceph_0ec838099064db9e5bbe31f9573445689339c4aa/qa/tasks/ceph_manager.py", line 3141, in wait_until_healthy
    'timeout expired in wait_until_healthy'
AssertionError: timeout expired in wait_until_healthy
2021-10-28T23:36:03.644 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1751b45979ba479a81bcd55c27d3f600
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_neha-ojha_ceph_0ec838099064db9e5bbe31f9573445689339c4aa/qa/tasks/ceph.py", line 1469, in healthy
    manager.wait_until_healthy(timeout=300)
  File "/home/teuthworker/src/github.com_neha-ojha_ceph_0ec838099064db9e5bbe31f9573445689339c4aa/qa/tasks/ceph_manager.py", line 3141, in wait_until_healthy
    'timeout expired in wait_until_healthy'
AssertionError: timeout expired in wait_until_healthy

[sebastian-philipp]

Given we never upgraded the alertmanager in this run, I think the upgrade test was broken for a while now 😢

[Daniel-Pivonka]

definitely a problem im also seeing this when manually testing an upgrade from octopus to master :( looking into a solution

[neha-ojha]

definitely a problem im also seeing this when manually testing an upgrade from octopus to master :( looking into a solution

thanks @Daniel-Pivonka @sebastian-philipp
@yuriw FYI, regarding https://tracker.ceph.com/issues/53097

[sebastian-philipp]

I dropped a86d0db from the Pacific backport #43728 , as it breaks the upgrade from Octopus.