octopus: mgr: update mon metadata when monmap is updated #39219

k0ste · 2021-02-02T07:02:12Z

backport trackers:

backport of

parent trackers:

this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/master/src/script/ceph-backport.sh

src/mgr/Mgr.cc

ideepika · 2021-03-04T16:13:37Z

jenkins test api

ideepika

teuthology passed: https://trello.com/c/sbaQayOI/1161-wip-yuri-testing-2021-03-01-1125-octopus

failures not related

epuertat · 2021-03-04T17:13:12Z

Failed API tests:

FAIL: test_inventory_list (tasks.mgr.dashboard.test_orchestrator.OrchestratorControllerTest)

2021-03-04 16:59:50,637.637 INFO:__main__:----------------------------------------------------------------------
2021-03-04 16:59:50,637.637 INFO:__main__:Traceback (most recent call last):
2021-03-04 16:59:50,638.638 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/test_orchestrator.py", line 120, in test_inventory_list
2021-03-04 16:59:50,638.638 INFO:__main__:    self.assertStatus(200)
2021-03-04 16:59:50,638.638 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/helper.py", line 407, in assertStatus
2021-03-04 16:59:50,638.638 INFO:__main__:    self.assertEqual(self._resp.status_code, status)
2021-03-04 16:59:50,638.638 INFO:__main__:AssertionError: 500 != 200
2021-03-04 16:59:50,638.638 INFO:__main__:
2021-03-04 16:59:50,638.638 INFO:__main__:----------------------------------------------------------------------

epuertat · 2021-03-04T17:13:20Z

jenkins test api

yuriw · 2021-03-04T20:17:37Z

@epuertat FYI

epuertat · 2021-03-04T20:32:51Z

😓 Different failure:

Collecting teuthology[test]
  Cloning https://github.com/ceph/teuthology to /tmp/pip-install-ybk409o5/teuthology_41923afde31a4a7495602519d342ad6a
  Running command git clone -q https://github.com/ceph/teuthology /tmp/pip-install-ybk409o5/teuthology_41923afde31a4a7495602519d342ad6a
ERROR: Could not find a version that satisfies the requirement apache-libcloud (from teuthology[test])
ERROR: No matching distribution found for apache-libcloud

epuertat · 2021-03-04T20:33:02Z

jenkins test api

yuriw · 2021-03-04T22:28:25Z

jenkins test api

yuriw · 2021-03-04T22:29:13Z

jenkins test api

yuriw · 2021-03-04T22:29:40Z

jenkins test this please

epuertat · 2021-03-05T13:07:40Z

Mmm, it's failing with the same error as before:

FAIL: test_inventory_list (tasks.mgr.dashboard.test_orchestrator.OrchestratorControllerTest)

2021-03-04 23:41:03,707.707 INFO:__main__:----------------------------------------------------------------------
2021-03-04 23:41:03,707.707 INFO:__main__:Traceback (most recent call last):
2021-03-04 23:41:03,707.707 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/test_orchestrator.py", line 120, in test_inventory_list
2021-03-04 23:41:03,707.707 INFO:__main__:    self.assertStatus(200)
2021-03-04 23:41:03,707.707 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/helper.py", line 407, in assertStatus
2021-03-04 23:41:03,708.708 INFO:__main__:    self.assertEqual(self._resp.status_code, status)
2021-03-04 23:41:03,708.708 INFO:__main__:AssertionError: 500 != 200
2021-03-04 23:41:03,708.708 INFO:__main__:
2021-03-04 23:41:03,708.708 INFO:__main__:----------------------------------------------------------------------

In fact it reported 3 times the same issue [1], [2], [3]. This doesn't like like a 'flapping' test but a real change (I don't yet mean bug, but at least the test is behaving in a different way between master and octopus).

Main difference regarding the test itself is that in octopus it uses the test_orchestrator, but not in master/pacific. Do you have any idea, @sebastian-philipp?

That test consumes mgr.get('osd_metadata') mgr API call and the Orchestrator client invetory list call.

Additionally I see in the Jenkins console log the following trace, but it appears in successful runs too:
Error EINVAL: DaemonDescription: __init__() got an unexpected keyword argument 'nodename'

epuertat · 2021-03-05T13:13:51Z

Just pushed a commit to increase verbosity in api tests output

epuertat · 2021-03-05T13:20:06Z

jenkins retest this please

epuertat · 2021-03-06T14:04:33Z

jenkins test api

epuertat · 2021-03-06T18:41:00Z

"Traceback (most recent call last):
  File \"/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/controllers/orchestrator.py\", line 111, in list
    device_osd_map = get_device_osd_map()
  File \"/home/jenkins-build/build/workspace/ceph-api/src/pybind/mgr/dashboard/controllers/orchestrator.py\", line 48, in get_device_osd_map
    result[hostname][device].append(int(osd_id))
ValueError: invalid literal for int() with base 10: 'a'

epuertat · 2021-03-06T19:37:02Z

The mgr.get('osd_metadata') API method, apart from the expected numeric OSD IDs (0, 1, 2, 3), it now returns 3 additional keys ('a', 'b' and 'c'), I guess corresponding to the 3 mon daemons.

{
   "0":{
      "hostname":"braggi16",
      "": "...",
      "rotational":"0"
   },
   "1":{
      "hostname":"braggi16",
      "": "...",
      "rotational":"0"
   },
   "2":{
      "hostname":"braggi16",
      "": "...",
      "rotational":"0"
   },
   "3":{
      "hostname":"braggi16",
      "": "...",
      "rotational":"0"
   },
   "a":{
      "hostname":"braggi16",
      "addrs":"[v2:172.21.2.16:40564/0,v1:172.21.2.16:40565/0]",
      "": "...",
      "os":"Linux"
   },
   "b":{
      "hostname":"braggi16",
      "addrs":"[v2:172.21.2.16:40566/0,v1:172.21.2.16:40567/0]",

      "os":"Linux"
   },
   "c":{
      "hostname":"braggi16",
      "addrs":"[v2:172.21.2.16:40568/0,v1:172.21.2.16:40569/0]",
      "": "...",
      "os":"Linux"
   }
}

tchaikov · 2021-03-09T04:57:22Z

The mgr.get('osd_metadata') API method, apart from the expected numeric OSD IDs (0, 1, 2, 3), it now returns 3 additional keys ('a', 'b' and 'c'), I guess corresponding to the 3 mon daemons.

should be fixed by #39937

k0ste · 2021-03-10T16:31:02Z

Kefu, I'll cherry-pick #39937 tomorrow

yuriw · 2021-03-10T16:35:37Z

@k0ste add needs-qa when done and I will retest

there is chance that some monitor(s) is updated / upgraded in a single monmap update without being removed from cluster state's metata first, so, without this change, we will not update the metadata associated with that monitor, hence the mgr modules which consumes the metadata is not updated accordingly and keep reporting the stale information. in this change, we always update the metadata associated with all monitor included by the latest monmap. multiple "mon metadata" commands are sent to monitor for retrieving their updated metadata, instead of sending a single one, so that we can reuse "MetadataUpdate" to update the metadata of a given daemon. as the number of monitors in a typical cluster is relatively small, and the frequency of monmap update is low, so this overhead should be fine. unlike other places where we ask mon for metadata in Mgr class, the code sending the mon command for updated monitor metata is located outside of `cluster_state.with_monmap()` block, the reason is that `with_monmap()` is guraded by the monc_lock under the hood, while `start_mon_command()` also need to acquire the monc_lock, which is not a recursive lock. so we have to do this out of the `with_monmap()` block. Fixes: https://tracker.ceph.com/issues/48905 Signed-off-by: Kefu Chai <kchai@redhat.com> (cherry picked from commit c037f4c) backport: - path: src/mgr/Mgr.cc comment: octopus don't declared `fmt`

this change addresses a regression introduced by c037f4c also remove the "P" before the json command. see also: https://tracker.ceph.com/issues/48905 Fixes: https://tracker.ceph.com/issues/49661 Signed-off-by: Kefu Chai <kchai@redhat.com> (cherry picked from commit 8fc290b)

k0ste · 2021-03-26T10:42:44Z

needs-qa

tchaikov reviewed Feb 2, 2021

View reviewed changes

src/mgr/Mgr.cc Show resolved Hide resolved

k0ste force-pushed the wip-49004-octopus branch from a2cfb01 to 11ab7c5 Compare February 3, 2021 10:03

smithfarm added this to the octopus milestone Feb 25, 2021

smithfarm added core mgr needs-qa octopus-batch-1 labels Feb 25, 2021

yuriw added the wip-yuri-testing label Mar 1, 2021

ideepika approved these changes Mar 4, 2021

View reviewed changes

k0ste requested review from a team, alfonsomthd and aaSharma14 and removed request for a team March 5, 2021 13:12

epuertat force-pushed the wip-49004-octopus branch 4 times, most recently from 646e4f5 to 0cdf54b Compare March 5, 2021 22:05

epuertat force-pushed the wip-49004-octopus branch 2 times, most recently from 37f2823 to 1c71ddb Compare March 6, 2021 17:54

epuertat added the DNM label Mar 6, 2021

epuertat force-pushed the wip-49004-octopus branch from 1c71ddb to ed6add5 Compare March 6, 2021 18:40

epuertat force-pushed the wip-49004-octopus branch from ed6add5 to 11ab7c5 Compare March 6, 2021 19:38

yuriw removed needs-qa wip-yuri-testing labels Mar 10, 2021

k0ste force-pushed the wip-49004-octopus branch 2 times, most recently from eda3796 to aa7c5ee Compare March 11, 2021 08:25

k0ste force-pushed the wip-49004-octopus branch from aa7c5ee to a487353 Compare March 26, 2021 07:58

k0ste force-pushed the wip-49004-octopus branch from a487353 to db97ea4 Compare March 26, 2021 08:00

ideepika added the needs-qa label Mar 27, 2021

yuriw added wip-yuri-testing and removed DNM labels Mar 31, 2021

yuriw merged commit a9655fd into ceph:octopus Apr 6, 2021

octopus: mgr: update mon metadata when monmap is updated #39219

octopus: mgr: update mon metadata when monmap is updated #39219

k0ste commented Feb 2, 2021 •

edited by smithfarm

ideepika commented Mar 4, 2021

ideepika left a comment

epuertat commented Mar 4, 2021

epuertat commented Mar 4, 2021

yuriw commented Mar 4, 2021

epuertat commented Mar 4, 2021

epuertat commented Mar 4, 2021

yuriw commented Mar 4, 2021 •

edited

yuriw commented Mar 4, 2021

yuriw commented Mar 4, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 6, 2021

epuertat commented Mar 6, 2021

epuertat commented Mar 6, 2021 •

edited

tchaikov commented Mar 9, 2021

k0ste commented Mar 10, 2021

yuriw commented Mar 10, 2021

k0ste commented Mar 26, 2021

octopus: mgr: update mon metadata when monmap is updated #39219

octopus: mgr: update mon metadata when monmap is updated #39219

Conversation

k0ste commented Feb 2, 2021 • edited by smithfarm

ideepika commented Mar 4, 2021

ideepika left a comment

Choose a reason for hiding this comment

epuertat commented Mar 4, 2021

epuertat commented Mar 4, 2021

yuriw commented Mar 4, 2021

epuertat commented Mar 4, 2021

epuertat commented Mar 4, 2021

yuriw commented Mar 4, 2021 • edited

yuriw commented Mar 4, 2021

yuriw commented Mar 4, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 5, 2021

epuertat commented Mar 6, 2021

epuertat commented Mar 6, 2021

epuertat commented Mar 6, 2021 • edited

tchaikov commented Mar 9, 2021

k0ste commented Mar 10, 2021

yuriw commented Mar 10, 2021

k0ste commented Mar 26, 2021

k0ste commented Feb 2, 2021 •

edited by smithfarm

yuriw commented Mar 4, 2021 •

edited

epuertat commented Mar 6, 2021 •

edited