mgr: add the ip addr of standbys #16476

renhwztetecs · 2017-07-21T10:04:52Z

we need to manage the ip addr of the "standbys" state,
because the hostname/gid is insufficient to locate the
Standby node.

[root@node173 build]# ceph mgr metadata
[
{
"id": "node173",
"addr": "10.118.202.173",
"arch": "x86_64",
"ceph_version": "ceph version 12.1.2-589-gc8c837f (c8c837f) luminous (rc)",
"cpu": "Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz",
"distro": "centos",
"distro_description": "CentOS Linux 7 (Core)",
"distro_version": "7",
"hostname": "node173",
"kernel_description": "#1 SMP Mon Jun 30 12:09:22 UTC 2014",
"kernel_version": "3.10.0-123.el7.x86_64",
"mem_swap_kb": "4079612",
"mem_total_kb": "3768768",
"os": "Linux"
}
]

Signed-off-by: huanwen ren ren.huanwen@zte.com.cn

jcsp · 2017-07-21T10:24:43Z

I'm not sure why this is needed -- what is it that needs to see the address of the client?

For human eyes, the name is what is meant to act as the identifier.

renhwztetecs · 2017-07-21T11:15:38Z

@jcsp
Usually host hostname only one, but the corresponding IP address may exist more than one, we do not know the use of Mgr is that ip address

liuchang0812 · 2017-07-21T11:48:09Z

I think it would be better to put the IP informations to ceph mgr metadata. see liuchang0812#4

renhwztetecs · 2017-07-21T14:42:40Z

@liuchang0812 👍
I would like to get the ip adds information from monc in simple way.

liuchang0812 · 2017-07-27T14:45:56Z

sage merged metadata command PR already, you could use pybind to get standby's addresses

jcsp · 2017-07-27T15:03:32Z

I'm still not clear why this is needed. Your mgr daemon name is unique, and if it is a hostname then you can resolve it to an IP address (externally) if you need to.

Can you explain (beyond just saying that you want to know the IP address), why you need it. Is there some external piece of software that wants to know?

renhwztetecs · 2017-07-27T15:12:32Z

@liuchang0812
thank you for your hint
@jcsp
yes, we have their own management system need to know all the state of mgr, including standby, and for our management system ip address is more intuitive than the host name

tchaikov · 2017-07-27T15:15:03Z

@renhwztetecs why it's more intuitive ? is there any additional information that you cannot get by resolving the IP addresss?

renhwztetecs · 2017-07-29T03:14:35Z

@tchaikov I'm later
If the standby provides ip address, we can directly through the MgrMap to obtain all the standby IP address in the cluster, if there is no standby IP address, you also need to go to hostname to find IP

liewegas · 2017-08-08T21:03:28Z

src/mgr/MgrStandby.cc

@@ -157,7 +157,7 @@ void MgrStandby::send_beacon()
  // as available in the map)
  bool available = active_mgr != nullptr && active_mgr->is_initialized();

-  auto addr = available ? active_mgr->get_server_addr() : entity_addr_t();
+  auto addr = available ? active_mgr->get_server_addr() : monc.get_my_addr();


The risk here seems to be that someone might assume the standby addr will become the active addr when that's not actually the case. That aside I do'nt have a philosophical problem with this

thanks for your suggest

jcsp · 2017-08-09T09:25:34Z

Since 1bf4a89, the mgr has sent metadata to the mons -- I would be okay with adding the IP address there. But let's make it just the IP address, and not the whole entity_addr_t that comes out of monc.get_my_addr().

I prefer to use metadata rather than mgrmap, because the metadata's purpose is to hold advisory information that may be interesting to external systems, whereas the mgrmap is for information that Ceph needs internally.

liewegas · 2017-08-09T13:52:33Z

👍

renhwztetecs · 2017-08-10T00:45:56Z

I will change to metadata

renhwztetecs · 2017-08-12T08:34:39Z

jenkins retest this please

renhwztetecs · 2017-08-14T10:27:33Z

changed to metadata

jcsp · 2017-08-14T12:30:09Z

src/mgr/MgrStandby.cc

-
+
+  const char *host_ip = NULL;
+  char addr_buf[129];


Avoid hardcoded buffer size, I guess this should be MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1?

Yes, this is more elegant.
For INET and INET6 do a different definition, but also do not need "+1"

jcsp · 2017-08-14T12:30:57Z

src/mgr/MgrStandby.cc

+  default:
+    break;
+  };
+  metadata["addr"] = host_ip;


inet_ntop returns null on errors -- must check the return value to avoid possibility of assigning null to a std string.

@jcsp
I assigned a value of “0.0.0.0” when "host_ip" is NULL, and log it,
I'm not sure if there is a better way.
thank you

jcsp · 2017-08-15T09:10:34Z

src/mgr/MgrStandby.cc

+    break;
+  };
+  if (NULL == host_ip) {
+    metadata["addr"] = "0.0.0.0";


If the addr isn't AF_INT or AF_INET6, then it's probably not IP at all (an RDMA messenger or something like that), so I'd just return an empty string rather than something IP-like.

yes,""is good

jcsp · 2017-08-15T09:11:43Z

src/mgr/MgrStandby.cc

+
+  switch(addr_standby.get_family()) {
+  case AF_INET:
+    char addr_buf_inet[INET_ADDRSTRLEN];


Defining the buffer inside the switch {} and then using the host_ip pointer to it from outside the {} seems incorrect -- I think the compiler is allowed to re-use that space after the end of the {}

@jcsp
Yes has this problem,thanks.
unified use char addr_buf[INET6_ADDRSTRLEN],
because INET6_ADDRSTRLEN=46 > INET_ADDRSTRLEN=16.
reference:https://linux.die.net/man/3/inet_pton

@jcsp
All changed it.

renhwztetecs · 2017-08-25T08:16:27Z

jenkins retest this please

liewegas · 2017-08-30T02:41:15Z

src/mgr/MgrStandby.cc

+                        INET6_ADDRSTRLEN);
+    break;
+  default:
+    break;


Can you move this into an entity_addr_t method (msg/msg_types.h) called something like std::string ip_only_to_str() const?

Then the change to this files becomes a single line,

metadata["addr"] = monc.get_my_addr().ip_only_to_str();

@liewegas
I have changed it

liewegas · 2017-08-31T03:25:04Z

src/msg/msg_types.h

+    } else {
+      ip_str = host_ip; 
+    }
+    return ip_str;


this can be simplified to

return host_ip ? host_ip : "";

and drop the ip_str decl

Otherwise, looks good, thanks!

@liewegas
“Less is More” :-)
updated it

tchaikov · 2017-08-31T08:27:58Z

src/msg/msg_types.h

@@ -384,6 +385,24 @@ struct entity_addr_t {
    }
  }

+  std::string ip_only_to_str() {


please mark this method const. and move the implementation to .cc file. as msg_types.h is included by quite a few places, put the implementation here could prolong the compiling time.

At first I also considered this problem, but found that all the implementation in "msg_types.h"

i don't get you. could you rephrase you reply?

sorry, I may not have described it clearly, I mean whether all the implementations in msg_types.h are placed in msg_types.cc？

let's put this one in the .cc file. we can move others later.

@liewegas
ok! updated it

tchaikov · 2017-09-01T13:47:30Z

src/msg/msg_types.cc

@@ -290,3 +290,22 @@ void entity_addrvec_t::generate_test_instances(list<entity_addrvec_t*>& ls)
  ls.back()->v.push_back(entity_addr_t());
  ls.back()->v.push_back(entity_addr_t());
 }
+
+const std::string ip_only_to_str() 


i mean, std::string ip_only_to_str() const. as this method is not changing in4_addr, IIUC.

@tchaikov
updated it, thank you

we need to manage the ip addr of the "standbys" state, because the hostname/gid is insufficient to locate the Standby node. we add ip of the mgr standby to metadata. Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

comments addressed

tchaikov · 2017-09-05T05:01:03Z

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch 3 times, most recently from 21dec25 to b29a658 Compare July 21, 2017 11:12

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch from b29a658 to 9cd8c73 Compare July 21, 2017 14:36

liewegas reviewed Aug 8, 2017

View reviewed changes

liewegas added mgr mon labels Aug 8, 2017

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch from 9cd8c73 to be62513 Compare August 12, 2017 08:02

jcsp reviewed Aug 14, 2017

View reviewed changes

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch from be62513 to 97ff0c7 Compare August 15, 2017 01:03

jcsp reviewed Aug 15, 2017

View reviewed changes

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch 2 times, most recently from c1fc759 to c56a279 Compare August 15, 2017 11:28

jcsp approved these changes Aug 24, 2017

View reviewed changes

liewegas previously requested changes Aug 30, 2017

View reviewed changes

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch from c56a279 to fa107a5 Compare August 30, 2017 08:48

liewegas reviewed Aug 31, 2017

View reviewed changes

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch 2 times, most recently from ae4c79a to 6fb8e48 Compare August 31, 2017 03:49

tchaikov reviewed Aug 31, 2017

View reviewed changes

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch 2 times, most recently from da39c8b to d52781a Compare September 1, 2017 09:23

tchaikov reviewed Sep 1, 2017

View reviewed changes

mgr: add the ip addr of standbys

389d676

we need to manage the ip addr of the "standbys" state, because the hostname/gid is insufficient to locate the Standby node. we add ip of the mgr standby to metadata. Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

renhwztetecs force-pushed the renhw-wip-mgr-standby-ip branch from d52781a to 389d676 Compare September 4, 2017 03:01

tchaikov approved these changes Sep 4, 2017

View reviewed changes

tchaikov added the needs-qa label Sep 4, 2017

jcsp approved these changes Sep 4, 2017

View reviewed changes

tchaikov added the wip-kefu-testing label Sep 4, 2017

tchaikov merged commit c46220e into ceph:master Sep 5, 2017

mgr: add the ip addr of standbys #16476

mgr: add the ip addr of standbys #16476

Conversation

renhwztetecs commented Jul 21, 2017 • edited

jcsp commented Jul 21, 2017

renhwztetecs commented Jul 21, 2017

liuchang0812 commented Jul 21, 2017

renhwztetecs commented Jul 21, 2017

liuchang0812 commented Jul 27, 2017

jcsp commented Jul 27, 2017

renhwztetecs commented Jul 27, 2017

tchaikov commented Jul 27, 2017

renhwztetecs commented Jul 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcsp commented Aug 9, 2017 • edited

liewegas commented Aug 9, 2017

renhwztetecs commented Aug 10, 2017

renhwztetecs commented Aug 12, 2017

renhwztetecs commented Aug 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renhwztetecs Aug 15, 2017 • edited

Choose a reason for hiding this comment

renhwztetecs commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Aug 31, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Sep 1, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov commented Sep 5, 2017

renhwztetecs commented Jul 21, 2017 •

edited

jcsp commented Aug 9, 2017 •

edited

renhwztetecs Aug 15, 2017 •

edited

tchaikov Aug 31, 2017 •

edited

tchaikov Sep 1, 2017 •

edited